SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[ ]: nu_hat_exp = (8 / alpha) * (p * L / 2 + kappa * 4 * p * (1 + p / beta) * L_hat**2 + rho * 4 * p * L**2 + rho * (8 * p / tau) * (2 * p * (1 + 2 * p / tau) * L**2 + 4 * p * tau**2 * omega * L_hat**2 / n)) nu_hat = Symbol(r'\hat{h}[\nu]\), nonnegative=True) # The left hand sides of the inequalities rho_lhs = nu_hat * (8 / (alpha * p * L_bar ** 2)) kappa_lhs = ((8 * p * omega) / (n * L_bar * beta) + nu_hat * (8 * omega) / (n * beta * L_bar**2) + rho * (8 * p) / tau * (8 * tau ** 2 * omega) / (n * beta))
4.2 Display Them
[ ]: display(nu_hat_exp)  
display(rho_lhs)  
display(kappa_lhs)
4.3 Symbolically Calculate The Steps From The Proof
[ ]: constraints_agg = ConstraintsAggregator()
rho_lhs = rho_lhs.subs(nu_hat, nu_hat_exp)
kappa_lhs = kappa_lhs.subs(nu_hat, nu_hat_exp)
# Group Terms w.r.t. kappa
kappa_lhs(poly = sympy.poly(kappa_lhs, kappa)
kappa_coeff = kappa_lhs.poly.all_coeffs()
kappa_lhs = kappa_lhs.exp().collect(kappa)
fw.write(simpy.latex(kappa_lhs) + "", \label{eq:kappa Expand})
# Find Conditions When The Coefficients Near Kappa <= 1/2
terms = sympy.exp(kappa_coeff[0]).args
latex_string = constraints_agg.addconstraints terms)
fw.write(latex_string)
# Define kappa  
kappa_solution = (2 * kappacoef[1]).expand()  
fw.write("\\kappa\eqdef " + sympy.latex(kappa_solution) + ".\label{eq:  $-\text{kappa\_sol}\}")$   
# Group Terms w.r.t. rho  
rho_lhs = rho_lhs.subs(kappa, kappa_solution)  
rho_lhs = rho_lhs.exp().collect(rho)  
fw.write(sympy.latex(rho_lhs) + ",\label{eq:rho Expand}")  
# Find Conditions When The Coefficients Near rho <= 1/2  
rho_lhs.poly = sympy.poly(rho_lhs, rho)  
rho coef = rho_lhs.poly.all_coeffs()  
terms = sympy.exp(rhocoef[0]).args  
latex_string = constraints_agg.add Constraints terms)  
fw.write(latex_string)  
# Define rho  
rho Solution = (2 * rhocoef[1]).expand()  
fw.write("\\rho\eqdef " + sympy.latex(rho_solution) + ".\label{eq:rho_sol}")
5 Assistant for "Second Symbolically Computed" Lemma
5.1 First Inequality
[ ]: bregmancoef  $=$  ((4 \* omega \* L_max)/ (n \* L_bar) + kappa \* 8 \* (1 + p / beta) \* L_max + (nu_hat / L_bar\*\*2) \* (4 \* omega \* L_max / (p \* n) + 8 \* L / $\rightarrow (\mathrm{p} * \mathrm{alpha}))$  + rho \* 8 \* L + lmbda \* (4 \* (1 + (2 \* p) / tau) \* L + 8 \* tau\*\*2 \* omega  $\rightarrow$  L_max / n)) display(bregmancoef) lmbda_solution  $=$  rho \* 8 \* p / tau # Substitue all known expressions to the term bregmancoef  $=$  \\ bregmancoef.subs(nu_hat, nu_hat_exp).subs(lmbda, lmbda_solution).  $\rightarrow$  subs(kappa, kappa_solution).subs(rho, rho_solution) bregmancoef  $=$  bregmancoef.exp().simplify().expand() latex_string = latex_repres_of_inequality(bregmancoef) fw.write(latex_string) # Find Conditions When bregmancoef is Less or Equal <= 1/2 terms  $=$  bregmancoef.args
latex_string = constraints_agg.add Constraints/terms)
fw.write(latex_string)
5.2 Second Inequality
[ ]: distcoef  $=$  (4 / L_bar) \* (L / 2 + kappa \* 4 \* (1 + p / beta) \* L_hat \*\* 2 + rho \* 4 \* L\*\*2 + lmbda \* (2 \* (1 + 2 \* p / tau) \* L\*\*2 + 4 \* tau\*\*2  $\Rightarrow$  omega \* L_hat\*\*2 / n)) display(distcoef) distcoef  $=$  \ -distcoef.subs(lmbda, lmbda_solution).subs(kappa, kappa_solution).subs(rho, rho Solution) distcoef  $=$  distcoef.exp().simplify().expand() latex_string  $=$  latex_repres_of_inequality(dist coef, rhs="1") fw.write(latex_string) terms  $=$  distcoef.args latex_string  $=$  constraints_agg.add Constraintsterms) fw.write(latex_string)
6 Check That The Constraints Follow From The Inequalities from The "Auxiliary Inequalities" Lemma.
6.1 The Inequalities from The "Auxiliary Inequalities" Lemma:
[ ]: from collections import OrderedDict
    _contant = 1
    proposals = OrderedDict(
        ("eq:lipt:max_2", (L_max * omega * p**2) / (beta**2 * n)), 
        ("eq:lipt:max_1", (L_max * omega * p) / (beta * n)), 
        ("eq:lipt:max", (L_max * omega) / (n)), 
        ("eq:lipt:l_l_max_2", (sympy.sqrt(L * L_max) * p * sympy.sqrt(omega * _tau)) / (alpha * beta * sympy.sqrt(n))), 
        ("eq:lipt:l_l_max_1", (sympy.sqrt(L * L_max) * sympy.sqrt(p * omega * _tau)) / (alpha * sympy.sqrt(beta * n))), 
        ("eq:lipt:l_l_max_p", (sympy.sqrt(L * L_max) * sympy.sqrt(p * omega * _tau)) / (alpha * sympy.sqrt(n))), 
        ("eq:lipt:hat_2", (L_hat * p * sympy.sqrt(omega * tau)) / (alpha * _beta * sympy.sqrt(n))), 
        ("eq:lipt:hat_1", (L_hat * sympy.sqrt(p * omega * tau)) / (alpha * _sympy.sqrt(beta * n))), 
        ("eq:lipt:hat_p", (L_hat * sympy.sqrt(p * omega * tau)) / (alpha * _sympy.sqrt(n))),
("eq:lipt:hat_alpha_2", (L_hat * p * sympy.sqrt(omega)) / (beta * sympy. sqrt(alpha * n))), ("eq:lipt:hat_alpha_1", (L_hat * sympy.sqrt(p * omega)) / (sympy.sqrt(beta * alpha * n))), ("eq:lipt:hat_no_alpha_2", (L_hat * p * sympy.sqrt(omega)) / (beta * sympy. sqrt(n))), ("eq:lipt:hat_no_alpha_1", (L_hat * sympy.sqrt(p * omega)) / (sympy.sqrt(beta * n))), ("eq:lipt:plain", (L) / (alpha)), ("eq:lipt:plain_p_alpha", (L * p) / (alpha * tau)), ("eq:lipt:plain_no_alpha", (L)), ("eq:lipt:double_lipt_2", sympy.cbrt((L * L_hat**2 * omega * p**4) / (alpha**2 * beta**2 * n * tau**2))), ("eq:lipt:double_lipt_1", sympy.cbrt((L * L_hat**2 * omega * p**3) / (alpha**2 * beta * n * tau**2))), ]) const = 660508 for k, proposal in proposals.items(): proposals[k] = (const * proposal / L_bar).expand() proposals Factors = [get Factors(proposal) for _, proposal in proposals.items())
6.2 Search The Right Inequalities for Each Constraint
[ ]: # Takes ~1 hour on a laptop  
import tqdm  
constraints = constraints_agg.get Constraints()  
constraints_inequalities = []  
for i, factor in enumerate(tqdm.tqdm(constraints)):  
    num_of_base Factors_to_use = -int(factor[L_bar])  
    path = search Factors(factor, num_of_base Factors_to_use, proposals Factors, pos_hints=[L, L_max, L_hat, omega, p], neg_hints=[alpha, n])  
    constraints_inequalities.append(path)  
    assert path is not None
[ ]: text = constraints_agg.prepare_text_from_proposals(proposals,  $\rightarrow$  constraints_inequalities) fw.write(text)
P.1 File utils.py
import os   
from collections import defaultdict   
from copy import copy   
import sympy   
class _defaultdictwithconst(defaultdict): def __init__(self, *args, **kwargs): super(_defaultdictwithconst, self).__init__((*args, **kwargs) self const  $=$  None   
def __copy__(self): obj  $=$  super(_defaultdictwithconst, self).__copy_(obj const  $=$  self const return obj   
def get Factors(term): "" Converts a sympy expression with the sympy.Mul type to a dictionary with factors. Ex: A\~2 / B\~(1/2) -> {A: 2, B: -1/2} Args: param1: sympy expression (sympy.Mul).   
factors  $\equiv$  _defaultdictwithconst(int) factors const  $= 1$  const Assigned  $\equiv$  False assert isinstance(term, sympy.Mul) for el in term.args: freesymbols  $=$  list(el.freesymbols) if len(freesymbols)  $= = 0$  : assert not const Assigned const Assigned  $\equiv$  True factors const  $=$  int(el) continue assert len(freesymbols)  $= = 1$  power  $=$  el.aspowers_dict() [free Symbols [0]] assert isinstance(power, sympy.Rational) factors[freesymbols [0]]  $=$  power return factors   
def get_term(factors): "" The inverse function to the get_factors function: converts factors to a term Ex: {A: 2, B: -1/2} -> A\~2 / B\~(1/2) Args: param1: dictionary with factors   
result  $=$  factors const for k, v in factors.items(): result  $=$  result \* k \*\* v return result   
def searchFactors(factors, num_of_base_factors_to_use, base Factors, pos_hints=[], neg_hints=[]):   
"" Checks if it possible to decompose 'factors' using num_of_base_factors from 'base Factors'   
Args: param1: dictionary with factors   
param2: number of base factors to use   
param3: list of dictionaries with factors
param4 and param5: hints to the algorithm that helps to improve performance Returns:
A list with indices (with repetitions) of 'base Factors' that compose 'factors'.
If it not possible, then the function returns None.
def_check_hints(factors):
    for k in pos_hints:
        assert factors_[k] >= 0
    for k in neg_hints:
        assert factors_[k] <= 0
    check_hints(factors)
for base in base Factors:
    check_hints(base)
def_search_factors(factors, num_of_baseFactors_to_use, path, base Factors):
    if num_of_baseFactors_to_use == 0:
        return factors_const <= 1 and all([factors[k] == 0 for k in factors])
for index, choice in enumerate(base Factors):
    factor_choice = copy(factors)
    factor_choice const = factor_choice const / choice const
    for k, v in choice.items():
        factor_choice[k] = factor_choice[k] - v
skip = False
for k in pos_hints:
    if factor_choice[k] < 0:
        skip = True
break
for k in neg_hints:
    if factor_choice[k] > 0:
        skip = True
break
if not skip and _search_factors(factor_choice, num_of_baseFactors_to_use
- 1, path, base Factors):
    path.append(index)
return True
return False
path = []
if _search_factors(factors, num_of_baseFactors_to_use, path, base Factors):
return path
else:
return None
class FileWriter(object):
def __init__(self, file_path):
    self._file_path = file_path
    assert not os.path.exists(file_path)
def write(self, text):
    with open(self._file_path, "a") as fd:
        fd.write("\{}\n".format(text))
class ConstraintsAggregator(object):
def __init__(self):
    self._constraints = []
def get Constraints(self):
    return self._constraints
def addConstraints(self, terms):
    denom_constant = self._denom_constant/terms)
    terms_global_index = []
    for term in terms:
        terms_global_index.append(len(self._constraints))
64   
65   
66   
67   
68   
69   
70   
71   
72   
73   
74   
75   
76   
77   
78   
79   
80   
81   
82   
83   
84   
85   
86   
87   
88   
89   
90   
91   
92   
93   
94   
95   
96   
97   
98   
99   
100   
101   
102   
103   
104   
105   
106   
107   
108   
109   
110   
111   
112   
113   
114   
115   
116   
117   
118   
119   
120   
121   
122
def prepare_text_from_proposals(self, proposals, constraints_inequalities):
    # assert len(constraints_inequalities) == len(self._constraints)
    text = ""
    proposals_labels = list(proposals.keys())
    for global_index, ineq in enumerate(constraints_inequalities):
        constraint_label = self._get_label(global_index)
        text += "&\eqref{[{}]}".format(constraint_label) + "\textnormal{follows from} "
        for i, index_proposal in enumerate(ineq):
            text +="\eqref{[{}]}.format(proposals_labels[index_proposal])
            if i == len(ineq) - 1:
                text +="."
            else:
                text +=","
            text +="\\""
        return text
def _denom_constant(self, terms):
    return 2 * len/terms)
def _get_label(self, global_index):
    return "eq:sympy:constraints:{}" format(global_index)
def _prepare_text(self, terms, terms_global_index):
    text = ""
    denom_constant = self._denom_constant/terms)
    num_per_column = 3
    for i, (global_index, term) in enumerate(zipterms_global_index, terms)):
        if i % num_per_column == 0:
            text += r"\begin{tabularx}{1.2\linewidth}{XXX}"text += r"\begin{equation}"text += sympy.latex(term) + "\leq\frac{\text{frac}\{1\}}{\{4\}}\{...\}}".format(
            i % num_per_column == (num_per_column - 1):
                text += r"\end{tabularx}"else:
                    text += "&"
    def ceildiv(a, b):
        return -(a // -b)
for i in range(len/terms), ceildiv(len/terms), num_per_column):
    if i % num_per_column == (num_per_column - 1):
        text += r"\end{tabularx}"else:
            text +="&"
return text
latex_repres_of_inequality(coef, rho="\frac{\text{frac}\{3\}{\{4\}}}{\{4\}}")
latex_string = "&"
for i, term in enumerate(coef.args):
latex_string = latex_string + sympy.latex(term)
if i == len(coef.args) - 1:
latex_string += " \leq" + rho
else:
if i % 3 == 2:
latex_string += " \ \ \ \ & +"
else:
latex_string += " + "
return latex_string
124   
125   
126   
127   
128   
129   
130   
131   
132   
133   
134   
135   
136   
137   
138   
139   
140   
141   
142   
143   
144   
145   
146   
147   
148   
149   
150   
151   
152   
153   
154   
155   
156   
157   
158   
159   
160   
161   
162   
163   
164   
165   
166   
167   
168   
169   
170   
171   
172   
173   
174   
175   
176   
177   
178   
179   
180   
181   
182   
183   
184
Q Experiments
Q.1 Setup
We now conduct experiments on the practical logistic regression task with LIBSVM datasets (Chang and Lin, 2011) (under the 3-clause BSD license). The experiments were implemented in Python 3.7.9. The distributed environment was emulated on machines with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz. In each plot we show the relations between the total number of coordinates transmitted from and to the server and function values. The parameters of the algorithms are taken as suggested by the corresponding theory, except for the step sizes that we fine-tune from a set  $\{2^i \mid i \in [-20, 20]\}$ . For 2Direction, we use parameters from Theorem 5.2 and finetune the step size  $L$ .
We solve the logistic regression problem:
$$
f _ {i} (x _ {1}, \ldots , x _ {c}) := - \frac {1}{m} \sum_ {j = 1} ^ {m} \log \left(\frac {\exp \left(a _ {i j} ^ {\top} x _ {y _ {i j}}\right)}{\sum_ {y = 1} ^ {c} \exp \left(a _ {i j} ^ {\top} x _ {y}\right)}\right),
$$
where  $x_{1},\ldots ,x_{c}\in \mathbb{R}^{d}$ ,  $c$  is the number of unique labels,  $a_{ij}\in \mathbb{R}^d$  is a feature of a sample on the  $i^{\mathrm{th}}$  worker,  $y_{ij}$  is a corresponding label and  $m$  is the number of samples located on the  $i^{\mathrm{th}}$  worker. The Rand  $K$  compressor is used to compress information from the workers to the server, the Top  $K$  compressor is used to compress information from the server to the workers. The performance of algorithms is compared on CIFAR10 (Krizhevsky et al., 2009) (# of features = 3072, # of samples equals 50,000), and real-sim (# of features = 20958, # of samples equals 72,309) datasets.
Q.2 Results
In Figure 2, 3 and 4 we provide empirical communication complexities of our experiments. For each algorithm, we show the three best experiments. The experiments are collaborative with our theory. 2Direction enjoys faster convergence rates than EF21-P + DIANA and AGD.
Figure 2: Logistic Regression with real-sim dataset. # of workers  $n = 100$ .  $K = 1000$  in all compressors.
Figure 3: Logistic Regression with CIFAR10 dataset. # of workers  $n = 10$ .  $K = 1000$  in all compressors.
Figure 4: Logistic Regression with CIFAR10 dataset. # of workers  $n = 100$ .  $K = 1000$  in all compressors.
R Convergence Rate of CANITA obtained by Li and Richtárik (2021)
In their Equation (54), Li and Richtárik (2021) derive the following bound for their CANITA method:
$$
\mathbb {E} \left[ F ^ {T + 1} \right] \leq \mathcal {O} \left(\max \left\{\frac {(1 + \omega) ^ {3}}{T ^ {3}}, \frac {(1 + b) (\beta + 3 / 2) L}{T ^ {2}} \right\}\right).
$$
In the regime when  $\omega \geq n$ , choosing  $b = \omega$  and  $\beta = \Theta \left(\frac{\omega}{n}\right)$  in their Equation (10) gives
$$
\begin{array}{l} \mathbb {E} \left[ F ^ {T + 1} \right] \leq \mathcal {O} \left(\max  \left\{\frac {(1 + \omega) ^ {3}}{T ^ {3}}, \frac {(1 + b) (\beta + 3 / 2) L}{T ^ {2}} \right\}\right) \\ = \mathcal {O} \left(\max  \left\{\frac {(1 + \omega) ^ {3}}{T ^ {3}}, \frac {\omega (\omega / n + 3 / 2) L}{T ^ {2}} \right\}\right) \\ = \mathcal {O} \left(\max \left\{\frac {(1 + \omega) ^ {3}}{T ^ {3}}, \frac {\omega^ {2} L}{n T ^ {2}} \right\}\right). \\ \end{array}
$$
This means that the correct convergence rate of the CANITA method (Li and Richtárik, 2021) is
$$
T = \left\{ \begin{array}{l l} \Theta \left(\frac {\omega}{\varepsilon^ {1 / 3}} + \frac {\omega}{\sqrt {n}} \sqrt {\frac {L}{\varepsilon}}\right), & \omega \geq n, \\ \Theta \left(\frac {\omega}{\varepsilon^ {1 / 3}} + \left(1 + \frac {\omega^ {3 / 4}}{n ^ {1 / 4}}\right) \sqrt {\frac {L}{\varepsilon}}\right), & \omega <   n. \end{array} \right. \tag {329}
$$
Comparing this result with our Theorem E.14 describing the convergence of our method 2Direction, one can see that in the low accuracy regimes (in particular, when  $\frac{\omega}{\varepsilon^{1/3}}$  dominates in (329)), our result improves  $\Theta\left(\frac{1}{\varepsilon^{1/3}}\right)$  to at least  $\Theta\left(\log \frac{1}{\varepsilon}\right)$ . However, the dependence  $\Theta\left(\log \frac{1}{\varepsilon}\right)$  should not be overly surprising as it was observed by Lan et al. (2019) already, albeit in a somewhat different context.
S Comparison with ADIANA
We now want to check that our rate (14) restores the rate from (Li et al., 2020). Since ADIANA only compresses from the workers to the server, let us take  $r = 0$ , the identity compressor operator  $\mathcal{C}^P(x) = x$  for all  $x \in \mathbb{R}^d$ , which does not perform compression, and, as in (Li et al., 2020), consider the optimistic case, when  $L_{\max} = L$ . For this compressor, we have  $\alpha = 1$  in (2). Note that  $\mu_{\omega, \alpha}^r = 0$ . Thus the iteration complexity (14) equals
$$
\begin{array}{l} T ^ {\text {o p t i m i s t i c}} = \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\mu}}, \sqrt {\frac {L (\omega + 1)}{n ^ {1 / 3} \mu}}, \sqrt {\frac {L (\omega + 1) ^ {3 / 2}}{\sqrt {n} \mu}}, \sqrt {\frac {L \omega (\omega + 1)}{n \mu}}, (\omega + 1) \right\}\right) \\ = \widetilde {\Theta} \left(\max  \left\{\sqrt {\frac {L}{\mu}}, \sqrt {\frac {L (\omega + 1) ^ {3 / 2}}{\sqrt {n} \mu}}, \sqrt {\frac {L \omega (\omega + 1)}{n \mu}}, (\omega + 1) \right\}\right), \tag {330} \\ \end{array}
$$
where we use Young's inequality:  $\sqrt{\frac{L}{\mu}}\sqrt{\frac{(\omega + 1)}{n^{1/3}}} \leq \sqrt{\frac{L}{\mu}}\sqrt{\frac{1}{3} \times 1^3 + \frac{2}{3}\frac{(\omega + 1)^{3/2}}{\sqrt{n}}}$ . Without the server-to-worker compression, Algorithm 1 has the same iteration (330) and communication complexity as (Li et al., 2020).