SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Summary of popular estimators with respective parameters  $A$ ,  $B$ ,  $C$ ,  $b$  and  $c$ , satisfying our general Biased ABC framework. Constants  $L_{i}$  are from Assumption 13,  $\Delta^{*}$  is defined in (26). For more estimators, see Table 8.
6 Convergence of biased SGD under the biased ABC assumption
Convergence rates of theorems below are summarized in Table 4 and compared to their counterparts.
Estimator\Assumption	A1	A2	A3	A4	A5	A6	A7	A8	A9
Biased independent sampling [This paper]	✘	✘	✘	✘	✘	✘	✘	✘	✓
Top- \( k \) sparsification [Aji and Heafield, 2017]	✓	✓	✓	✓	✓	✓	✘	✓	✓
Rand- \( k \) [Stich et al., 2018]	✓	✓	✘	✓	✘	✓	✘	✓	✓
Biased Rand- \( k \) [Beznosikov et al., 2020]	✓	✓	✓	✓	✘	✓	✘	✓	✓
Adaptive random sparsification [Beznosikov et al., 2020]	✓	✓	✓	✓	✘	✓	✘	✓	✓
General unbiased rounding [Beznosikov et al., 2020]	✓	✓	✘	✓	✘	✓	✘	✓	✓
Natural compression [Horváth et al., 2022]	✓	✓	✓	✓	✘	✓	✘	✓	✓
Scaled integer rounding [Sapio et al., 2021]	✓	✓	✘	✓	✓	✓	✓	✓	✓
Table 3: Coverage of popular estimators by known frameworks. For more estimators, see Table 9.
6.1 General nonconvex case
Theorem 3 Let Assumptions 0 and 9 hold. Let  $\delta^0 \stackrel{\text{def}}{=} f(x^0) - f^*$ , and choose the stepsize such that  $0 < \gamma \leq \frac{b}{LB}$ . Then the iterates  $\{x^t\}_{t \geq 0}$  of BiasedSGD (Algorithm (1)) satisfy
$$
\min  _ {0 \leq t \leq T - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \leq \frac {2 (1 + L A \gamma^ {2}) ^ {T}}{b \gamma T} \delta^ {0} + \frac {L C \gamma}{b} + \frac {c}{b}. \tag {20}
$$
While one can notice the possibility of an exponential blow-up in (20), by carefully controlling the stepsize we still can guarantee the convergence of BiasedSGD. In Corollaries 5 and 6 (see the appendix) we retrieve the results of Theorem 2 and Corollary 1 from [Khaled and Richtárik, 2023] for the unbiased case. In Corollary 7 (see the appendix) we retrieve the result that is worse than that in [Ajalloeian and Stich, 2020, Theorem 4] by a multiplicative factor and an extra additive term, but under milder conditions (cf. Biased ABC and BND in Figure 1; see also Claim 1). If we set  $A = c = 0$ , we recover the result of [Bottou et al., 2018, Theorem 4.8] (see Corollary 8 in the appendix).
6.2 Convergence under PL-condition
One of the popular generalizations of strong convexity in the literature is the Polyak-Lojasiewicz assumption [Polyak, 1963; Karimi et al., 2016; Lei et al., 2019]. First, we define this condition.
Assumption 10 (Polyak-Lojasiewicz) There exists  $\mu > 0$  such that  $\| \nabla f(x)\|^2 \geq 2\mu (f(x) - f^*)$ , for all  $x \in \mathbb{R}^d$ .
We now formulate a theorem that establishes the convergence of BiasedSGD for functions satisfying this assumption and Assumption 9.
Theorem 4 Let Assumptions 0, 9 and 10 hold. Choose a stepsize such that
$$
0 <   \gamma <   \min  \left\{\frac {\mu b}{L (A + \mu B)}, \frac {1}{\mu b} \right\}. \tag {21}
$$
Letting  $\delta^0\stackrel {def}{=}f(x^0) - f^*$ , for every  $T\geq 1$ , we have
$$
\mathbb {E} \left[ f (x ^ {T}) - f ^ {*} \right] \leq \left(1 - \gamma \mu b\right) ^ {T} \delta^ {0} + \frac {L C \gamma}{2 \mu b} + \frac {c}{\mu b}. \tag {22}
$$
When  $c = 0$ , the last term in (22) disappears, and we recover the best known rates under the Polyak-Lojasiewicz condition [Karimi et al., 2016], but under milder conditions (see Corollary 10 in the appendix). Further, if we set  $A = 0$ , we obtain a result that is slightly weaker than the one obtained by Ajalloeian and Stich [2020, Theorem 6], but under milder assumptions (cf. Biased ABC and BND in Figure 1; see also Claim 1).
6.3 Strongly convex case
Assumption 11 Let  $f$  be  $\mu$ -strongly-convex and continuously differentiable.
Theorem	Convergence rate	Compared to	Rate we compare to	Match?
Thm 3	O\(\left(\frac{\delta^0L}{\varepsilon^2}\max\left\{B,\frac{126^0A}{\varepsilon^2},\frac{2C}{\varepsilon^2}\right\}\right)\)	34-Thm 2	O\(\left(\frac{\delta^0L}{\varepsilon^2}\max\left\{B,\frac{126^0A}{\varepsilon^2},\frac{2C}{\varepsilon^2}\right\}\right)\)	✓
Thm 3	O\(\left(\max\left\{\frac{8(M+1)(m+1)}{(1-m)^2\varepsilon},\frac{16(M+1)\varphi^2+2\sigma^2}{(1-m)^2\varepsilon^2}\right\}L\delta^0\right)\)	1-Thm 4	O\(\left(\max\left\{\frac{M+1}{(1-m)\varepsilon},\frac{2\sigma^2}{(1-m)^2\varepsilon^2}\right\}L\delta^0\right)\)	✗
Thm 3	O\(\left(\max\left\{\frac{8Q}{\varepsilon^2q^2},\frac{4(U+u^2)}{\varepsilon^2q^2}\right\}L\delta^0\right)\)	6-Thm 4.8	O\(\left(\max\left\{\frac{8Q}{\varepsilon^2q^2},\frac{4(U+u^2)}{\varepsilon^2q^2}\right\}L\delta^0\right)\)	✓
Thm 4	\(\tilde{\mathcal{O}}\left(\max\left\{\frac{2(M+1)(m+1)}{1-m},\frac{2(M+1)\varphi^2+\sigma^2}{\epsilon\mu(1-m)+2\varphi^2}\right\}\frac{\kappa}{1-m}\right)\)	1-Thm 6	\(\tilde{\mathcal{O}}\left(\max\left\{(M+1),\frac{\sigma^2}{\epsilon\mu(1-m)+\varphi^2}\right\}\frac{\kappa}{1-m}\right)\)	✗
Thm 12	\(\tilde{\mathcal{O}}\left(\max\left\{2,\frac{L(U+u^2)}{q^2\mu},\frac{LQ}{\varepsilon\mu^2q^2}\right\}\right)\)	6-Thm 4.6	\(\tilde{\mathcal{O}}\left(\max\left\{2,\frac{L(U+u^2)}{q^2\mu},\frac{LQ}{\varepsilon\mu^2q^2}\right\}\right)\)	✓
Thm 12	\(\tilde{\mathcal{O}}\left(\left(\frac{\beta^2}{\alpha}\right)^2\frac{L}{\mu}\right)\)	5-Thm 12	\(\tilde{\mathcal{O}}\left(\frac{\beta^2}{\alpha}\frac{L}{\mu}\right)\)	✗
Thm 12	\(\tilde{\mathcal{O}}\left(\left(\frac{\beta}{\tau}\right)^2\frac{L}{\mu}\right)\)	5-Thm 13	\(\tilde{\mathcal{O}}\left(\frac{\beta}{\tau}\frac{L}{\mu}\right)\)	✗
Thm 12	\(\tilde{\mathcal{O}}\left(\delta^2\frac{L}{\mu}\right)\)	5-Thm 14	\(\tilde{\mathcal{O}}\left(\delta\frac{L}{\mu}\right)\)	✗
Table 4: Complexity comparison. We examine whether we can achieve the same convergence rate as obtained under stronger assumptions. In most cases, we ensure the same rate, albeit with inferior multiplicative factors due to the broader scope of the analysis. The notation  $\tilde{\mathcal{O}} (\cdot)$  hides a logarithmic factor of  $\log \frac{2\delta^0}{\varepsilon}$ .
Since Assumption 10 is more general than Assumption 11, Theorem 4 can be applied to functions that satisfy Assumption 11. If we set  $A = c = 0$ , we recover [Bottou et al., 2018, Theorem 4.6] (see Corollary 13 in the appendix). If  $A = C = c = 0$ , we retrieve results comparable to those in [Beznosikov et al., 2020, Theorems 12-14], up to a multiplicative factor (see Corollary 14 in the appendix). Due to  $\mu$ -strong convexity, our result (22) also implies an iterate convergence, since we have  $\left\| x^T - x^* \right\|^2 \leq \frac{2}{\mu} \mathbb{E}\left[f(x^T) - f(x^*)\right]$ . However, in this case an additional factor of  $\frac{2}{\mu}$  arises. Below we present a stronger result, yet, at a cost of imposing a stricter condition on the control variables from Assumption 9.
Assumption 12 Let  $A, B, C$  and  $b$  be parameters from Assumption 9. Let  $\mu$  be a strong convexity constant. Let  $L$  be a smoothness constant. Suppose  $A + L(B + 1 - 2b) < \mu$  holds.
Under Assumptions 9 and 12 we establish a similar result as the one obtained by Hu et al. [2021a, Theorem 1]. The authors impose a restriction of  $\frac{1}{\kappa}$  from above on a constant with an analogous role as  $B + 1 - 2b$  in Assumptions 9 and 12 with  $A = 0$ . However, unlike us, the authors consider only a finite sum case which makes our result more general. Moreover, only a biased version of SGD with a simple sampling strategy is analyzed by Hu et al. [2021a]. Our results are applicable to a larger variety of gradient estimators and obtained under milder assumptions. Also, our proof strategy is different, and much simpler.
Theorem 5 Let Assumptions 0, 9, 11 and 12 hold. For every positive  $s$ , satisfying  $A + L(B + 1 - 2b) < s < \mu$ , choose a stepsize  $\gamma$  such that
$$
0 <   \gamma \leq \min  \left\{\frac {1 - \frac {1}{s} (A + L (B + 1 - 2 b))}{A + L B}, \frac {1}{\mu - s} \right\}. \tag {23}
$$
Then the iterates of BiasedSGD (Algorithm 1) for every  $T \geq 1$  satisfy
$$
\mathbb {E} \left[ \left\| x ^ {T} - x ^ {*} \right\| ^ {2} \right] \leq (1 - \gamma (\mu - s)) ^ {T} \left\| x ^ {0} - x ^ {*} \right\| ^ {2} + \frac {\gamma C + \frac {C + 2 c}{s}}{\mu - s}. \tag {24}
$$
In the standard result for (unbiased) SGD, the convergence neighborhood term has the form of  $\frac{\gamma C}{\mu}$ , and it can be controlled by adjusting the stepsize. However, due to the generality of our analysis in the biased case, in (24) we obtain an extra uncontrollable neighborhood term of the form  $\frac{C + 2c}{s(\mu - s)}$ .
When  $A = C = c = 0$ ,  $B = 1$ ,  $b = 1$ ,  $s \to 0$ , we recover exactly the classical result for GD.
7 Experiments
Datasets, Hardware, and Code Implementation. The experiments utilized publicly available LibSVM datasets Chang and Lin [2011], specifically the splice, a9a, and w8a. These algorithms were developed using Python 3.8 and executed on a machine equipped with 48 cores of Intel(R) Xeon(R) Gold 6246 CPU @ 3.30GHz.
Experiment: Problem Setting. To validate our theoretical findings, we conducted a series of numerical experiments on a binary classification problem. Specifically, we employed logistic regression with a non-convex regularizer:
$$
\min  _ {x \in \mathbb {R} ^ {d}} \left[ f (x) \stackrel {\text {d e f}} {=} \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} (x) \right], \text {w h e r e} f _ {i} (x) \stackrel {\text {d e f}} {=} \log \left(1 + \exp \left(- y _ {i} a _ {i} ^ {\top} x\right)\right) + \lambda \sum_ {j = 1} ^ {d} \frac {x _ {j} ^ {2}}{1 + x _ {j} ^ {2}},
$$
and  $(a_{i},y_{i})\in \mathbb{R}^{d}\times \{-1,1\}$ $i = 1,\dots ,n$  represent the training data samples. In all experiments, we set the regularization parameter  $\lambda$  to a fixed value of  $\lambda = 1$ . We use datasets from the open LibSVM library [Chang and Lin, 2011]. We examine the performance of the proposed BiasedSGD method with biased independent sampling without replacement (we call it BiasedSGD-ind) in various settings (see Definition 1). The primary goal of these numerical experiments is to demonstrate the alignment of our theoretical findings with the observed experimental results. To assess the performance of the methods throughout the optimization process, we monitor the metric  $\| \nabla f(x^t)\|^2$ , recomputed after every 10 iterations. The algorithms are terminated after completing 5000 iterations. For each method, we use the largest theoretical stepsize. Specifically, for BiasedSGD-ind, the stepsize is determined according to Corollary 4 and Claim 2 with  $\gamma = \min \left\{\frac{1}{\sqrt{LAK}},\frac{b}{LB},\frac{c}{LC}\right\}$ , where  $c = 0$ ,  $A = \max_i\frac{L_i}{\min_i p_i}$ ,  $B = 0$ ,  $C = 2A\Delta^{*} + s^{2}$ ,  $b = \min_{i}p_{i}$  and  $s = 0$ .
More experimental details are provided in Appendix A.
Figure 2: The performance of BiasedSGD-ind with different choices of probabilities.
Experiment: The impact of the parameter  $p$  on the convergence behavior. In the first experiment, we investigate how the convergence of BiasedSGD-ind is affected as we increase the probabilities  $p_i$ , while keeping them equal for all data samples. According to the Corollary 4, larger  $p_i$  values (resulting in an increase of the expected batch size) allow for a larger stepsize, which, in turn, improves the overall convergence. This behavior is evident in Figure 2. The experiment visualized in Figure 2 involves varying the probability parameter  $p$  within the set  $\{0.01, 0.1, 0.5\}$ . This manipulation directly influences the value of  $A$ , consequently affecting the theoretical stepsize  $\gamma$ . In the context of BiasedSGD-ind, the stepsize  $\gamma$  is defined as  $\frac{1}{\sqrt{LAK}}$ . A comprehensive compilation of these parameters is represented in Table 7.
In this work, we consolidate various recent assumptions regarding the convergence of biasedSGD and elucidate their implication relationships. Moreover, we introduce a weaker assumption, referred to as Biased ABC. We also demonstrate that Biased ABC encompasses stochastic gradient oracles that previous assumptions excluded. With this assumption, we provide a proof of biasedSGD convergence across multiple scenarios, including strongly convex, non-convex, and under the PL-condition. Convergence rates that we obtain are the same up to a constant factor due to the broader setting and in some cases they coincide with the rates obtained under stricter assumptions. Furthermore, we examine the most widely used estimators in the literature related to SGD, represent them within the context of our Biased ABC framework, and analyze their compatibility with all previous frameworks.
This work of all authors was supported by the KAUST Baseline Research Scheme (KAUST BRF). The work of Y. Demidovich and P. Richtárik was supported by the KAUST Extreme Computing Research Center (KAUST ECRC), and the work of P. Richtárik was supported by the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).
Ahmad Ajalloeian and Sebastian U Stich. Analysis of SGD with biased gradient estimators. arXiv preprint arXiv:2008.00051, 2020.
Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 440-445, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-[]1045.
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cedric Renggli. The convergence of sparsified gradient methods. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pages 5973-5983, 2018.
El Houcine Bergou, Eduard Gorbunov, and Peter Richtárik. Stochastic three points method for unconstrained smooth minimization. SIAM Journal on Optimization, 30(4):2726-2749, 2020.
Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased compression for distributed learning. arXiv preprint arXiv:2002.12410, 2020.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223-311, 2018.
Soumia Boucherouite, Grigory Malinovsky, Peter Richtárik, and EL Bergou. Minibatch stochastic three points method for unconstrained smooth minimization. arXiv preprint arXiv:2209.07883, 2022.
Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):1-27, 2011.
Congliang Chen, Li Shen, Haozhi Huang, and Wei Liu. Quantized Adam with error feedback. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1-26, 2021a.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 15-26, 2017.
Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 25294-25307. Curran Associates, Inc., 2021b.
Tianyi Chen, Yuejiao Sun, and Wotao Yin. Solving stochastic compositional optimization is nearly as easy as solving stochastic optimization. IEEE Transactions on Signal Processing, 69:4937-4948, 2021c. doi: 10.1109/TSP.2021.3092377.
Tianyi Chen, Yuejiao Sun, and Wotao Yin. A single-timescale stochastic bilevel optimization method. arXiv preprint arXiv:2102.04671, 2021d.
Laurent Condat, Kai Yi, and Peter Richtárik. EF-BV: A unified theory of error feedback and variance reduction mechanisms for biased and unbiased compression in distributed optimization. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Jean-Baptiste Cordonnier. Convex optimization using sparsified stochastic gradient descent with memory. Technical report, 2018.