SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dwivedi and Mackey minimize the maximum mean discrepancy (MMD) between a selected coreset and an empirical distribution. They develop a new self-balancing Hilbert walk on differences of pairs of examples to select exactly half of the dataset points, and solve coreset selection by iteratively halving the input vector sequence into balanced coresets then selecting and refining a candidate coreset to minimize MMD with the input sequence.
Algorithm 2 PairBalance
$\triangleright$  The inputs, outputs and subroutine for this algorithm are order-sensitive
input: current running sum  $r$ , paired vectors  $z_{1}, z_{2}$
compute:  $s,r\gets$  RandomizedBalance  $(r,z_{1} - z_{2})$
return:  $s$  (sign for  $\pmb{z}_1$ )
$-s$  (sign for  $z_{2}$ )
$\pmb{r}$  (updated running sum)
Adapted from Alweiss et al. 2
define subroutine: RandomizedBalance  $(r,c)$
compute:  $p \leftarrow \frac{1 - \langle \mathbf{r}, \mathbf{c} \rangle}{2}$
compute:  $s\gets +1$  with probability  $p$
$s\gets -1$  with probability  $1 - p$
update:  $r\gets r + sc$
return:  $s, r$
Figure 1: Left: The PairBalance algorithm, which the server runs online. Right: CD-GraB running on one server (top) and two workers (bottom). The workers do not share data examples.
We describe CD-GraB at two levels of abstraction: a high-level illustration (Figure 1), steps 1-7) and a detailed pair of worker-server algorithm statements (Figure 2). Since the workers only have access to a subset of the training data, in parallel they compute local, per-example stochastic gradients and send them to the server. The server simultaneously calls PairBalance online, which coordinates information from all the workers' gradients (i.e., using adjacent example-specific gradients) to determine the next epoch's worker-specific permutations. In more detail:
In epoch  $t$ , (Figure 1 step 1) the two workers have permutations  $\pi_{t,1}$  and  $\pi_{t,2}$ , respectively. Each worker computes per-example gradients  $g_j^i$  (2; Algorithm 34), and sends them to the server (3; Algorithm 35). The server we implement functions as a parameter server [20]: It computes the average of the workers' per-example gradients (Algorithm 46), and sends it back to all workers (Algorithm 47) so that they can update their local models (Algorithm 36-7). Simultaneously, as the server receives gradients (Algorithm 45), it calls PairBalance (Algorithm 2 on adjacent vectors (4; Algorithm 44-13). PairBalance produces signs to supply to the reordering algorithm (Algorithm 1), which, using the current worker permutations  $\pi_{t,i}$ , produces the new per-worker permutations for the next epoch (5; Algorithm 414). In Figure 1, these correspond to  $\pi_{t+1,1}$  and  $\pi_{t+1,2}$ , which the server then sends back to the respective workers (6; Algorithm 415). Lastly, before the start of the next epoch, the workers reorder their examples according to the new permutations (7; Algorithm 39).
We next demonstrate formally that our CD-GraB algorithm (Section 3.3) confers the efficiency benefits of centralized GraB (Section 2.1) to the distributed setting. In brief, our main theoretical results show that CD-GraB enjoys a linear speedup in convergence rate under two sets of conditions: smoothness (Theorem 2) and the Polyak-Lojasiewicz (P.L.) condition (Theorem 3). Both results guarantee that CD-GraB is faster than distributed random reshuffling (D-RR). Our proofs rely on Corollary 7 from Dwivedi and Mackey [10], which shows that, with high probability, RandomizedBalance (subroutine in Algorithm 2 from Alweiss et al. 2]) guarantees a  $\tilde{O}(1)$  bound to the signed herding objective (4).
To begin, we restate this result to cohere with our framework, for which the vectors  $\mathbf{z}_j$  are gradients in an optimization context:
Theorem 1 (Corollary 7, Dwivedi and Mackey [10]). Consider any vectors  $\{\pmb{z}_j\}_{j=1}^N$  ( $\pmb{z}_j \in \mathbb{R}^d$ ) with  $\| \pmb{z}_j \|_2 \leq 1$  supplied as input to the RandomizedBalance subroutine in Algorithm [2]. Then for any  $\delta > 0$  with probability at least  $1 - \delta$ , RandomizedBalance outputs a sequence of signs  $\{s_j\}_{j=1}^N \in \{-1,1\}$  that satisfy  $\max_{k \in [N]} \left\| \sum_{j=1}^k s_j z_j \right\|_\infty \leq \tilde{A}$ , where  $\tilde{A} = \sqrt{2 \log \left( \frac{4d}{\delta} \right) \log \left( \frac{4N}{\delta} \right)} = \tilde{O}(1)$ .
9Corollary 7 from Dwivedi and Mackey improves the result of Theorem 1.1 from Alweiss et al. [2].
Algorithm 3 CD-GraB Workers
require:  $m$  workers,  $n\coloneqq \frac{N}{m}$  ex. per worker  $\{\pi_{1,i}\}_{i = 1}^{m}$  input: initial  $\boldsymbol {w}_1^1$  , epochs  $T$  , learning rate  $\alpha$  require:  $m$  workers,  $n\coloneqq \frac{N}{m}$  ex. per worker input: epochs  $T$    
1: receive: initial permutations  $\leftarrow \left\{\pi_{1,i}\right\}_{i = 1}^{m}$  1: send: initial permutations  $\{\pi_{1,i}\}_{i = 1}^{m}$    
2: for epoch  $t\coloneqq 1\dots T$  do 2: for epoch  $t\coloneqq 1\dots T$  do   
3:  $\triangleright$  Run in parallel for workers  $i = 1\dots m$  3: initialize: running sum  $h = 0$  ; empty list S   
4: for example  $j\coloneqq 1\dots n$  do 4: for example  $j\coloneqq 1\dots n$  do   
5: send:  $\pmb {g}_j^i\xrightarrow{j\text{-th stochastic grad.}}\pmb {g}_j^i$  5: receive:  $\{g_j^i\}_{i = 1}^m$  from all workers i   
6: receive:  $\bar{g}_j\gets \frac{\mathrm{avg.~}j\text{-th stochastic grad.~}\bar{g}_j}{\mathrm{avg.~}j\text{-th stochastic grad.~}\bar{g}_j}$  6: compute: avg. gradient:  $\bar{g}_j\gets \frac{1}{m}\sum_{i = 1}^{m}\pmb{g}_j^i$    
7: update:  $\pmb {w}_t^{j + 1}\gets \pmb {w}_t^j -\alpha \bar{g}_j$  8: for worker  $i\coloneqq 1\dots m$  do 9: if  $j\bmod 2 = 0$    
8: end for 10: h,s-s-s-s- PairBalance(h,g-g-1,g)   
11: S.append(s)-; S.append(s)   
12: end for   
13: end for   
14: Call Alg. [for  $i = 1\dots m$  on  $\pi_{t,i}$  and S   
15: compute: next permutations  $\{\pi_{t + 1,i}\}_{i = 1}^{m}$    
16: send:  $\{\pi_{t + 1,i}\}_{i = 1}^{m}$  to each worker i   
17: end for   
end for   
return: w+1:=w+1
Figure 2: CD-GraB worker and server (here, a parameter server [20]) algorithms.
To integrate this result with our parallel setting, we need some additional assumptions that are standard in the literature on distributed optimization — that the variance of the per-example gradients on each worker is uniformly bounded (Assumption 1), and that the variance between worker-specific gradients is similarly bounded (Assumption 2). More precisely, following the distributed setup in (7), we denote the global loss gradient to be  $\nabla f(\boldsymbol{w})$ , each  $i$ -th worker's local loss gradient to be  $\nabla f^i(\boldsymbol{w})$  ( $\forall i \in [m]$ ), and each  $i$ -th worker's per-example loss gradients to be  $\nabla f^i(\boldsymbol{w}; j)$  ( $\forall j \in [n]$ ). We assume:
Assumption 1 (Bounded Gradient Variance). For all  $i \in [m]$  there exists a constant  $\sigma > 0$  such that for all  $j \in [n]$  and for all  $\mathbf{w} \in \mathbb{R}^d$ , it holds that  $\left\| \nabla f^i(\mathbf{w}; j) - \nabla f^i(\mathbf{w}) \right\|_2^2 \leq \sigma^2$ .
Assumption 2 (Bounded Data Heterogeneity). There exists a constant  $\varsigma >0$  such that  $\forall i\in [m]$ $\left\| \nabla f^{i}(\boldsymbol {w}) - \nabla f(\boldsymbol {w})\right\|_{2}^{2}\leq \varsigma^{2}$
Lastly, we include one additional assumption from the original GraB paper [24]: we assume a cross norm  $L_{2,\infty}$  (which can be easily adapted to  $L_{2}$ -smoothness by setting  $L_{2,\infty}$  to be  $\sqrt{d} L_{2}$ ).
Assumption 3 (Smoothness). There exists constant  $L_{2,\infty} > 0$  such that for any  $\pmb{w},\pmb{v}\in \mathbb{R}^d$ , any  $i\in [m]$  and any  $j\in [n]$ , it holds that  $\left\| \nabla f^{i}(\pmb {w};j) - \nabla f^{i}(\pmb {v};j)\right\|_{2}\leq L_{2,\infty}\| \pmb {w} - \pmb {v}\|_{\infty}$ .
Given these assumptions, we can prove a convergence guarantee for CD-GraB:
Theorem 2. Suppose that Assumptions [12] and [3] hold. For any  $\delta >0$ , if we set learning rate  $\alpha$  to be
$$
\alpha = \min  \left\{\frac {1}{1 6 L _ {2 , \infty} (2 n + \tilde {A} / m)}, \overline {{\left(\frac {4 F _ {1} m ^ {2}}{4 2 L _ {2 , \infty} ^ {2} (\varsigma + \sigma) ^ {2} \tilde {A} ^ {2} n T + 1 8 L _ {2 , \infty} ^ {2} m ^ {2} n ^ {3} \sigma^ {2}}\right) ^ {1 / 3}}} \right\},
$$
where  $F_{1} = f(\pmb{w}_{1}) - \inf_{\pmb{w}\in \mathbb{R}^{d}}f(\pmb{w})$  and  $\tilde{A}$  comes from Theorem 1. Then, with probability at least  $1 - T\delta$ ,
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (\boldsymbol {w} _ {t}) \| _ {2} ^ {2} \leq \frac {9 (F _ {1} L _ {2 , \infty} (\varsigma + \sigma) \tilde {A}) ^ {2 / 3}}{(m n T) ^ {2 / 3}} + \frac {(7 2 F _ {1} L _ {2 , \infty} \sigma) ^ {2 / 3} + 6 4 F _ {1} L _ {2 , \infty} (2 + \tilde {A} / (m n))}{T} \\ = \tilde {O} \left(\frac {1}{(m n T) ^ {2 / 3}} + \frac {1}{T}\right). \\ \end{array}
$$
We can also prove an accelerated rate for CD-GraB if we additionally assume the P.L. condition:
Assumption 4 (P.L. Condition). We say the loss function  $f$  fulfills the P.L. condition if there exists  $\mu > 0$  such that for any  $\boldsymbol{w} \in \mathbb{R}^d$ ,  $\frac{1}{2} \| \nabla f(\boldsymbol{w}) \|_2^2 \geq \mu (f(\boldsymbol{w}) - \inf_{\boldsymbol{v} \in \mathbb{R}^d} f(\boldsymbol{v}))$ .
Theorem 3. Suppose that Assumptions  $\boxed{1}$ $\boxed{2}$ $\boxed{3}$  and  $\boxed{4}$  hold. For any  $\delta > 0$ , we set constants  $\tilde{W}$  and  $C_3$  to be
$$
C _ {3} = \frac {\left(F _ {1} + \sigma^ {2} / L _ {2 , \infty}\right) \mu^ {2}}{2 2 4 L _ {2 , \infty} ^ {2} (\varsigma + \sigma) ^ {2} \tilde {A} ^ {2}} \quad a n d \quad \tilde {W} = W _ {0} (T ^ {2} m ^ {2} n ^ {2} C _ {3}),
$$
where  $\tilde{A}$  comes from Theorem  $F_{1}$  and  $W_{0}$  is the Lambert-W function. If we set learning rate  $\alpha = \frac{2\tilde{W}}{Tn\mu}$  and if the number of epochs  $T$  satisfies
$$
T \geq 1 0 + \frac {1}{\mu} 3 2 L _ {2, \infty} (2 + \tilde {A} / (m n)) W _ {0} ((m n T) ^ {2} C _ {3}) = \tilde {O} (1),
$$
then, with probability at least  $1 - T\delta$ , it holds that
$$
F _ {T + 1} \leq \frac {1}{(m n T) ^ {2}} \left(\frac {\left(F _ {1} + L _ {2 , \infty} ^ {2} \sigma^ {2}\right) \tilde {W}}{C _ {3}} + \frac {1 1 2 L _ {2 , \infty} ^ {2} (\varsigma + \sigma) ^ {2} \tilde {A} ^ {2} \tilde {W} ^ {2}}{\mu^ {3}}\right) = \tilde {O} \left(\frac {1}{(m n T) ^ {2}}\right),
$$
where  $F_{T + 1} = f(\pmb{w}_{T + 1}) - \inf_{\pmb{w}\in \mathbb{R}^d}f(\pmb {w})$
We prove Theorems 2 and 3 in the Appendix. Together, they show that CD-GraB exhibits a linear speedup in the number of workers  $m$  over GraB [24]s convergence rates  $(\tilde{O}((nT)^{-2/3})$  and  $\tilde{O}((nT)^{-2})$ , respectively) under both smoothness and the P.L. condition. Further, CD-GraB's convergence rate of  $\tilde{O}((mnT)^{-2})$  is faster than many previous rates such as the high probability bound of  $\tilde{O}((mn)^{-1}T^{-2})$  for D-RR in Yun et al. [48].
5 CD-GraB in Practice: Distributed and Simulation Experiments
We next verify CD-GraB's accelerated convergence on a variety of empirical tasks. For ease of comparison, we follow the experimental plan from the original GraB paper and add some additional large-scale logistic regression experiments. We also run an ablation study to isolate the effects of different improvements in CD-GraB. We do this because online PairBalance exhibits performance benefits that are separate from parallelism — namely, removing the need for gradient centering with a stale mean and allowing for higher learning rates (Section 3.2).
Evaluating CD-GraB's convergence speedup. We use the following three tasks for evaluating distributed training efficiency: logistic regression on a large-scale mortgage application (New York 2017 subset, 244,107 examples with 18 features) [7] (Figure 3a), Long Short-Term Memory (LSTM) [17] on the WikiText-2 dataset [29] (Figure 3b), and autoregressive Multi-Layer Perceptron (MLP) on the M4 Weekly dataset [25] (Figure 3c). We measure the loss incurred on the entire training set (Full Train Loss) and task-appropriate test metrics during evaluation, with respect to both the number of epochs and wall-clock time. Regarding test metrics, we measure test accuracy for the mortgage application, perplexity for WikiText-2, and SMAPE for M4. Additional details regarding the datasets, models, and test metrics can be found in the Appendix.
For all three tasks, we use a single 128 GiB memory machine with 4 NVIDIA GeForce RTX 2080 Ti GPUs. For the mortgage application and WikiText-2 (Figures  $3a$  and  $3b$ ), we launch  $m = 4$  workers (processes), where each worker runs on one GPU. For the M4 task, we launch  $m = 32$  workers, where each of the 4 GPUs hosts 8 process workers. We use NCCL as the distributed communication backend [33] for the mortgage application and WikiText-2 tasks, and GLOO [1] as the distributed communication backend for the M4 task.
As shown in Figure3 we compare CD-GraB's convergence to the standard distributed-training example-ordering method: random reshuffling (D-RR). From all subfigures in Figure3 we observe that CD-GraB outperforms the D-RR baseline significantly and consistently: CD-GraB exhibits better training loss and test metrics, measured against both the number of epochs and wall-clock time. We also note that the
10For centralized GraB, the total number of examples  $N = n$  and  $m = 1$
These exclusively focus on the P.L. case, so we compare CD-GraB to them under the same condition.
<sup>12</sup>Our GitHub repository is https://github.com/GarlGuo/CD-GraB
13Following Lu et al. [24], for our LSTM experiment on WikiText-2, we set the embedding dimension to 32. We note that we can improve perplexity if we set the dimension higher.
$^{14}$ GraB can also implement online PairBalance, in place of Balance [22] (Appendix).
(a) Logistic regression on mortgage application (NY 2017 subset) [7]
(b) LSTM [17] on WikiText-2 [29]
Figure 3: Convergence of CD-GraB in comparison to D-RR. For each experiment, we show train loss over epochs and time (left of each subfigure) and test performance over epochs and time (right of each subfigure). We run at least 3 random seeds, and plot the mean  $\pm$  STD.
(c) Autoregressive MLP on M4 Weekly [25]
results for CD-GraB are much smoother than for D-RR. This is likely due to the variance of stochastic gradients during training, which CD-GraB reduces as a side-effect (so, too, does GraB, in comparison to RR). For smoother D-RR results, we can reduce the learning rate (Appendix). CD-GraB allows for the use of a larger learning rate, which accelerates training while preserving the final model's performance.
Ablation simulation study: the importance of coordination at large scale. CD-GraB has several design benefits over the original centralized GraB algorithm [24]: coordinating parallel workers' specific permutations using PairBalance on the server (Algorithm 2) and removing the dependency on a stale mean (Section 2.1), which enables the ability to using larger learning rates reliably (Section 3.2). Clearly, not all of these benefits come directly from distributing training. For example, being able to use larger learning rates, is a side effect of our solution to develop CD-GraB, not our main contribution. Therefore, we run a simulation ablation study to disentangle the relative importance of each of CD-GraB's efficiency benefits over GraB. To do so, we compare the convergence of CD-GraB to two additional baselines in the distributed setting, beyond D-RR: (1) ID-GraB (Bal), where each independent worker runs GraB locally using RandomizedBalance (subroutine in Algorithm 2) to perform gradient vector balancing; (2) ID-GraB (PairBal), where each independent worker runs GraB locally using PairBalance.
Figure4 summarizes the results, with convergence curves for  $m \in \{4,8,16,32,64\}$  workers training LeNet on CIFAR-10. We choose this task and architecture to cohere with the experiments done in the original GraB paper. For these experiments, we denote  $B$  to be the aggregated minibatch across all the workers, which refers to the number of stochastic examples used for an overall optimization step; each worker thus has a subset of this minibatch — an equivalently-sized subset of  $B$  examples[15]We make two main observations. First, when scaling up training with more workers, CD-GraB converges increasingly faster than the no-coordination-ordering methods ID-GraB (Bal) and ID-GraB (PairBal). This result aligns with our theory and intuition that, when the number of workers  $m$  increases, the parallel herding bound (8) will increase linearly if there is no coordination. Second, as we scale up to larger  $m$ , the convergence curves of ID-GraB (Bal) and ID-GraB (PairBal) gradually approach
For example, if we have 4 workers with an aggregated minibatch size of 32, each worker would compute their respective local gradients with 8 examples, and then all-reduce these gradients to obtain the aggregated minibatch gradient for all 32 examples for the optimization step. We discard  $N$  mod  $B$  examples at random to ensure  $n$  examples per worker.
Figure 4: Convergence for CD-GraB, D-RR, ID-GraB (Bal), and ID-GraB (PairBal) training LeNet on CIFAR-10, with  $m \in \{4,8,16,32,64\}$  workers. For each, the aggregated minibatch size per update is 64.
the curve for D-RR: at larger scales, herding-based example ordering will be no better than randomly permuting the dataset. Both observations give strong evidence that coordination (i.e., running online PairBalance on the server to coordinate per-worker permutations) is critical for accelerating training.
We note that all of these experiments use SGD, since both the theoretical results of the original GraB paper and our results for CD-GraB here are for SGD. In the Appendix, we additionally include results for training GPT-2 on WikiText-103, for which we use AdamW as the optimizer. We find that GraB with AdamW works in practice; however, our theory results do not directly apply to these experiments. We additionally include results on memory usage in the Appendix, which show that CD-GraB results in negligible overhead in practice.
6 Conclusion and Future Work: Toward an Order Server Architecture
We elevate the benefits of provably faster, permutation-based example ordering to the contemporary ML distributed-training setting. We focus on reformulating the online Gradient Balancing algorithm (GraB) [24] because, even though it is the provably optimal permutation-based example-ordering method [6], it is limited by design to centralized settings (Section 3.1). To overcome these limitations, we redesign GraB's herding and balancing framework to account for parallel workers: A parallel herding objective, which we solve with an online PairBalance subroutine, based on key insights from kernel thinning [3][10][11]. PairBalance operates on ordered pairs of vectors to do balancing, which enables our full-stack, low-overhead, Coordinated and Distributed online CD-GraB algorithm. We give a full specification of our online CD-GraB algorithm (Section 3.3), provide convergence rate guarantees regarding its speedups on both 1) smooth non-convex and 2) P.L. objectives (Section 4), and verify these speedups in practice on single-node distributed tasks and a simulated ablation study (Section 5).
Both our theory and experiments demonstrate that CD-GraB really shines when there are multiple training epochs (Appendix). This is another reason that we do not emphasize experiments involving fine-tuning pre-trained models like GPT-2, as fine-tuning can be achieved in just a couple of epochs. As noted above, it is also more common to train such models using optimizers from the Adam family. In future work, we intend to extend the theory on GraB and CD-GraB to such optimizers, which would make the results on optimal, permutation-based example ordering more useful for base-model pre-training.
Pre-training from scratch would demonstrate the tremendous power of CD-GraB to scale to very large models; however, we did not have the training budget to perform such experiments for the present work. Further, to truly exercise the benefits of CD-GraB in such large-scale settings, future work should investigate moving beyond the single-node setup that we present. Notably, to train larger models, our results suggest a novel distributed training architecture. The ordering operation performed by the server (Algorithm 4) is not very latency sensitive; the server has the duration of the entire epoch  $t$  to compute the new permutations for the next,  $t + 1$  epoch. Given this relaxed latency requirement, and the success of our algorithmic results, it would be an exciting direction for future ML-systems research to invest in building an Order Server architecture. Such an architecture, which could be composed with traditional parameter servers, would afford the scalability benefits of CD-GraB to a host of massive-scale ML applications.
A. Feder Cooper is supported by Christopher De Sa's NSF CAREER grant. Yucheng Lu is supported by Meta Ph.D. Fellowship. We also acknowledge a gift from SambaNova Systems.
[39] Mark Schmidt, Nicolas Le Roux, and Francis R. Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162(1-2):83-112, 2017.
[40] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't Decay the Learning Rate, Increase the Batch Size. In International Conference on Learning Representations, 2018.
[41] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, pages 1-40, 2022.
[42] Evangelos Spiliotis, Andreas Kouloumos, Vassilios Assimakopoulos, and Spyros Makridakis. Are forecasting competitions data representative of the reality? International Journal of Forecasting, 36(1):37-53, 2020.
[43] Merity Stephen, Xiong Caiming, Bradbury James, and Richard Socher. Pointer sentinel mixture models. Proceedings of ICLR, 2017.
[44] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5446. URL https://aclanthology.org/W18-5446
[45] Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1121-1128, 2009.
[46] Bicheng Ying, Kun Yuan, Stefan Vlaski, and Ali H. Sayed. On the performance of random reshuffling in stochastic learning. In 2017 Information Theory and Applications Workshop (ITA), pages 1-5. IEEE, 2017.
[47] Binhang Yuan, Yongjun He, Jared Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy S Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 35:25464-25477, 2022.
[48] Chulhee Yun, Shashank Rajput, and Suvrit Sra. Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond. In International Conference on Learning Representations, 2021.
[49] Chulhee Yun, Suvrit Sra, and Ali Jabbabaie. Open Problem: Can Single-Shuffle SGD be Better than Reshuffling SGD and GD? In Conference on Learning Theory, 2021.