SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Plugging into the above bound on  $r$  tells us that  $x_r > x_0$  as long as  $r \geq \frac{L_1 M \left( f(x_0) - f^* - \frac{15\epsilon^2}{16 L_0} \right)}{2\epsilon^2 \left( 1 + \log \frac{L_1 M}{L_0} \right)}$ . The result follows from the fact that  $|f'(x_r)| = \epsilon$  for all  $x_r > x_0$ .
We can now combine Lemmas 13, 15, and 16 to prove Theorem 2.
Proof of Theorem 2. Let  $\gamma, \eta > 0$  be given. We will prove the requirement on  $R$  for three cases of  $\gamma, \eta$ . In the case  $\frac{\gamma}{\eta} \leq M$ , then Lemma 13 demonstrates that clipped minibatch SGD cannot guarantee to find an  $\epsilon$ -stationary point with probability greater than  $1 - \delta$  for any  $R$ . Second, if  $\frac{\gamma}{\eta} > M$  and  $\eta \geq \frac{2}{L_1M}\left(1 + \log \frac{L_1M}{L_0}\right)$ , then Lemma 15 demonstrates that clipped minibatch SGD can fail to find an  $\epsilon$ -stationary point for any  $R$ . In the remaining case, i.e.  $\frac{\gamma}{\eta} > M$  and  $\eta < \frac{2}{L_1M}\left(1 + \log \frac{L_1M}{L_0}\right)$ , Lemma 16 demonstrates that for finding an  $\epsilon$ -stationary point, the number of iterations clipped minibatch SGD requires is at least
$$
R \geq \frac {L _ {1} M \left(f (x _ {0}) - f ^ {*} - \frac {1 5 \epsilon^ {2}}{1 6 L _ {0}}\right)}{2 \epsilon^ {2} \left(1 + \log \frac {L _ {1} M}{L _ {0}}\right)}.
$$
C.1 Hyperparameter Information
Learning Rate and Clipping Threshold For each dataset, we tune the hyperparameters  $\gamma$  and  $\eta$  with separate grid searches. Specifically, we first tune the clipping threshold  $\frac{\gamma}{\eta}$  with grid search, then we tune the learning rate  $\eta$  with grid search. During the grid search for  $\eta$ ,  $\gamma$  is chosen so that the clipping threshold  $\frac{\gamma}{\eta}$  is equal to the tuned value from the search for  $\frac{\gamma}{\eta}$ . Due to computational constraints, we do not perform this search for every algorithm in every setting. Instead, we follow the tuning process of [9]: we use the above procedure to tune  $\eta$  and  $\gamma$  for CELGC [32] separately for each  $S \in \{2, 4, 6, 8\}$ , while fixing  $s = 30\%$  (SNLI) or  $s = 10\%$  (Sent140). We then reuse the tuned values of  $\gamma$  and  $\eta$  for all other algorithms and settings sharing the same dataset and  $S$ . Our theory suggests that the best learning rate  $\eta$  depends on the number of participating clients  $S$ , and we follow this guidance by tuning the parameters separately for each value of  $S$ . The search ranges and tuned values for each dataset are shown in Table 3.
Table 3: Hyperparameter tuning information for each dataset.
Parameter	Search range	Tuned value
SNLI	γ/η	{0.01, 0.03, 0.1, 0.3, 1.0, 3.0}	S = 2: 1.0
S = 4: 1.0
S = 6: 1.0
S = 8: 1.0
S = 2: 0.03
η	{0.003, 0.01, 0.03, 0.1}	S = 4: 0.03
S = 6: 0.03
S = 8: 0.03
Sent140	γ/η	{0.01, 0.03, 0.1, 0.3, 1.0, 3.0}	S = 2: 0.3
S = 4: 0.3
S = 6: 0.3
S = 8: 0.3
S = 2: 0.03
η	{0.003, 0.01, 0.03, 0.1}	S = 4: 0.03
S = 6: 0.03
S = 8: 0.03
Network Architecture For both datasets, the network is composed of a one-layer bidirectional RNN encoder followed by a three-layer fully connected classifier. The encoder has hidden size 2048 and max pooling, and the decoder has hidden size 512 with tanh activations. Input sentences are encoded as sequences of GloVe vectors. For SNLI, where each input is a pair of sentences, each sentence is passed through the encoder separately, and the resulting representations are concatenated and used as input for the encoder.
C.2 Learning Curves
Figures 4 and 5 contain learning curves (by communication rounds) of training loss and testing accuracy for all settings, for SNLI and Sentiment140 respectively. The experiments show that our proposed algorithm EPISODE++ significantly outperform all other baselines in various settings on these two datasets, including different client participation ratio and different heterogeneity level.
C.3 Training with Homogeneous Data
Here we include learning curves for an additional experiment that uses homogeneous data across clients, i.e.  $s = 0\%$ . In this setting, we use  $N = 8$  and  $S = 4$ , and all other settings are the same as described in Section 6. We compare the six algorithms described in Section 6. Results are shown in Figure 6. The results are consistent with the experiments that use heterogeneous data: EPISODE++ outperforms all other algorithms in terms of training loss and testing accuracy.
Figure 4: Learning curves for SNLI under all settings. For NaiveParallelClip, we show the first 5375 rounds to compare all algorithms with a fixed number of communication rounds.
Figure 5: Learning curves for Sentiment140 under all settings. For NaiveParallelClip, we show the first 2000 rounds to compare all algorithms with a fixed number of communication rounds.
Figure 6: Learning curves for SNLI and Sentiment140 with  $N = 8$ ,  $S = 4$ , using homogeneous data, i.e.  $s = 100\%$ . We compare all algorithms with a fixed number of communication rounds.