SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ F (\mathbf {x} + \Delta \mathbf {x}) - F (\mathbf {x}) \right] \leq - \frac {K \eta}{2} \| \nabla F (\mathbf {x}) \| ^ {2} + \frac {L K \eta^ {2} \sigma^ {2}}{S} + \frac {L ^ {2} \eta}{2 S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \\ - \frac {\eta}{2 S} (1 - 2 L K \eta) \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} \right] \\ \end{array}
$$
Since  $\eta \leq \frac{1}{6LK}$ , the last term on the right hand side in the preceding inequality is negative. Then
$$
\mathbb {E} \left[ F (\mathbf {x} + \Delta \mathbf {x}) - F (\mathbf {x}) \right] \leq - \frac {K \eta}{2} \| \nabla F (\mathbf {x}) \| ^ {2} + \frac {L K \eta^ {2} \sigma^ {2}}{S} + \frac {L ^ {2} \eta}{2 S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right]
$$
E.3.1 Bounding the client drift with Assumption 3a
Lemma 14. Let Assumptions 1, 2, 3b hold. If the learning rate satisfies  $\eta \leq \frac{1}{6LK}$ , then the client drift is bounded:
$$
\mathcal {E} _ {r} \leq \frac {9}{4} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {3}{2} K ^ {3} \eta^ {2} \zeta^ {2} + \frac {3}{2} K ^ {3} \eta^ {2} (\beta^ {2} + 1) \mathbb {E} \left[ \left\| \nabla F \left(\mathbf {x} ^ {(r)}\right) \right\| ^ {2} \right] \tag {79}
$$
We use Jensen's inequality to bound the term  $\mathbb{E}\left[\| \mathbf{x}_{m,k} - \mathbf{x}\| ^2\right] = \eta^2\mathbb{E}\left[\left\| \sum_{j = 0}^{k - 1}\mathbf{g}_{\pi_m,j}\right\| ^2\right]$ :
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \leq 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} (\mathbf {g} _ {\pi_ {m}, j} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j})) \right\| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} (\nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j}) - \nabla F _ {\pi_ {m}} (\mathbf {x})) \right\| ^ {2} \right] \\ + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F (\mathbf {x})\right) \right\| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} \nabla F (\mathbf {x}) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \leq 4 \eta^ {2} \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {g} _ {\pi_ {m}, j} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j}) \| ^ {2} \right] + 4 k \eta^ {2} \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j}) - \nabla F _ {\pi_ {m}} (\mathbf {x}) \| ^ {2} \right] \\ + 4 k ^ {2} \eta^ {2} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F (\mathbf {x}) \| ^ {2} \right] + 4 k ^ {2} \eta^ {2} \mathbb {E} \left[ \| \nabla F (\mathbf {x}) \| ^ {2} \right] \\ \end{array}
$$
The first term can be bounded by  $4k\eta^2\sigma^2$  with Assumption 2. The second term can be bounded by  $4L^{2}k\eta^{2}\sum_{j = 0}^{k - 1}\mathbb{E}\left[\| \mathbf{x}_{m,j} - \mathbf{x}\| ^2\right]$  with Assumption 1. The third term can be bounded by  $4k^{2}\eta^{2}\left(\beta^{2}\| \nabla F(\mathbf{x})\|^{2} + \zeta^{2}\right)$  with Assumption 3a. Thus, we have
$$
\mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \leq 4 k \eta^ {2} \sigma^ {2} + 4 L ^ {2} k \eta^ {2} \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, j} - \mathbf {x} \| ^ {2} \right] + 4 k ^ {2} \eta^ {2} (\beta^ {2} + 1) \| \nabla F (\mathbf {x}) \| ^ {2} + 4 k ^ {2} \eta^ {2} \zeta^ {2}
$$
Then turning to  $\mathcal{E}_r\coloneqq \frac{1}{S}\sum_{m = 1}^{S}\sum_{k = 0}^{K - 1}\mathbb{E}\left[\| \mathbf{x}_{m,k} - \mathbf{x}\| ^2\right]$ , we have
$$
\mathcal {E} _ {r} = 4 \eta^ {2} \sigma^ {2} \sum_ {k = 0} ^ {K - 1} k + 4 L ^ {2} \eta^ {2} \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} k \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, j} - \mathbf {x} \| ^ {2} \right] + 4 \eta^ {2} (\beta^ {2} + 1) \| \nabla F (\mathbf {x}) \| ^ {2} \sum_ {k = 0} ^ {K - 1} k ^ {2} + 4 \eta^ {2} \zeta^ {2} \sum_ {k = 0} ^ {K - 1} k ^ {2}
$$
$$
\mathcal {E} _ {r} \leq 2 K ^ {2} \eta^ {2} \sigma^ {2} + 2 L ^ {2} K ^ {2} \eta^ {2} \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, j} - \mathbf {x} \| ^ {2} \right] + \frac {4}{3} K ^ {3} \eta^ {2} (\beta^ {2} + 1) \| \nabla F (\mathbf {x}) \| ^ {2} + \frac {4}{3} K ^ {3} \eta^ {2} \zeta^ {2}
$$
$$
(1 - 2 L ^ {2} K ^ {2} \eta^ {2}) \mathcal {E} _ {r} \leq 2 K ^ {2} \eta^ {2} \sigma^ {2} + \frac {4}{3} K ^ {3} \eta^ {2} \zeta^ {2} + \frac {4}{3} K ^ {3} \eta^ {2} (\beta^ {2} + 1) \| \nabla F (\mathbf {x}) \| ^ {2}
$$
$$
\mathcal {E} _ {r} \leq \frac {9}{4} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {3}{2} K ^ {3} \eta^ {2} \zeta^ {2} + \frac {3}{2} K ^ {3} \eta^ {2} (\beta^ {2} + 1) \| \nabla F (\mathbf {x}) \| ^ {2}
$$
E.3.2 Proof of non-convex case of Theorem 2
Proof of non-convex case of Theorem 2. Substituting (52) into (48) and using  $\eta \leq \frac{1}{6LK(\beta + 1)}$ , we can simplify the recursion as follows:
$$
\mathbb {E} \left[ F (\mathbf {x} ^ {(r + 1)}) - F (\mathbf {x} ^ {(r)}) \right] \leq - \frac {1}{3} K \eta \mathbb {E} \left[ \left\| \nabla F (\mathbf {x} ^ {(r)}) \right\| ^ {2} \right] + \frac {L K \eta^ {2} \sigma^ {2}}{S} + \frac {9}{8} L ^ {2} K ^ {2} \eta^ {3} \sigma^ {2} + \frac {3}{4} L ^ {2} K ^ {3} \eta^ {3} \zeta^ {2}
$$
Letting  $\tilde{\eta} \coloneqq K\eta$ , subtracting  $F^{*}$  from both sides and then rearranging the terms, we have
$$
\mathbb {E} \left[ F (\mathbf {x} ^ {(r + 1)}) - F ^ {*} \right] \leq \mathbb {E} \left[ F (\mathbf {x} ^ {(r)}) - F ^ {*} \right] - \frac {\tilde {\eta}}{3} \mathbb {E} \left[ \left\| \nabla F (\mathbf {x} ^ {(r)}) \right\| ^ {2} \right] + \frac {L \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {9 L ^ {2} \tilde {\eta} ^ {3} \sigma^ {2}}{8 K} + \frac {3}{4} L ^ {2} \tilde {\eta} ^ {3} \zeta^ {2}
$$
Then applying Lemma 8 with  $t = r$  ( $T = R$ ),  $\gamma = \tilde{\eta}$ ,  $r_t = \mathbb{E}\left[F(\mathbf{x}^{(r)}) - F^*\right]$ ,  $b = \frac{1}{3}$ ,  $s_t = \mathbb{E}\left[\|\nabla F(\mathbf{x}^{(r)})\|^2\right]$ ,  $w_t = 1$ ,  $c_1 = \frac{L\sigma^2}{SK}$ ,  $c_2 = \frac{9L^2\sigma^2}{8K} + \frac{3}{4}L^2\zeta^2$  and  $\frac{1}{d} = \frac{1}{6L(\beta + 1)}$  ( $\tilde{\eta} = K\eta \leq \frac{1}{6L(\beta + 1)}$ ), we have
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \left\| \nabla F \left(\mathbf {x} ^ {(r)}\right) \right\| ^ {2} \right] \leq \frac {3 (F \left(\mathbf {x} ^ {0}\right) - F ^ {*})}{\tilde {\eta} R} + \frac {3 L \tilde {\eta} \sigma^ {2}}{S K} + \frac {2 7 L ^ {2} \tilde {\eta} ^ {2} \sigma^ {2}}{8 K} + \frac {9}{4} L ^ {2} \tilde {\eta} ^ {2} \zeta^ {2} \tag {80}
$$
where we use  $\min_{0\leq r\leq R}\mathbb{E}\left[\left\| \nabla F(\mathbf{x}^{(r)})\right\| ^2\right]\leq \frac{1}{R + 1}\sum_{r = 0}^{R}\mathbb{E}\left[\left\| \nabla F(\mathbf{x}^{(r)})\right\| ^2\right]$ . Then, tuning the learning rate carefully, we get
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \left\| \nabla F \left(\mathbf {x} ^ {(r)}\right) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {\left(L \sigma^ {2} A\right) ^ {1 / 2}}{\sqrt {S K R}} + \frac {\left(L ^ {2} \sigma^ {2} A ^ {2}\right) ^ {1 / 3}}{K ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L ^ {2} \zeta^ {2} A ^ {2}\right) ^ {1 / 3}}{R ^ {2 / 3}} + \frac {L \beta A}{R}\right) \tag {81}
$$
where  $A \coloneqq F(\mathbf{x}^0) - F^*$ . Eq. (80) and Eq. (81) are the upper bounds with partial client participation. In particular, when  $S = M$ , we get the claim of the non-convex case of Theorem 2 and Corollary 2.
F Simulations on quadratic functions
Nine groups of simulated experiments with various degrees of heterogeneity are provided in Table 6 as a extension of the experiment in Subsection 4.1. Figure 7 plots the results of PFL and SFL with various combinations of  $\delta$  and  $\zeta_{*}$ .
Table 6: Settings of simulated experiments. Each setting has two local objectives (i.e.,  $M = 2$ ) and shares the same global objective. Choosing large value of  $\zeta_*$  and  $\delta$  means higher heterogeneity. The definitions of  $\zeta_*$  and  $\delta$  can be found in Subsection 4.1.
Settings	ζ* = 1	ζ* = 10	ζ* = 100
δ = 0	{F1(x) = 1/2x2 + x}	{F1(x) = 1/2x2 + 10x}	{F1(x) = 1/2x2 + 100x}
{F2(x) = 1/2x2 - x}	{F2(x) = 1/2x2 - 10x}	{F2(x) = 1/2x2 - 100x}
δ = 1/3	{F1(x) = 2/3x2 + x}	{F1(x) = 2/3x2 + 10x}	{F1(x) = 2/3x2 + 100x}
{F2(x) = 1/3x2 - x}	{F2(x) = 1/3x2 - 10x}	{F2(x) = 1/3x2 - 100x}
δ = 1	{F1(x) = x2 + x}	{F1(x) = x2 + 10x}	{F1(x) = x2 + 100x}
{F2(x) = -x}	{F2(x) = -10x}	{F2(x) = -100x}
(a)  $\delta = 0, \zeta_{*} = 1$
(b)  $\delta = 0,\zeta_{*} = 10$
(c)  $\delta = 0, \zeta_{*} = 100$
(d)  $\delta = \frac{1}{3},\zeta_{*} = 1$
(g)  $\delta = 1, \zeta_{*} = 1$
Figure 7: Simulations on quadratic functions. The best learning rates are chosen from [0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6] with grid search. We run each experiments for 5 random seeds. Shaded areas show the min-max values.
(e)  $\delta = \frac{1}{3},\zeta_{*} = 10$
(h)  $\delta = 1$ $\zeta_{*} = 10$
(f)  $\delta = \frac{1}{3},\zeta_{*} = 100$
(i)  $\delta = 1,\zeta_{*} = 100$
G More experimental details
This section serves as a supplement and enhancement to Section 4. Our code is partly from Gao et al. (2021); Zeng et al. (2021); J Hunj Hunwala et al. (2023) (more references are included in the code), and it is available at https://github.com/liyipeng00/convergence.
G.1 Extended Dirichlet partition
(a) 10 clients, ExDir(1, 100.0)
(d) 10 clients, ExDir(2, 100.0)
(g) 10 clients, ExDir(5, 100.0)
(j) 10 clients, ExDir(10, 100.0)
Figure 8: Visualization of partitioning results on MNIST by Extended Dirichlet strategy. The  $x$ -axis indicates client IDs and the  $y$ -axis indicates labels. The value in each cell is the number of data samples of a label belonging to that client. For the first row, there are only one possible result in the case where each client owns one label with 10 clients and 10 labels in total, so these three partitions are the same. For the second, third and forth rows, data heterogeneity increases from left to right.
(k) 10 clients, ExDir(10, 10.0)
(1) 10 clients, ExDir(10, 1.0)
Baseline. There are two common partition strategies to simulate the heterogeneous settings in the FL literature. According to Li et al. (2022), they can be summarized as follows:
a) Quantity-based class imbalance: Under this strategy, each client is allocated data samples from a fixed number of classes. The initial implementation comes from McMahan et al. (2017), and has extended by Li et al. (2022) recently. Specifically, Li et al. first randomly assign  $C$  different classes to each client. Then, the samples of each class are divided randomly and equally into the clients which owns the class.
b) Distribution-based class imbalance: Under this strategy, each client is allocated a proportion of the data samples of each class according to Dirichlet distribution. The initial implementation, to the best of our knowledge, comes from Yurochkin et al. (2019). For each class  $c$ , Yurochkin et al. draw  $p_c \sim \mathrm{Dir}(\alpha q)$  and allocate a  $p_{c,m}$  proportion of the data samples of class  $k$  to client  $m$ . Here  $q$  is the prior distribution, which is set to 1.
Extended Dirichlet strategy. This is to generate arbitrarily heterogeneous data across clients by combining the two strategies above. The difference is to add a step of allocating classes (labels) to determine the number of classes per client (denoted by  $C$ ) before allocating samples via Dirichlet distribution (with concentrate parameter  $\alpha$ ). Thus, the extended strategy can be denoted by  $\operatorname{ExDir}(C, \alpha)$ . The implementation is as follows (one partitioning example is shown in Figure 8):
- Allocating classes: We randomly allocate  $C$  different classes to each client. After assigning the classes, we can obtain the prior distribution  $\mathbf{q}_c$  for each class  $c$  (see Figure 9).
- Allocating samples: For each class  $c$ , we draw  $p_c \sim \mathrm{Dir}(\alpha q_c)$  and then allocate a  $p_{c,m}$  proportion of the samples of class  $c$  to client  $m$ . For example,  $q_c = [1, 1, 0, 0, \dots, ]$  means that the samples of class  $c$  are only allocated to the first 2 clients.
This strategy have two levels, the first level to allocate classes and the second level to allocate samples. We note that Reddi et al. (2021) use a two-level partition strategy to partition the CIFAR-100 dataset. They draw a multinomial distribution from the Dirichlet prior at the root  $(\mathrm{Dir}(\alpha))$  and a multinomial distribution from the Dirichlet prior at each coarse label  $(\mathrm{Dir}(\beta))$ .
Figure 9: Allocating 2 classes (4 classes in total) to 5 clients.
G.2 Gradient clipping.
Two partitions, the extreme setting  $C = 1$  (i.e.,  $\mathrm{ExDir}(1,10.0)$ ) and the moderate settings  $C = 2$  (i.e.,  $\mathrm{ExDir}(2,10.0)$ ) are used in the main body. For both settings, we use the gradient clipping to improve the stability of the algorithms as done in previous works Acar et al. (2021); Jhunjhunwala et al. (2023). Further, we note that the gradient clipping is critical for PFL and SFL to prevent divergence in the learning process on heterogeneous data, especially in the extreme setting. Empirically, we find that the fitting "max norm" of SFL is larger than PFL. Thus we trained VGG-9 on CIFAR-10 with PFL and SFL for various values of the max norm of gradient clipping to select the fitting value, and had some empirical observations for gradient clipping in FL. The experimental results are given in Table 7, Table 8 and Figure 10. The empirical observations are summarized as follows:
1) The fitting max norm of SFL is larger than PFL. When the max norm is set to be 20 for both algorithms, gradient clipping helps improve the performance of PFL, while it may even hurt that of SFL (e.g., see the 12-th row in Table 8).
2) The fitting max norm for the case with high data heterogeneity is larger than that with low data heterogeneity. This is suitable for both algorithms. (e.g., see the 12, 24-th rows in Table 8)
3) The fitting max norm for the case with more local update steps is larger than that with less local update steps. This is suitable for both algorithms. (e.g., see the 4, 8, 12-th rows in Table 8)
4) Gradient clipping with smaller values of the max norm exhibits a preference for a larger learning rate. This means that using gradient clipping makes the best learning rate larger. This phenomenon is more obvious when choosing smaller values of the max norm (see Table 7).
5) The fitting max norm is additionally affected by the model architecture, model size, and so on.
Finally, taking into account the experimental results and convenience, we set the max norm of gradient clipping to 10 for PFL and 50 for SFL for all settings in this paper.
Table 7: Test accuracies when using gradient clipping with various values of the max norm for VGG-9 on CIFAR-10. Other settings without being stated explicitly are identical to that in the main body. The results are computed over the last 40 training rounds (with 1000 training rounds in total). The highest test accuracy among different learning rates is marked in cyan for both algorithms.
Setting	PFL10-2.0	10-1.5	10-1.0	10-0.5	SFL10-2.5	10-2.0	10-1.5	10-1.0
VGG-9, C=1, K=5, w/o clip	∞	25.43	30.95	30.63	10.00	∞	43.93	53.79	57.69	10.00
VGG-9, C=1, K=5, w/ clip	20	21.92	31.04	32.41	24.99	100	43.93	53.79	57.69	10.00
VGG-9, C=1, K=5, w/ clip	10	12.51	25.67	34.89	28.77	50	43.79	53.73	58.63	10.00
VGG-9, C=1, K=5, w/ clip	5	10.54	16.72	27.01	35.11	20	43.12	53.17	57.96	10.00
VGG-9, C=1, K=20, w/o clip	∞	24.23	27.53	26.91	10.00	∞	35.63	10.00	10.00	10.00
VGG-9, C=1, K=20, w/ clip	20	19.44	27.60	26.41	15.00	100	35.63	10.00	10.00	10.00
VGG-9, C=1, K=20, w/ clip	10	11.51	22.81	28.79	21.10	50	34.55	27.11	10.00	10.00
VGG-9, C=1, K=20, w/ clip	5	10.39	14.56	22.48	27.26	20	30.49	10.00	10.00	10.00
VGG-9, C=1, K=50, w/o clip	∞	22.44	23.70	20.97	10.00	∞	25.11	10.00	10.00	10.00
VGG-9, C=1, K=50, w/ clip	20	17.82	23.80	20.72	10.00	100	25.11	10.00	10.00	10.00
VGG-9, C=1, K=50, w/ clip	10	10.14	20.58	21.95	10.00	50	23.41	10.00	10.00	10.00
VGG-9, C=1, K=50, w/ clip	5	10.31	10.44	18.25	18.02	20	18.27	10.00	10.00	10.00
VGG-9, C=2, K=5, w/o clip	∞	41.36	51.34	55.22	10.00	∞	58.33	69.14	71.58	10.00
VGG-9, C=2, K=5, w/ clip	20	41.46	51.30	56.54	47.47	100	58.33	69.14	71.58	10.00
VGG-9, C=2, K=5, w/ clip	10	38.50	50.99	57.09	53.46	50	58.35	68.75	71.24	10.00
VGG-9, C=2, K=5, w/ clip	5	26.07	46.49	58.26	56.71	20	57.81	69.17	70.94	10.00
VGG-9, C=2, K=20, w/o clip	∞	55.64	64.04	10.00	10.00	∞	60.56	67.70	64.94	10.00
VGG-9, C=2, K=20, w/ clip	20	55.70	64.14	66.57	10.00	100	59.48	68.11	64.94	10.00
VGG-9, C=2, K=20, w/ clip	10	54.70	63.97	68.07	61.92	50	60.82	67.93	66.50	10.00
VGG-9, C=2, K=20, w/ clip	5	47.32	61.61	67.11	63.56	20	58.36	67.65	67.74	10.00
VGG-9, C=2, K=50, w/o clip	∞	63.93	67.85	10.00	10.00	∞	61.93	68.05	61.77	10.00
VGG-9, C=2, K=50, w/ clip	20	63.94	68.26	67.15	59.46	100	62.36	67.01	62.84	10.00
VGG-9, C=2, K=50, w/ clip	10	62.72	67.57	69.11	64.21	50	62.27	67.52	62.59	10.00
VGG-9, C=2, K=50, w/ clip	5	58.52	65.34	66.77	64.75	20	59.80	68.72	64.26	38.22
Figure 10: The corresponding training curves of Table 8 (VGG-9 on CIFAR-10).
Table 8: The best learning rate (selected in Table 7) and its corresponding test accuracies in the short run (1000 training rounds) and in the long run (4000 training rounds). The results are computed over the last 40 training rounds in the short run (the 5-th, 9-th columns) and 100 training rounds in the long run (the 6-th, 10-th columns). That the algorithms diverge when without gradient clipping makes the result with  $\dagger$ . The results that deviate from the vanilla case (w/o gradient clipping) considerably (more than  $2\%$ ) are marked in magenta and teal.
Setting	PFL	Lr	Acc.	Acc.	SFL	Lr	Acc.	Acc.
{CIFAR-10, C = 1}
1	VGG-9, K = 5, w/o clip	∞	10-1.5	30.95	60.75	∞	10-1.5	57.69	78.56
2	VGG-9, K = 5, w/ clip	20	10-1.0	32.41	67.97 (+7.2)	100	10-1.5	57.69	78.75
3	VGG-9, K = 5, w/ clip	10	10-1.0	34.89 (+3.9)	69.10 (+8.4)	50	10-1.5	58.63	78.56
4	VGG-9, K = 5, w/ clip	5	10-0.5	35.11 (+4.2)	71.07 (+10.3)	20	10-1.5	57.96	79.06
5	VGG-9, K = 20, w/o clip	∞	10-1.5	27.53	56.89	∞	10-2.5	35.63	72.90
6	VGG-9, K = 20, w/ clip	20	10-1.5	27.60	57.01	100	10-2.5	35.63	73.06
7	VGG-9, K = 20, w/ clip	10	10-1.0	28.79	64.11 (+7.2)	50	10-2.5	34.55	73.16
8	VGG-9, K = 20, w/ clip	5	10-0.5	27.26	62.31 (+5.4)	20	10-2.5	30.49 (-5.1)	69.66 (-3.2)
9	VGG-9, K = 50, w/o clip	∞	10-1.5	23.70	48.29	∞	10-2.5	25.11	69.10
10	VGG-9, K = 50, w/ clip	20	10-1.5	23.80	47.64	100	10-2.5	25.11	69.01
11	VGG-9, K = 50, w/ clip	10	10-1.0	21.95	46.21 (-2.1)	50	10-2.5	23.41	68.71
12	VGG-9, K = 50, w/ clip	5	10-1.0	18.25 (-5.5)	22.58 (-25.7)	20	10-2.5	18.27 (-6.8)	62.70 (-6.4)
{CIFAR-10, C = 2}
13	VGG-9, K = 5, w/o clip	∞	10-1.0	55.22	76.98	∞	10-1.5	71.58	82.09
14	VGG-9, K = 5, w/ clip	20	10-1.0	56.54	78.28	100	10-1.5	71.58	82.17
15	VGG-9, K = 5, w/ clip	10	10-1.0	57.09	78.18	50	10-1.5	71.24	82.18
16	VGG-9, K = 5, w/ clip	5	10-1.0	58.26 (+3.0)	76.69	20	10-1.5	70.94	82.48
17	VGG-9, K = 20, w/o clip	∞	10-1.5	64.04	77.21	∞	10-2.0	67.70	81.31
18	VGG-9, K = 20, w/ clip	20	10-1.0	66.57 (+2.5)	78.87	100	10-2.0	68.11	82.08
19	VGG-9, K = 20, w/ clip	10	10-1.0	68.07 (+4.0)	78.85	50	10-2.0	67.93	81.50
20	VGG-9, K = 20, w/ clip	5	10-1.0	67.11 (+3.1)	76.66	20	10-1.5	67.74	77.59 (-3.7)
21	VGG-9, K = 50, w/o clip	∞	10-1.5	67.85	10.00†	∞	10-2.0	68.05	79.30
22	VGG-9, K = 50, w/ clip	20	10-1.5	68.26	77.83	100	10-2.0	67.01	78.88
23	VGG-9, K = 50, w/ clip	10	10-1.0	69.11	78.13	50	10-2.0	67.52	79.42
24	VGG-9, K = 50, w/ clip	5	10-1.0	66.77	75.42	20	10-2.0	68.72	78.51
G.3 Grid search
We use the grid search to find the best learning rate on one random seed "1234". Since we have observed that the best learning rate of PFL is smaller than SFL empirically, the grid for PFL is  $\{10^{-2.0}, 10^{-1.5}, 10^{-1.0}, 10^{-0.5}\}$  (\{0.01, 0.03, 0.1, 0.3\} in fact) and the grid for SFL is  $\{10^{-2.5}, 10^{-2.0}, 10^{-1.5}, 10^{-1.0}\}$  (\{0.003, 0.01, 0.03, 0.1\} in fact). We use these grids for all tasks in this paper, including MNIST and FMNIST in the next subsection.
One practical method used in J Hunjhuwala et al. (2023) to find the best learning rate is running the algorithms for a fewer training rounds and then comparing the short-run results by some metrics (e.g., training accuracy) when the computation resources are restrictive and the task is complex (e.g., CIFAR-10). However, we should pay attention to whether the chosen learning rates are appropriate in the specific scenarios, as the best learning rate in the short run may not be the best in the long run. One alternative method is using the short-run results to find some alternatives (coarse-grained search) and then using the long-run results to find the best one (fine-grained search).
In this paper, for CIFAR-10 and CINIC-10, we run the algorithms for 1000 training rounds to find the candidate learning rates (with a less than  $3\%$  difference to the best result in test accuracy), called as coarse-grained search; and then run the algorithms for 4000 training rounds with the candidate learning rates to find the best learning rate (with the highest test accuracy), called as fine-grained search. The max norm of gradient clipping is set to 10 for PFL and 50 for SFL for all settings (see subsection G.2). Other hyperparameters are identical to that in the main body.
The results of the coarse-grained search are collected in Figure 12, Table 10. Taking the setting training VGG-9 on CIFAR-10 as an example. We first find the candidate learning rates, whose short-run test accuracies are only  $3\%$  or less below the best accuracy. The candidate learning rates are summarized in Table 9. The training curves are in Figure 11. We then find the best learning rate, whose long-run test accuracy is the highest among the candidate learning rates. The final best learning rates are in Table 9. For fine-grained search of other settings, please refer to the code.
Table 9: Best learning rates found by the fine-grained search. The candidate learning rates are collected in the cell and the correspond long-run test accuracies are in the parentheses. According to the long-run test accuracies, we keep the best learning rate and cross out the others.
Settings	PFL	SFL
CIFAR-10, VGG-9, C = 1, K = 5	0.1	0.03
CIFAR-10, VGG-9, C = 1, K = 20	0.1	0.003
CIFAR-10, VGG-9, C = 1, K = 50	0.03(28.72), 0.1 (46.21)	0.003
CIFAR-10, VGG-9, C = 2, K = 5	0.1	0.01(82.05), 0.03 (82.18)
CIFAR-10, VGG-9, C = 2, K = 20	0.1	0.01 (81.50), 0.03 (78.35)
CIFAR-10, VGG-9, C = 2, K = 50	0.03(76.14), 0.1 (78.13)	0.01
Figure 11: The corresponding training curves of Table 9. We mark the best learning rate in the short run with “*” and other candidate learning rates with “***” in the legend. The top row shows the first three settings and the bottom row shows the last three settings in Table 9.
(a) Left: Logistic/MNIST.Middle: MLP/MNIST. Right: LeNet-5/MNIST
(b) Left: LeNet-5/FMNIST. Right: CNN/FMNIST
(c) Left: VGG-9/CIFAR-10. Right: ResNet-18/CIFAR-10
(d) Left: VGG-9/CINIC-10. Right: ResNet-18/CINIC-10
Figure 12: Test accuracies after training for 1000 rounds for various settings. Details are in Table 10.
G.4 More experimental results
More experimental results are provided in this subsection. These include results on MNIST (LeCun et al., 1998), FMNIST (Xiao et al., 2017) and additional results on CIFAR-10 and CINIC-10.