SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Some abbreviations of the used terms in the proof of bounded training error.
Notation	Formulation	Description
w	-	parameters trained with set C
\(\widetilde{w}\)	-	parameters trained with set \(\widetilde{C}\)
\(\Delta^t\)	\(\frac{1}{C}\sum_{i\in\mathcal{C}}\mathbb{E}\|w_{i,K}^{t-1}-w^t\|^2\)	inconsistency/divergence term in round t
Then we introduce some important lemmas in our proofs.
A.3.1 Important Lemmas
Lemma 5 (Lemma 3.11 in [11]) We follow the definition in [11, 51] to upper bound the uniform stability term after each communication round in FL paradigm. Different from their vanilla calculations, FL considers the finite-sum function on heterogeneous clients. Let non-negative objective  $f$  is  $L$ -smooth and  $L_{G}$ -Lipschitz. After training  $T$  rounds on  $\mathcal{C}$  and  $\tilde{\mathcal{C}}$ , our method generates two models  $w^{T + 1}$  and  $\widetilde{w}^{T + 1}$  respectively. For each data  $z$  and every  $t_0 \in \{1,2,3,\dots,S\}$ , we have:
$$
\mathbb {E} \| f \left(w ^ {T + 1}; z\right) - f \left(\tilde {w} ^ {T + 1}; z\right) \| \leq \frac {U t _ {0}}{S} + \frac {L _ {G}}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \left[ \| w _ {i, K} ^ {T} - \tilde {w} _ {i, K} ^ {T} \| | \xi \right]. \tag {33}
$$
Proof Let  $\xi = 1$  denote the event  $\| w^{t_0} - \widetilde{w}^{t_0}\| = 0$  and  $U = \sup_{w,z}f(w;z)$ , we have:
$$
\begin{array}{l} \mathbb {E} \| f (w ^ {T + 1}; z) - f (\widetilde {w} ^ {T + 1}; z) \| \\ = P (\{\xi \}) \mathbb {E} \left[ \| f (w ^ {T + 1}; z) - f (\widetilde {w} ^ {T + 1}; z) \| | \xi \right] + P (\{\xi^ {c} \}) \mathbb {E} \left[ \| f (w ^ {T + 1}; z) - f (\widetilde {w} ^ {T + 1}; z) \| | \xi^ {c} \right] \\ \leq \mathbb {E} \left[ \| f (w ^ {T + 1}; z) - f (\tilde {w} ^ {T + 1}; z) \| | \xi \right] + P (\{\xi^ {c} \}) \sup  _ {w, z} f (w; z) \\ \leq L _ {G} \mathbb {E} \left[ \| w ^ {T + 1} - \widetilde {w} ^ {T + 1} \| | \xi \right] + U P (\{\xi^ {c} \}) \\ = L _ {G} \mathbb {E} \left[ \| \frac {1}{C} \sum_ {i \in \mathcal {C}} (w _ {i, K} ^ {T} - \widetilde {w} _ {i, K} ^ {T}) \| | \xi \right] + U P (\{\xi^ {c} \}) \\ \leq \frac {L _ {G}}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \left[ \left\| w _ {i, K} ^ {T} - \tilde {w} _ {i, K} ^ {T} \right\| \mid \xi \right] + U P (\{\xi^ {c} \}). \\ \end{array}
$$
Before the  $j^{\star}$ -th data on  $i^{\star}$ -th client is sampled, the iterative states are identical on both  $\mathcal{C}$  and  $\widetilde{\mathcal{C}}$ . Let  $\widetilde{j}$  is the index of the first different sampling, if  $\widetilde{j} > t_0$ , then  $\xi = 1$  hold for  $t_0$ . Therefore, we have:
$$
P (\{\xi^ {c} \}) = P (\{\xi = 0 \}) \leq P (\widetilde {j} \leq t _ {0}) \leq \frac {t _ {0}}{S},
$$
where  $\widetilde{j}$  is uniformly selected. This completes the proof.
Lemma 6 (Lemma 1.1 in [51]) Different from their calculations, we prove the similar inequalities on  $f$  in the stochastic optimization. Let non-negative objective  $f$  is  $L$ -smooth w.r.t  $w$ . The local updates satisfy  $w_{i,k+1}^t = w_{i,k}^t - \eta g_{i,k}^t$  on  $\mathcal{C}$  and  $\widetilde{w}_{i,k+1}^t = \widetilde{w}_{i,k}^t - \eta \widetilde{g}_{i,k}^t$  on  $\widetilde{\mathcal{C}}$ . If at  $k$ -th iteration on each round, we sample the same data in  $\mathcal{C}$  and  $\widetilde{\mathcal{C}}$ , then we have:
$$
\mathbb {E} \| w _ {i, k + 1} ^ {t} - \widetilde {w} _ {i, k + 1} ^ {t} \| \leq (1 + \eta L) \mathbb {E} \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| + 2 \eta \sigma_ {l}. \tag {34}
$$
Proof In each round  $t$ , by the triangle inequality and omitting the same data  $z$ , we have:
$$
\begin{array}{l} \mathbb {E} \| w _ {i, k + 1} ^ {t} - \widetilde {w} _ {i, k + 1} ^ {t} \| \\ = \mathbb {E} \| w _ {i, k} ^ {t} - \eta g _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} - \eta \widetilde {g} _ {i, k} ^ {t} \| \\ \leq \mathbb {E} \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| + \eta \mathbb {E} \| g _ {i, k} ^ {t} - \widetilde {g} _ {i, k} ^ {t} \| \\ = \mathbb {E} \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| + \eta \mathbb {E} \| (g _ {i, k} ^ {t} - \nabla f _ {i} (w _ {i, k} ^ {t})) - (\widetilde {g} _ {i, k} ^ {t} - \nabla f _ {i} (\widetilde {w} _ {i, k} ^ {t})) + (\nabla f _ {i} (w _ {i, k} ^ {t}) - \nabla f _ {i} (\widetilde {w} _ {i, k} ^ {t})) \| \\ \leq \mathbb {E} \| w _ {i, k} ^ {t} - \tilde {w} _ {i, k} ^ {t} \| + \eta \mathbb {E} \| g _ {i, k} ^ {t} - \nabla f _ {i} (w _ {i, k} ^ {t}) \| + \eta \mathbb {E} \| \tilde {g} _ {i, k} ^ {t} - \nabla f _ {i} (\tilde {w} _ {i, k} ^ {t}) \| + \eta \mathbb {E} \| \nabla f _ {i} (w _ {i, k} ^ {t}) - \nabla f _ {i} (\tilde {w} _ {i, k} ^ {t}) \| \\ \leq (1 + \eta L) \mathbb {E} \left\| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \right\| + 2 \eta \sigma_ {l}. \\ \end{array}
$$
The final inequality adopts assumptions of  $\mathbb{E}\| g_{i,k}^{t} - \nabla f_{i}(w_{i,k}^{t})\| \leq \sqrt{\mathbb{E}\|g_{i,k}^{t} - \nabla f_{i}(w_{i,k}^{t})\|^{2}}\leq \sigma_{l}$ . This completes the proof.
Lemma 7 (Lemma 1.2 in [51]) Different from their calculations, we prove the similar inequalities on  $f$  in the stochastic optimization. Let non-negative objective  $f$  is  $L$ -smooth and  $L_{G}$ -Lipschitz w.r.t  $w$ . The local updates satisfy  $w_{i,k+1}^{t} = w_{i,k}^{t} - \eta g_{i,k}^{t}$  on  $\mathcal{C}$  and  $\widetilde{w}_{i,k+1}^{t} = \widetilde{w}_{i,k}^{t} - \eta \widetilde{g}_{i,k}^{t}$  on  $\widetilde{\mathcal{C}}$ . If at  $k$ -th iteration on each round, we sample the different data in  $\mathcal{C}$  and  $\widetilde{\mathcal{C}}$ , then we have:
$$
\mathbb {E} \left\| w _ {i, k + 1} ^ {t} - \widetilde {w} _ {i, k + 1} ^ {t} \right\| \leq \mathbb {E} \left\| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \right\| + 2 \eta \left(\sigma_ {l} + L _ {G}\right). \tag {35}
$$
Proof In each round  $t$ , let by the triangle inequality and denoting the different data as  $z$  and  $\widetilde{z}$ , we have:
$$
\begin{array}{l} \mathbb {E} \left\| w _ {i, k + 1} ^ {t} - \widetilde {w} _ {i, k + 1} ^ {t} \right\| \\ = \mathbb {E} \| w _ {i, k} ^ {t} - \eta g _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} - \eta \widetilde {g} _ {i, k} ^ {t} \| \\ \leq \mathbb {E} \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| + \eta \mathbb {E} \| g _ {i, k} ^ {t} - \widetilde {g} _ {i, k} ^ {t} \| \\ = \mathbb {E} \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| + \eta \mathbb {E} \| g _ {i, k} ^ {t} - \nabla f _ {i} \left(w _ {i, k} ^ {t}; z\right) - \widetilde {g} _ {i, k} ^ {t} - \nabla f _ {i} \left(\widetilde {w} _ {i, k} ^ {t}; \widetilde {z}\right) + \nabla f _ {i} \left(w _ {i, k} ^ {t}; z\right) - \nabla f _ {i} \left(\widetilde {w} _ {i, k} ^ {t}; \widetilde {z}\right) \| \\ \leq \mathbb {E} \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| + 2 \eta \sigma_ {l} + \eta \mathbb {E} \| \nabla f _ {i} (w _ {i, k} ^ {t}; z) - \nabla f _ {i} (\widetilde {w} _ {i, k} ^ {t}; \widetilde {z}) \| \\ \leq \mathbb {E} \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| + 2 \eta (\sigma_ {l} + L _ {G}). \\ \end{array}
$$
The final inequality adopts the  $L_{G}$ -Lipschitz. This completes the proof.
A.3.2 Bounded Uniform Stability
According to Lemma 5, we firstly bound the recursive stability on  $k$  in one round. If the sampled data is the same, we can adopt Lemma 6. Otherwise, we adopt Lemma 7. Thus we can bound the second term in Lemma 5 as:
$$
\begin{array}{l} \mathbb {E} \left[ \| w _ {i, k + 1} ^ {t} - \widetilde {w} _ {i, k + 1} ^ {t} \| | \xi \right] \\ = P (z) \mathbb {E} \left[ \| w _ {i, k + 1} ^ {t} - \tilde {w} _ {i, k + 1} ^ {t} \| | \xi , z \right] + P (\tilde {z}) \mathbb {E} \left[ \| w _ {i, k + 1} ^ {t} - \tilde {w} _ {i, k + 1} ^ {t} \| | \xi , \tilde {z} \right] \\ \leq \left(1 - \frac {1}{S}\right) (1 + \eta L) \mathbb {E} \left[ \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| | \xi \right] + 2 \eta \sigma_ {l} + \frac {1}{S} \mathbb {E} \left[ \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| | \xi \right] + \frac {2 \eta L _ {G}}{S} \\ = \left(1 + \left(1 - \frac {1}{S}\right) \eta L\right) \mathbb {E} \left[ \| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \| | \xi \right] + \frac {2 \eta L _ {G}}{S} + 2 \eta \sigma_ {l} \\ \leq e ^ {\left(1 - \frac {1}{S}\right) \eta L} \mathbb {E} \left[ \left\| w _ {i, k} ^ {t} - \widetilde {w} _ {i, k} ^ {t} \right\| | \xi \right] + \frac {2 \eta L _ {G}}{S} + 2 \eta \sigma_ {\ell}. \\ \end{array}
$$
At the beginning of each round  $t$ , FL paradigm will aggregate the last state of each client  $w_{i,K}^{t-1}$ , according to our method,  $w_{i,0}^{t} = w^{t} + \beta (w^{t} - w_{i,K}^{t-1})$ , thus the relationship between them is:
$$
\begin{array}{l} \frac {1}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \| w _ {i, 0} ^ {t} - w _ {i, K} ^ {t - 1} \| = (1 + \beta) \frac {1}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \| w ^ {t} - w _ {i, K} ^ {t - 1} \| \leq (1 + \beta) \frac {1}{C} \sum_ {i \in \mathcal {C}} \sqrt {\mathbb {E} \| w ^ {t} - w _ {i , K} ^ {t - 1} \| ^ {2}} \\ \leq (1 + \beta) \sqrt {\frac {1}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \| w ^ {t} - w _ {i , K} ^ {t - 1} \| ^ {2}} \leq (1 + \beta) \sqrt {\Delta^ {t}}. \\ \end{array}
$$
It could be seen that if we consider the  $w_{i,0}^{t} - w_{i,K}^{t-1}$  as a general update step, it is independent to the dataset. Hence, we assume a virtual update between  $w_{i,K}^{t-1}$  and  $w_{i,0}^{t}$  which could be bounded by the divergence term  $\Delta^t$ . Then we bound the recursive term on  $t$ .
We know that before  $t^{\star}K + k^{\star} = t_{0}$ , no different data is sampled, which is,  $w_{i,k+1}^{t} = \widetilde{w}_{i,k+1}^{t}$  for  $\forall tK + k \leq t^{\star}K + k^{\star}$ . After  $t_{0} + 1$ , they become different. Thus, when  $t^{\star}K + k^{\star} > t_{0}$ , let learning rate  $\eta_{t}$  to be a constant
within each round  $t$  and  $\eta = \frac{c}{t}$ , then we have:
$$
\begin{array}{l} \frac {1}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \left[ \| w _ {i, K} ^ {T} - \widetilde {w} _ {i, K} ^ {T} \| | \xi \right] \leq \left(\frac {2 L _ {G}}{S} + 2 \sigma_ {l}\right) \sum_ {t = t ^ {\star} K + k ^ {\star}} ^ {T K} \eta_ {t} \exp \left(\left(1 - \frac {1}{S}\right) L \sum_ {\tau = t} ^ {T K} \eta_ {\tau}\right) \\ + (1 + \beta) ^ {\frac {1}{\beta c L}} \sum_ {t = t ^ {\star} + 1} ^ {T} \exp \left(\left(1 - \frac {1}{S}\right) L \sum_ {\tau = t} ^ {T K} \eta_ {\tau}\right) \sqrt {\Delta^ {t}}. \\ \end{array}
$$
We adopt the same learning rate  $\eta = \frac{c}{t}$  where  $c = \frac{\mu_0}{K}$  is a positive constant, then
$$
\begin{array}{l} \frac {1}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \left[ \| w _ {i, K} ^ {T} - \widetilde {w} _ {i, K} ^ {T} \| | \xi \right] \\ \leq 2 c \left(\frac {L _ {G}}{S} + \sigma_ {l}\right) \sum_ {t = t ^ {*} K + k ^ {*}} ^ {T K} \frac {1}{t} \exp \left(\left(1 - \frac {1}{S}\right) c L \sum_ {\tau = t} ^ {T K} \frac {1}{\tau}\right) \\ + (1 + \beta) ^ {\frac {1}{\beta c L}} \sum_ {t = t ^ {*} + 1} ^ {T} \exp \left(\left(1 - \frac {1}{S}\right) c L \sum_ {\tau = t} ^ {T K} \frac {1}{\tau}\right) \sqrt {\Delta^ {t}} \\ \leq 2 c \left(\frac {L _ {G}}{S} + \sigma_ {l}\right) \sum_ {t = t ^ {\star} K + k ^ {\star}} ^ {T K} \frac {1}{t} \exp \left(\left(1 - \frac {1}{S}\right) c L \log \left(\frac {T K}{t}\right)\right) \\ + (1 + \beta) ^ {\frac {1}{\beta c L}} \sum_ {t = t ^ {\star} + 1} ^ {T} \exp \left(\left(1 - \frac {1}{S}\right) c L \log \left(\frac {T K}{t}\right)\right) \sqrt {\Delta^ {t}} \\ \leq 2 c \left(\frac {L _ {G}}{S} + \sigma_ {l}\right) (T K) ^ {\left(1 - \frac {1}{S}\right) c L} \sum_ {t = t ^ {\star} K + k ^ {\star}} ^ {T K} \left(\frac {1}{t}\right) ^ {1 + \left(1 - \frac {1}{S}\right) c L} + (1 + \beta) ^ {\frac {1}{\beta c L}} \sum_ {t = t ^ {\star} + 1} ^ {T} \left(\frac {T K}{t}\right) ^ {\left(1 - \frac {1}{S}\right) c L} \sqrt {\Delta^ {t}} \\ \leq \frac {2 (L _ {G} + S \sigma_ {l})}{(S - 1) L} \left(\frac {T K}{t ^ {\star} K + k ^ {\star}}\right) ^ {c L} + (1 + \beta) ^ {\frac {1}{\beta c L}} \sum_ {t = t ^ {\star} + 1} ^ {T} \left(\frac {T K}{t ^ {\star}}\right) ^ {c L} \sqrt {\Delta^ {t}}. \\ \end{array}
$$
A.3.3 Proof of Theorem 3
Theorem 11 Under the Assumptions 6, 7, 9, and 10, let all conditions above satisfied, we can bound the uniform stability of our proposed FedInit as:
$$
\begin{array}{l} \mathbb {E} \| f (w ^ {T + 1}; z) - f (\widetilde {w} ^ {T + 1}; z) \| \\ \leq \frac {U ^ {\frac {c L}{1 + c L}}}{S - 1} \left[ \frac {2 \left(L _ {G} ^ {2} + S L _ {G} \sigma_ {l}\right) T K ^ {c L}}{L} \right] ^ {\frac {1}{1 + c L}} + (1 + \beta) ^ {\frac {1}{\beta c L}} \left[ \frac {U L T K}{2 \left(L _ {G} ^ {2} + S L _ {G} \sigma_ {l}\right)} \right] ^ {\frac {c L}{1 + c L}} \sum_ {t = 1} ^ {T} \sqrt {\Delta^ {t}}. \tag {36} \\ \end{array}
$$
Proof According to Lemma 5, we have:
$$
\mathbb {E} \| f (w ^ {T + 1}; z) - f (\widetilde {w} ^ {T + 1}; z) \| \leq \frac {U t _ {0}}{S} + \frac {L _ {G}}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \left[ \| w _ {i, K} ^ {T} - \widetilde {w} _ {i, K} ^ {T} \| \mid \xi \right].
$$
The second term is bounded by uniform stability term as:
$$
\begin{array}{l} \frac {1}{C} \sum_ {i \in \mathcal {C}} \mathbb {E} \left[ \| w _ {i, K} ^ {T} - \widetilde {w} _ {i, K} ^ {T} \| | \xi \right] \leq \frac {2 (L _ {G} + S \sigma_ {l})}{(S - 1) L} \left(\frac {T K}{t ^ {\star} K + k ^ {\star}}\right) ^ {c L} + (1 + \beta) ^ {\frac {1}{\beta c L}} \sum_ {t = t ^ {\star} + 1} ^ {T} \left(\frac {T K}{t ^ {\star}}\right) ^ {c L} \sqrt {\Delta^ {t}} \\ \leq \frac {2 (L _ {G} + S \sigma_ {l})}{(S - 1) L} \left(\frac {T K}{t ^ {\star} K + k ^ {\star}}\right) ^ {c L} + (1 + \beta) ^ {\frac {1}{\beta c L}} \sum_ {t = 1} ^ {T} \left(\frac {T K}{t ^ {\star}}\right) ^ {c L} \sqrt {\Delta^ {t}}. \\ \end{array}
$$
Let the  $t_0 = t^\star K + K^\star = \left[\frac{2(L_G^2 + SL_G\sigma_l)}{UL} (TK)^{cL}\right]^{\frac{1}{1 + cL}}$ , then  $t^\star >\left[\frac{2(L_G^2 + SL_G\sigma_l)}{UL}\right]^{\frac{1}{1 + cL}}$ . We have:
$$
\begin{array}{l} \mathbb {E} \| f (w ^ {T + 1}; z) - f (\widetilde {w} ^ {T + 1}; z) \| \\ \leq \frac {U ^ {\frac {c L}{1 + c L}}}{S - 1} \left[ \frac {2 \left(L _ {G} ^ {2} + S L _ {G} \sigma_ {l}\right)}{L} \right] ^ {\frac {1}{1 + c L}} (T K) ^ {\frac {c L}{1 + c L}} \\ + (1 + \beta) ^ {\frac {1}{\beta c L}} \left[ \frac {U L}{2 (L _ {G} ^ {2} + S L _ {G} \sigma_ {l})} \right] ^ {\frac {c L}{1 + c L}} (T K) ^ {\frac {c L}{1 + c L}} \sum_ {t = 1} ^ {T} \frac {\sqrt {\Delta^ {t}}}{T} \\ \end{array}
$$
$$
= \frac {U ^ {\frac {c L}{1 + c L}}}{S - 1} \left[ \frac {2 (L _ {G} ^ {2} + S L _ {G} \sigma_ {l}) (T K) ^ {c L}}{L} \right] ^ {\frac {1}{1 + c L}} + (1 + \beta) ^ {\frac {1}{\beta c L}} \left[ \frac {U L T K}{2 (L _ {G} ^ {2} + S L _ {G} \sigma_ {l})} \right] ^ {\frac {c L}{1 + c L}} \sum_ {t = 1} ^ {T} \frac {\sqrt {\Delta^ {t}}}{T}.
$$
In this section, we mainly provide the detailed experimental setups in our paper, including the introduction of the benchmarks, dataset, hyperparameters selections, and adding some more experiments.
B.1 Setups
Dataset. We follow the previous works and select the CIFAR-10/100 [20] dataset in our experiments. In the CIFAR-10 dataset, there is a total of 50,000 training images and 10,000 test images which contain 10 categories. Each data sample is a color image with a size of  $32 \times 32$ . In the CIFAR-100 dataset, there is also a total of 50,000 training images and 10,000 test images. It contains 100 categories of the same size as CIFAR-10. For their limited resolutions, we only use general data augmentations. On each local heterogeneous dataset, we use general normalization on the images with specific mean and variance. For the training process, we randomly crop a  $32 \times 32$  patch from the vanilla images with a zero padding of 4. For the test process, we use the raw images.
Heterogeneity. We follow Hsu et al. [14] to introduce the label imbalance as the heterogeneous dataset. According to the Dirichlet distribution, we first generate a specific vector with respect to a constant  $D_r$  to control its variance level. Usually, heterogeneity becomes stronger when  $D_r$  decreases. Then according to the vector, we sample the images from the training dataset. Here we enable the sampling with replacement to generate the local dataset, which means the local clients may have the same data sample if they are assigned to the same category. This is more related to the real scenario. At the same time, it also will lose some data samples, we assume this case is due to the offline devices. This is a common case because the FL has an unreliable network connection across the devices.
Benchmarks. In this paper, we use FedAvg [26], FedAdam [30], FedSAM [29], SCAFFOLD [19], FedDyn [1], and FedCM [46] as the benchmarks. FedAvg propose the general FL paradigm based on the local SGD method. It allows partial participation training via uniformly selecting a subset of local clients. A series of developments followed it to improve its performance. FedAdam studies the efficient adaptive optimizer on the global server update, which extends the scope of the FL paradigm. SCAFFOLD indicates that FL suffers from the client-drift problem which is due to the inconsistency of local optimum. Beyond this, it uses the variance reduction technique to further reduce the divergence across the local clients. To further alleviate, FedDyn studies the primal-dual method via adopting the ADMM to solve the problem. The consistency condition works as a constraint during the optimization. It proves that when the global model converges, the local objectives will be aligned with the global one. FedCM proposes an efficient momentum-based method, dubbed client-level momentum. It communicates the global update as a correction to correct each local update to force the local client updates in a similar direction. It maintains very high consistency via a biased correction. Therefore, it relies on an accurate global direction estimation. FedSAM considers the generalization performance. Generally, we adopt empirical risk minimization (ERM) to perform the optimization process. While the sharpness-aware-minimization (SAM) studies that it could search for a flat loss landscape. Flatness guarantees a higher generalization performance. Though our focus is not the generalization, we theoretically prove that even in the FedAvg method divergence term affects the generalization error bound more than the optimization error bound. From this perspective, generalization-efficiency methods may also be connected with consistency guarantees. These are all the SOTA benchmarks in the FL community that concern more on enhancing consistency.
Hyperparameters selection. Here we detail our hyperparameter selection in our experiments. For each splitting, we fix the total communication rounds  $T$ , local interval  $K$ , and mini-batchsize for all the benchmarks and our proposed FedInit. The other selections are stated as follows.
$\star$  means different selections according to the specific setups.
We fix the most hyperparameters of testing the whole benchmarks for a fair comparison. The other algorithm-specific hyperparameters are subjected to specific circumstances. The ResNet-18-GN and VGG-11 adopt the same set of selections. Then we show algorithm-specific hyperparameters:
Special hyperparameter selections. In the FedAdam method, we test that it is very sensitive to the global learning rate. Though we report the best selection is 0.1, it still requires some finetuning based on the dataset and experimental setups. In the FedSAM method, we test it is very sensitive to the perturbation learning rate. Usually, it should be selected as 0.1 in most cases. However, in some poor-sampling cases, i.e. low participation ratio, it should be selected as 0.01. In the FedDyn, we test it is very sensitive to the regularization coefficient.
Table 6: General hyperparameters introductions.
Dataset	CIFAR-10	best selection
communication round T	500	-
local interval K	5	-
minibatch	50	-
weight decay	1e-3	-
local learning rate	[0.01, 0.1, 0.5, 1]	0.1
global learning rate	[0.01, 0.1, 1.0]	1.0/0.1
learning rate decay	[0.995, 0.998, 0.9995]	0.998
relaxed coefficient β	[0.01, 0.02, 0.05, 0.1, 0.15]	0.1/0.01
Dataset	CIFAR-100	best selection
communication round T	500	-
local interval K	5	-
minibatch	50	-
weight decay	1e-3	-
local learning rate	[0.01, 0.1, 0.5, 1]	0.1
global learning rate	[0.01, 0.1, 1, 0]	1.0/0.1
learning rate decay	[0.998, 0.9995, 0.9998]	0.998/0.9995
relaxed coefficient β	[0.01, 0.02, 0.05, 0.1, 0.15]	★
Table 7: Algorithm-specific hyperparameter introductions.
Method	specific hyperparameter	introduction	selection	best selection
FedAdam	global learning rate	adaptive learning rate	[0.01, 0.05, 0.1, 1]	0.1
FedSAM	perturbation learning rate	ascent step update	[0.01, 0.1, 1]	0.1
FedDyn	regularization coefficient	coefficient of prox-term	[0.001, 0.01, 0.1, 1]	★
FedCM	client-level coefficient	ratios in local updates	[0.05, 0.1, 0.5, 0.9]	★
Generally, it adopts the regularization coefficient to be 0.1 on CIFAR-10 and  $0.01 / 0.001$  on CIFAR-100. In FedCM, we select the client-level coefficient as 0.1 which is followed by Xu et al. [46] in most cases. However, on the VGG-11 model, it fails to converge with a small client-level coefficient.
B.2 Experiments
B.2.1 Curves
In this section, we show the curves of our results.
(a) Dir-0.6  $10\% - 100$
(b) Dir-0.1  $10\% - 100$
(c) Dir-0.6  $5\%$  -200.
Figure 3: Loss on the CIFAR-10 dataset.
(d) Dir-0.1  $5\% -200$
(c) Dir-0.6  $5\%$  -200.
Figure 4: Loss on the CIFAR-100 dataset.
(a) Dir-0.6  $10\% - 100$
Figure 5: Test accuracy on the CIFAR-10 dataset.
(c) Dir-0.6  $5\%$  -200.
(a) Dir-0.6  $10\% - 100$
Figure 6: Test accuracy on the CIFAR-100 dataset.
(c) Dir-0.6  $5\% -200$
To show the stable accuracy curves, we use the third-party tsmoothie.smother to smooth the raw curve via the function ConvolutionSmotherwindow_len=100, window_type='hanning'). On most setups, our proposed FedInit achieves the SOTA results. It effectively avoids negative impacts from local overfitting.
B.2.2 Consistency of Different Initialization
In this part, we mainly test the consistency level of different  $\beta$ . The coefficient  $\beta$  controls the divergence level of the local initialization states. We select the FedAvg and SCAFFOLD to show the efficiency of the proposed relaxed initialization.
(a) Dir 0.6  $10\%$  -100 Accuracy and Consistency.
Figure 7: Experiments of FedAvg on the CIFAR-10 dataset.
(b) Dir 0.1  $5\%$  -200 Accuracy and Consistency.
Figure 8: Experiments of SCAFFOLD on the CIFAR-10 dataset.
(b) Dir  $0.15\% -200$  Accuracy and Consistency.
These experiments show that the relaxed initialization (RI) effectively reduces the consistency and improves the test accuracy. In all tests, when  $\beta = 0$  (green curve), it represents the vanilla method without RI. After incorporating the RI, the test accuracy achieves at least  $2\%$  improvement on each setup.
B.2.3 Communication, Calculation and Storage Costs
In this part, we mainly compare the communication, calculation, and storage costs theoretically and experimentally. By assuming the total model maintain  $d$  dimensions, we summarize the costs of benchmarks and our proposed FedInit as follows:
Table 8: Communication, calculation, and storage costs per communication round.
Method	communication	ratio	gradient calculation	ratio	total storage	ratio
FedAvg	Nd	1×	NKd	1×	Cd	1×
FedAdam	Nd	1×	NKd	1×	Cd	1×
FedSAM	Nd	1×	2NKd	2×	2Cd	2×
SCAFFOLD	2Nd	2×	NKd	1×	3Cd	3×
FedDyn	Nd	1×	NKd	1×	3Cd	3×
FedCM	2Nd	2×	NKd	1×	2Cd	3×
FedInit	Nd	1×	NKd	1×	Cd	1×
where  $N$  is the number of participating clients,  $C$  is the total number of clients, and  $K$  is the local training interval.