# FedRDS: Federated Learning on Non-IID Data via Regularization and Data Sharing
## 3 Proposed Method
### Data Sharing  
Although the inclusion of regular terms in the calculation of weights can alleviate the issue of client drift resulting from non-IID data in a heterogeneous network environment to some extent, the parameter bias gradually accumulates and amplifies during the training of the model using FL, and this is a crucial factor that contributes to the algorithm's decreased accuracy and slower convergence rate. It has been proven that shortening the dissimilarity of the local and global data distribution allows the local target to gradually approach the global target, quantified via the EMD (Earth Mover's Distance) [11]. The same conclusion holds even when regularization is applied in parallel. Therefore, we can introduce additional strategies to further improve the efficiency and effectiveness of FL algorithms under non-IID conditions. To this end, we instantiate \(l_{k}(\Theta)\) with the cross-entropy loss for further explanation:  
\[l_{k}(\Theta)=\sum_{i=1}^{I}\mathrm{p}(y=i)\mathbb{E}_{\mathrm{x}|y=i}[\log f _{i}(\mathrm{x},y)]. \tag{6}\]  
If the generalization error is ignored and the total training loss is directly optimized, the problem turns:  
\[\operatorname*{argmin}_{\Theta}\sum_{i=1}^{I}\mathrm{p}(y=i)\mathbb{E}_{ \mathrm{x}|y=i}[\log f_{i}(\mathrm{x},\Theta)]. \tag{7}\]  
Using the SGD optimization algorithm to calculate the value of \(\Theta\), the weight of the central server after the \(t\)-th update will be:  
\[ \begin{split}&\Theta^{t}=\Theta^{t-1}-\eta\sum_{i=1}^{I}\mathrm{p }(y=i)\
abla_{\Theta}\mathbb{E}_{\mathrm{x}|y=i}[\log f_{i}(\mathrm{x},\Theta^ {t-1})]\\ &-\sigma(\Theta^{t}-\Theta^{t-1}).\end{split}  \tag{8}\]Generally, there are \(K\) clients, and a separate SGD optimization is performed locally at each client. Correspondingly, the weights of client \(k\) after the \(t\)-th round of updates will be:  
\[ \begin{split}&\Theta_{k}^{t}=\Theta_{k}^{t-1}-\eta\sum_{i=1}^{l} \mathrm{P}(y=i)\
abla_{\Theta}\mathbb{E}_{\mathbf{x}|y=i}[\log f_{i}(\mathbf{ x},\Theta_{k}^{t-1})]\\ &-\sigma(\Theta_{k}^{t}-\Theta^{t-1}).\end{split}  \tag{9}\]  
Suppose that synchronization is performed every \(T\) optimization round at the central server; thus, the global model weight after \(T\) synchronizations will be:  
\[\Theta_{f}^{mT}=\sum_{k=1}^{K}\frac{n_{k}}{n}\Theta_{k}^{mT}. \tag{10}\]  
Denote \(\mathrm{p}_{k}\) as the data distribution of client \(k\) and suppose that \(\
abla_{\Theta}E_{\mathbf{x}|y=i}\log f_{i}(\mathbf{x},\Theta)\) is \(\lambda-Lipschitz\) for each class \(i\in[I]\), where the extent of client drift (a.k.a the weight variance) after \(m\)-th synchronization for each class \(i\) follows the inequality (Equation (10)) every \(T\) steps, according to [11]:  
\[ \begin{split}&\|\Theta_{f}^{mT}-\Theta_{c}^{mT}\|\leq\sum_{k=1}^{ K}\frac{n_{k}(a_{k})^{T}}{n}\underbrace{\|\Theta_{f}^{(m-1)T}-\Theta_{c}^{(m-1)T} \|}_{(1)}\\ &+\eta\sum_{k=1}^{K}n_{k}\underbrace{\sum_{i=1}^{l}\|\mathrm{p}_{ k}(y=i)-\mathrm{p}(y=i)\|}_{(2)}\\ &+\sum_{j=1}^{T-1}(a_{k})^{j}g_{max}(\Theta_{c}^{mT-1-k})\end{split}  \tag{11}\]  
where  
\[ \begin{split}& g_{max}(\Theta)=max_{i=1}^{l}\|\
abla_{\Theta} \mathbb{E}_{\mathbf{x}|y=i}\log f_{i}(\mathbf{x},\Theta)\|,\text{and}\\ & a_{k}=1+\sigma+\eta\sum_{i=1}^{l}\mathrm{p}_{k}(y=i)\lambda_{ \mathbf{x}|y=i}.\end{split}  \tag{12}\]  
The weight variance after the \(m\)-th synchronization is as follows: (1) the weight variance after the \((m-1)\)-th synchronization; and (2) the weight variance of the probability distance of the client \(k\) data distribution relative to the actual joint distribution. Let all clients be initialized from the same weights, and the client drift will be mainly attributed to the component (1) and the coefficient \(a_{k}\). The component (1) measures the EMD between the data distribution of client k and the collective distribution on the central server [11]. Consequently, the regularization coefficient \(\sigma\) plays a significant role in \(a_{k}\). The larger the EMD and \(\sigma\), the greater the degree of client drift.