# Lazy Aggregation for Heterogeneous Federated Learning
## 3 Methods
### FedLAM  
FedLA eliminates redundant aggregations and allows more devices to train sequentially, enabling more data to be sampled and alleviating data sparsity (e.g., few samples, class imbalance, and poor quality). Continuous training sequentially without aggregation occurs more jitters than vanilla FL on vertical optimization. It is more urgent for momentum mechanism, especially for cross-devices. It can correct current gradients by previous gradients and raise the utility of historical gradients. Firstly, we assign \(K\) groups of gradient momentums on the server side, the initial value is zero gradient 0 and its transfer equation is:  
\[m_{t}^{k}\leftarrow\mu m_{t-1}^{k}+\Delta w_{t}^{k} \tag{7}\]  
where \(\mu\) is the momentum coefficient, the bigger coefficient means that historical gradients are dominant and have a longer survival period; \(m_{t}^{k}\) is the momentum of the \(k\)-th group in the \(t\)-th round; and \(\Delta w_{t}^{k}\) indicates the \(k\)-th updated gradient in the \(t\)-th round. Therefore, we propose the FedLA with momentum (FedLAM) that adds the CDM mechanism. Algorithm 2 is as follows:  
```
Input:\(T\), \(K\), \(E\), \(B\), \(w_{0}\), \(d_{0}\), \(\varepsilon\), \(\mu\)  Initialize \(m_{0}^{0},m_{1}^{1}\dotsm_{0}^{K-1}\leftarrow\) 0; \(n_{0}^{0},n_{0}^{1}\dots n_{0}^{K-1}\gets\) 0; \(w_{0}^{0},w_{0}^{1}\dots w_{0}^{K-1}\gets w_{0}\) for\(t=0,1,2,\dots,T-1\)do  Sample devices \(S_{t}\in[N],|S_{t}|=K\leq N\)  Transmit \(w_{0}^{0}\ w_{1}^{1},\ \dots,\ w_{t}^{K-1}\) to selected \(K\) devices, respectively for each device \(k\in S_{t}\) in parallel do \(n_{k}\), \(\Delta w_{t+1}^{k}=ClientUpdate\Big{(}w_{t}^{k},\ E,\ B\Big{)}\)  Transmit device \(n_{k}\), \(\Delta w_{t+1}^{k}\) to server endfor for\(k=0,1,2,\dots,T-1\)do \(n_{t+1}^{k}\gets n_{t}^{k}+n_{k}\) \(m_{t+1}^{k}\leftarrow\mu m_{t}^{k}+\Delta w_{t+1}^{k}\) \(w_{t+1}^{k}\gets w_{t}^{k}+m_{t+1}^{k}\) endfor \(d_{t+1}=W\Big{(}w_{t+1}^{0},\ w_{t+1}^{1}\dots\ w_{t+1}^{K-1}\Big{)}\) if\(\Big{|}\frac{d_{t+1}-d_{t+1}}{d_{t+1}}\Big{|}\leq\epsilon\)do \(w_{t+1}\leftarrow\frac{1}{\sum_{k=0}^{K-1}n_{t+1}^{k}}\sum_{k=0}^{K-1}\ n_{t+1}^{k}w_{t+1}^{k}\); \(w_{t+1}^{0},w_{t+1}^{1},w_{t+1}^{1}\dots w_{t+1}^{K-1}\gets w_{t+1}\) \(m_{t+1}\leftarrow\frac{1}{\sum_{k=0}^{K-1}n_{t+1}^{k}}\sum_{k=0}^{K-1}\ n_{t+1}^{k} \boldsymbol{n}_{t+1}^{k}\); \(m_{0}^{0},m_{0}^{1}\dots m_{0}^{K-1}\gets m_{t+1}\) (*) \(n_{t+1}^{0},n_{t+1}^{1}\dots n_{t+1}^{k-1}\gets 0\); \(d_{t+1}=0\) else \(w_{t+1}\gets w_{t}\) endif endfor Output:\(w_{T}\)
```  
**Algorithm 2**Federated Lazy Aggregate with Momentum (FedLAM)  
It is noteworthy that the selected devices send their gradients instead of weights, and the weights of each group are updated by corrected gradients which enhance the horizontal optimization and offset the vertical optimization jitters. In non-IID settings, the local models fall easily into the local optimum [7], which is further passed to the global model through their gradients [27]. The step of momentum aggregation is optional in Algorithm 2(*). It is suggested to train early and close it later to expand the search optimization space, which contributes to the walking out of the saddle point. CDM could help FL alleviate gradient disappearance and take the global performance to a higher level. Compared with [28], our scheme doesn't pass momentum on the device, which undoubtedly reduces the traffic between the device side and server side.