# Lazy Aggregation for Heterogeneous Federated Learning
## 1 Introduction  
With the development of information technology, the amount of data generated by human beings is unprecedented. To mine the potential value of data, a series of emerging information technologies have been promoted (for example, Internet of Things, Big Data, and Data Mining). Nevertheless, it is tough for industry competition, monopoly, and data protection laws to achieve data sharing. Thus, it is necessary to discuss how to break the \"data silos\" to achieve secure data sharing. In 2016, Google proposed a privacy-constrained distributed neural network training paradigm named FL. Different from centralized training, FL aggregates the models' weights of devices to integrate data knowledge of all parties without disclosing local data and improves the performance of the model on unknown data. FL realizes the efficient and safe sharing of data knowledge among multiple parties. FL has been widely explored and applied in data-sensitive fields such as medical treatment, personalized recommendation, and monetary risk assessment. Unfortunately, there are some challenges in FL [1, 2]. First of all, the open distributed architecture requires network transmission for synchronous training. In practice, the local training delay is much smaller than network transmission delay, especially in the case of large scale and big model. The network quality directly determines training efficiency. Secondly, the central server cannot access local data and data statistics of the participants, which restricts data preprocessing operations (e.g., data cleaning, deduplication, and enhancement). In addition, different distributions and data sparsity greatly weaken the final effect and efficiency, and even fail to converge, which has become a dark cloud over FL. In order to resolve the issues of communication and data heterogeneity in FL, researchers have carried out a series of explorations. [PERSON] et al. [3] proposed the vanilla FL framework federated averaging (FedAvg), whose optimization focuses on increasing local epochs to reduce the synchronization frequency, and just some devices are chosen in each round. In Independent and Identically Distributed (IID), FedAvg per- forms very well and takes the first step of FL from theory to practice. However, FedAvg not only slows down the convergence speed but also greatly reduces the performance of the global model in non-IID. [PERSON] et al. [4] analyze theoretically convergence of FedAvg for the strongly convex and smooth problem, different data distributions expand the weight difference, and more epochs further exacerbate gradient offsets. [PERSON] et al. [5] proved that it could also affect FL performance in the case of global unbalanced dataset, and they proposed to use a global shared and balanced dataset to alleviate gradient offsets caused by different distributions. As sharing proportion increased, it performs better, which maybe cause data leakage unfortunately. [PERSON] et al. [6] proposed a novel FL framework FedGSP, which counteracts perturbations for heterogeneous data by sequential execution. Sequential training is more robust to heterogeneous data than parallel training; however, the efficiency of sequential training is lower. For this reason, the author proposes the inter-cluster grouping (IGP), which is achieved inter-group homogeneity and intra-group heterogeneity. Grouping requires device characteristics, which has security risks. In this paper, we propose a novel method for FL, which is shown in Figure 1. Federated lazy aggregating (FedLA) is used to address data heterogeneity. On the basis of FedAvg, FedLA allows more devices, which are from successional rounds, to train sequentially. FedLA doesn't need extra informations about devices. As paper [6] said, finding out heterogeneous device groups is an NP-hard problem. Due to sequential training performs well under both homogeneous and heterogeneous data, just finding out sufficient sampling to aggregate in time.