FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation
Authors: Haozhao Wang
Haoran Xu
Yichen Li
Yuan Xu
Ruixuan Li
Tianwei Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Federated Learning
Aggregation
Cross-round
Divergence-aware
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A new model aggregation method for Federated Learning that aggregates cross-round local models

Abstract: In Federated Learning (FL), model aggregation is pivotal. It involves a global server iteratively aggregating client local trained models in successive rounds without accessing private data. Traditional methods typically aggregate the local models from the current round alone. However, due to the statistical heterogeneity across clients, the local models from different clients may be greatly diverse, making the obtained global model incapable of maintaining the specific knowledge of each local model. In this paper, we introduce a novel method, FedCDA, which selectively aggregates cross-round local models, decreasing discrepancies between the global model and local models.
The principle behind FedCDA is that due to the different global model parameters received in different rounds and the non-convexity of deep neural networks, the local models from each client may converge to different local optima across rounds. Therefore, for each client, we select a local model from its several recent local models obtained in multiple rounds, where the local model is selected by minimizing its divergence from the local models of other clients. This ensures the aggregated global model remains close to all selected local models to maintain their data knowledge. Extensive experiments conducted on various models and datasets reveal our approach outperforms state-of-the-art aggregation methods.

Published as a conference paper at ICLR 2024
FEDCDA: FEDERATED LEARNING WITH CROSS-
ROUND DIVERGENCE-AWARE AGGREGATION
Haozhao Wang', Haoran Xu², Yichen Li", Yuan Xu', Ruixuan Li Tianwei Zhang*
Sa-La, Nanyang Technological University "Zhejiang University Department of Computer Science, Huazhong University of Science and Technology *Nanyang Technological University (hz.wang,
edu.t..dcccc cn, tiarwei. wtaeeeet.aannnnuuudeeeee edu.
ABSTRACT
In Federated Learning (FL), model aggregation is pivotal. It involves a global server iteratively aggregating client local trained models in successive rounds without accessing private data. Traditional methods typically aggregate the local models from the current round alone. However, due to the statistical heterogeneity across clients, the local models from different clients may be greatly diverse, mak- ing the obtained global model incapable of maintaining the specific knowledge of each local model. In this paper, we introduce a novel method, F FedCDA, which se- lectively aggregates cross-round local models, decreasing discrepancies between the global model and local models. The principle behind FedCDA is that due to the different global model parameters received in different rounds and the non- convexity of deep neural networks, the local models from each client may con- verge to different local optima across rounds. Therefore, for each client, we select a local model from its several recent local models obtained in multiple rounds, where the local model is selected by minimizing its divergence from the local models of other clients. This ensures the aggregated global model remains close to all selected local models to maintain their data knowledge. Extensive experi- ments conducted on various models and datasets reveal our approach outperforms state-of-the-ee aggregation methods.
INTRODUCTION
Federated Learning (FL) has emerged as key framework for training deep neural networks (DNNs)
through client collaboration without the need share original datasets (McMahan et al. 2017b
Wang et al.) 2022: Li ct al. 2022b). It has been extensively utilized in areas like medical image processing (Liu al. Guo al.) Xu et al.) and recommendation systems (Ramaswamy al. 2019 Ammad-ud-din et al.
2019). FL. is an iterative procedure in which each round involves the local
model training across various individual clients, and then aggregating these models centrally on a
server (McMahan et al. 2017a).
In this paper, we focus on the aggregation of FL, which is the critical step to obtain the global model from multiple local models. The typical aggregation method is FedAvg, which computes the coordinate- wise weighted average of parameters of local models with the weight as the ratio of the data size (McMahan et al. 2017b). Although the implementation of this method is straightforward, some works (Yurochkin et al., 22119a Li et al. 2022b Liu et al. 2022 Wang et al. (2020a) consider that the coordinate-wiig wise average will reduce the performance due the Nonill> (i.e., not indepen- dently and identically) data among clients. Specifically, they identify that the parameter ordering of different local models may be varied due the permutation invariance of neural network (NN) parameters. Thus, they propose re-ordering the parameters before applying the weighted average. Another type of work considers that the NonlID data also affects the aggregation weights and they propose adaptively setting the weights using learnable approach (Li et al. 2023 Although these
Hanzhho Wang, Haoran Xu, and Yichen Li contribute equally to this work. Ruixuan Li is the Correspond-
ing author.

---

Published as a conference paper at ICLR 2024
methods have achieved great success separately, they mainly aggregate the local models from the current single round, which may limit the improvement of FL performance.
Orthogonal to these works, in this paper, we focus on the aggregation of cross-round local models to further unleash the potential aggregation performance. Intuitively, to acquire the data knowledge of some specific client, it is necessary for the global model to be close to its locally trained model (Kirk- patrick et al. (017). Nevertheless, the local models of different clients in the same single round may have a large divergence from each other due the statistical heterogeneity. Thus, as shown in Fig- ure [[a] the aggregated global model may greatly deviate from these local models. To tackle this challenge, we consider common fact that each client is usually able achieve convergence in different rounds after the startup training stage, especially when FL prefers a larger interval for the local training process to save the communication cost (Sun et al. 2023a).. addition, due to receiv- ing different global models in different rounds and the existence of multiple local optima in deep neural networks (Wu et al. 2017 Kawaguchi] 2016) Xie al. 202T). each client often converges to different models in different rounds, each of which can usually learn local data well, especially when training data using the most advanced optimizer (Loshchilov Hutter 2019 Chaudhari et al.) 2017). Therefore, natural idea is that the global model can also fit the local data of some specific client once it approaches any of the different local models in multiple rounds. Motivated by this, the global model can essentially be obtained by aggregating selected local models from different rounds to reduce their divergence and maintain their knowledge.
Based the above motivation, we propose a novel aggregation method named FedCDA, which
Local update
Wi
Global model
selectively aggregates cross-round
Two-cound local models of
local models. More specifically, we design divergence-awarc se- w
client as w'
client w
w.
W
lection strategy that selects local models from multiple rounds with minimum divergence to their ag- gregated model and only aggre- gates the selected local models to obtain the global model. In this way, as shown in Figure [(b)] the global model approaches selected local models and thus maintains the data knowledge of clients, Considering the selection prob- lem is a combinatorial optimiza- tion problem with large search space, we further design an approximation version by selecting local models in a batch way to reduce the selection cost. Then, we establish theories to provide better understanding and guaran- tee the convergence of our method. We conduct extensive experiments on various datasets, and the results show that FedCDA outperforms state-of-tt of-the-ant-t the-art baselines. Our contributions are:
w
W
W
H
Two local optima of clicut
client
w
W
(a) Single round
(b) Cross round
(c) Legend
Figure 1: Comparison of different aggregation methods on a two-client FL. (a) The global model W is the aggregation of the local model W of client and w of the client 2 in the same t 1-th round. (b) W++1 is the aggregation of the t-th round local model w; and the L+ 1-th round local model w W obtained through cruss-round aggregation is close to the local optimum of client I and the local optimum 2 of client 2, while the single-round aggregation is distant from any local optima. A practical example is in Appendix[A]
To the best of our knowledge, this paper is the first to study the aggregation of cross-round local models. We identify that cross-round local models among clients may have smaller divergence than those in the single round and selectively aggregating them can make the global model ap- proach local models more closely, thus maintaining their local data knowledge.
We propose a new cross-round aggregation method named FedCDA. obtains the global model by aggregating local models selected from multiple rounds based on the criterion of the minimum divergence. Besides, we design an approximation strategy to reduce the cost of selection.
We establish comprehensive theories for our method. Specifically, we provide theoretical insights for understanding our algorithm and show that the approximation selection error is bounded by the convergence of the local model. Besides, we also prove the convergence of our method.
We conduct extensive experiments over various deep-learning models and datasets. The efficiency superiority of FedCDA is demonstrated by comparing our proposed aggregation method with traditional aggregation methods, which achieves the best performance.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORKS
Many previous methods have been proposed to improve the performance of FL. For example, some works propose regularizing the update of the local model to mitigate the NonIID issue (Li et al. 2020 Sun al. 2023b). Orthogonal these works, this paper focuses on the aggregation of local models. Generally, there are three main types of FL. aggregation methods.
Aggregation Weights One of the typical researches is to determine adaptive aggregation weights (Li et al. 2022a Rehman et al.) 2023). For instance, FEDAVG (Xia et al. (021) tailors weights based on distinct institutional medical datasets to enable personalized medicine, whereas L2C (Li et al. 2022a) identifies similar peers in decentralized FL by adapting weights using local data. While these approaches have proven effective, they primarily emphasize the creation of personalized mod- els for individual clients. In contrast, our work centers acquiring a global model. Recently, FedLAW (Li et al.] 2023) aims to obtain a global model by learning the weights. Nevertheless, all of these methods rely the proxy dataset in the server while aggregation method does not.
Model Fusion Due to the permutation invariance of neural network parameters, some works con- sider that the parameters ordering of different local models across clients may be varied especially when their data is NonIID Yu et al. 20211; (iingh & Jaggi 2020; Li et al., 2022b). In this case, the coordinate-wee average of local models will lead mismatch between the same-position param- eters of cross-client local models, degrading the performance of the aggregated model. Hence, these works seek fuse these local models by re-ordering the parameters to match them across clients such as using Hungarian matching algorithm (Wang et al. 2020a), Bayesian approach (Yurochkin al. 2019b), or graph matching algorithm (Liu et al. 2022).
Federated Distillation Different from the two above types of methods that compute the average of model parameters, federated distillation employs an ensemble distillation computing the average of their logits over the aggregation of local models (Wu & Gong 2021 Guo et al. 2020 Bistritz al.) 2020 Wang et al. 2023). Notably, [in et al. (2020): & Chao Chao (2021) initially ly introduced tech- nique that harnesses knowledge distillation on the server side. This approach transfers knowledge from multiple local models to the global model using an unlabeled proxy dataset. However, these methods depend on the availability of an auxiliary dataset on the server, which may not be present in real-world scenarios. In response to this limitation, recent studies (Zhu et al.) [002]] Zhang et al.) 2022 Wang al. (202)) proposed replacing the proxy dataset with generated data, enabling en- semble federated distillation in a data-free manner. We in this paper focus on the average of model parameters, which is orthogonal to these works.
SETUP
Federated learning allows N clients with server to solve the following optimization problem:
min F(w) WER
N Σ F.(w), s.t., F. = Ee
obtain the global model w. The function F. (w) Rd + denotes the expected loss over the data distribution of client n. D, denotes the data distribution of the n-th client. fa(w; g) denotes the loss value with respect model w and random data sample E. Without causing confusion, we use fr (w) to denote a mini-batch of fa (w: {) for simplicity. Besides, we make the following assumptions for these objectives which are widely adopted in FL (Dinh et al. 2020 Wang et al.) 2020b).
Assumption (L-smoothness)) The objective function F,, is L-smooth with Lipschitz constant L i.e., [w) "VF,(w') L.[lw- 22 all w, w'.
Assumption 2 (Bounded Variance). For all parameters the variance of the local stochastic gradient in each client is bounded by of: B(IV VF, (w))3) S of. Besides, the global variance of gradients among clients is bounded by oz: 1 EN || oz
N
Assumption (Bounded Gradient). For all parameters w. the stochastic gradient with respect to the loss is bounded by constant M: [[[I fn (n)) & M
3

---

Published as a conference paper at ICLR 2024
4 METHODOLOGY
In this part, we will introduce our proposed aggregation method. To minimize the objective first apply Assumption Ito each local loss function F (w):
we
1
min ((W) WE WER"
(x)
[n V Ww w") + + L w" w (2)
Then, we turn to minimize the upper bound of the objective function (T) which corresponds to the right-hand side term of the inequality (2) by aggregating local models w" of each client n from multiple rounds. Given the set of recent local models W = [w? f, f.:....
.....,,,, w of each client n
obtained in multiple rounds, the server seeks to solve the following objective:
Σ [Fa("") Vwn (n w") L
min
w
(3)
wEBd w/ EWW
EWE
The problem 3 is strongly convex in terms of w for any combination of the local models w". Therefore, the global model has closed-form solution with respective to the local models w":
Σ V Fw"""
Given equation (4). the problem 33 is equivalent to combinatorial optimization problem to select local models w However, solving this problem requires computing the full gradient w F,, (w*) on the local dataset of each client leading to extra expensive computation and communication cost. Considering the local model w" may nearly approach one of the local optima or saddle point w especially at the end of the FL training stage when the number of local epochs is large,
take an approximation as V Vw (w"" 2 0. The problem
(3) can be ce-fomulated as:
N
L
N
N
min wiEWl W EWN
Fn (w")
w
w
12
s.t., W
w
(5)
N
2N
N
==1
==1
R=1
N
N
N
1
min W'EW! .... EW N
Σ
Fu
Σ
|
2 W
s.t., W
w"
(6)
2N
N
n=
Equation ( reveals that the criterion for choosing local models can be understood as the selec- tion of cross-round local models that exhibit minimal divergence among each other, i.e. variance
N E ||w 1..... particularly when the difference in loss F, (w") tends to be small among clients. Although solving (5) can obtain the optimal combination of cross-round local models, the computation complexity and memory cost are large. An approach reducing the computation cost is to utilize the equivalent version of ( i.e., (6). which is derived using - (x; z)² ==
- E xt _ with J Σ Ti. In this way, the la norm of ||w"|B || can be cached once it is computed to avoid repeated computations. Yet, the search space for all combinations still large. Denoting the model size as C, we have the following conclusion.
Proposition 1 The computation complexity of solving 6 is O(KM) and the memory cost KNC.
Due to the exponential complexity of computation, directly solving (6 is not affordable even by a cloud for large K and N. Therefore. further propose selecting local models with approximately minimum divergence to reduce the cost. Our strategy includes two steps.
First, selection for partial clients. We propose only selecting local models from P clients n E P, that participate in the current round and fixing the local models of other clients n E N - Pe by using those selected in previous rounds:
min wT WWW NHEP,
Σ C.(w")) )
Σ REN-P.
Ln(w))
1. || 2
N
N
NEP.
Fixed in current round
where the aggregated model w remains the same as w N EN w and (n(")) denotes C, (w") = F., (w") + 1 [[] As the local models of non pontpatiiiipainng clients are fixed in the current round, this leads to great reduction in computational complexity and memory requirements.
4

---

Published as a conference paper at ICLR 2024
Proposition? 2 The computation complexity of solving 7 is O(K)) and the memory cost is K
Second, batch-based selection. To further reduce the computation complexity, we propose select- ing local models in a stochastic greedy manner. Specifically, we randomly group participated clients into B equal-size batches P. = Pi U .... P. and select local models for these clients batch by batch. When selecting local models for clients in the b-th batch, the local models of clients in batches to b are fixed and batches b + to are excluded, and the objective
min ""EWW,BEEP' N P+ bP
1
Σ (n(w")+ + nEP
Σ
La La((w")
L w 2
(8)
EEIN PPP
Fixed the 6-th subset selection where the model w is aggregated by computing the average of local models of non- participated clients and the 1-st to b-th batch of participated clients, i.e., w w". This can also be viewed as selecting local models that are N-P+ 25 -
close that of clients participating in previous rounds and thus maintaining the memory of their data. The complexity of the computation is further reduced.
Proposition 3 The computation complexity of solving
( is O(BK B) and the memory cost is KPC.
While the computational complexity remains exponential, we retain the flexibility to manually adjust the value of B for control. In practice, we can maintain f/2 as constant, effectively reducing the complexity to an acceptable level. In an extreme scenario, we can set B = P, resulting in linear complexity with respect to the value of K. Our experiments have shown that even small value of K, such as K 3. produces satisfactory performance, rendering the computational complexity acceptable for practical applications. Additionally, the memory cost K KCC is also manageable when K' is small, because the number of sampled clients P is usually a small ratio of the total clients. The local models of non-participated clients can be stored on the disk which has sufficient storage space. For example, 1TB hard drive can store approximately 20, 000 copies of ResNet- 18, which is widely adopted on the edge. Given that even a mobile phone is equipped with 1 TB storage, we believe that the cost is within the budget of the aggregation node which is typically hosted by cloud.
As compared to other aggregation methods like weight setting (Li et al. 2023) or ensemble distil- lation Lin al. (2020); Chen & Chao (2021), our approach has a distinct advantage. We do not depend on an additional public dataset, which can be challenging to acquire due to the requirement for a similar distribution as the global dataset. Moreover, our method does not introduce higher computational complexity compared to existing methods. Many existing aggregation methods in- volve performing gradient descent (Li et al. 2023 Chen & Chao 202T) or solve maximum bipartite matching problems (Wang et al. 2020a), which can be computationally intensive.
4.1 FEDCDA ALGORITHM
The complete procedure of our method is given in Algorithm T by assuming the base algorithm is FedAvg |McMahan et al., 2017a). FedCDA differs from FedAvg primarily in lines 9 and 10 of its implementation. When it receives local models from a subset of clients, the server updates its cached local models. This update involves replacing the oldest round's local model with the most recently received one, as indicated in line 9. After this update, the server selects local models for aggregation by solving the problem (8) in line 10. Finally, the global model is obtained by averaging both selected and fixed local models to retain knowledge contributed by all clients.
In practice, we usually apply FedCDA after warmup training stage using FedAvg or other base- lines to ensure that the local models can approach the convergence during the local training process. It is also worthwhile to note that most existing methods usually employ improved techniques over the step of line 11, e.g., re-setting aggregation weights (Li et al. 2023) or using ensemble distilla-
tion (Lin et al. 2020 Chen & Chao 2021), which orthogonal
5 THEORETICAL ANALYSIS
In this section, we provide theories for better understanding the principles and bounding the error of the proposed algorithm. We first prove that the selection error of using the approximated objective
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 FedCDA Algorithm
Input: Number of cached local models K, number of subsets B, learning rate 7), number of sam- pling clients P. and total communication rounds T'.
Output: Converged global model w.
1: Initialize the model parameter Wo:
2: Distribute Wo to all clients;
3: for each communication round / E 21. do
4: Randomly select a set of clients Pz;
5: for each selected client P. in parallel do
6: Initialize the local model with the received global model: w = Wei
7: Solve the local problem by updating w,, for E local mini-batch SGD steps and accumu-
late the local loss Fx (w") in the last local epoch: w" = w - 7 (w");
8: Update the cached set W for E Pe by replacing the oldest model with received w";
9: Select local models w" for each client n E P3 by solving the problem 8
10: Aggregate both selected and fixed local models to obtain: Wr+1 N 2 w";
return global model WT
(5) to the exact objective (3) is bounded by the convergence degree of local models. Then, present the benefits of FedCDA on idealized cases and conditions. Finally, we establish the convergence theories for our algorithm.
Theorem (Approximation Selection Error) Define the local optima closest to w as w and the maximum distance between any two local optima that are close cached local models across clients and rounds as D, i.e., D) maa E[M) (ile) SE(K) (llw?," -w n' II). If the distance between the local model wT and its approximated critical point w." is limited by a constant e O,
i.e, wT de.l
w, a.* < e, then the disparity in the global loss between aggregating local models selected
using and (3) is constrained by S ALLe2 2LDe.
The proof can be found in Appendix B.I The theorem indicates that the approximation error of using (5) 33) becomes smaller when local models are convergent. It implicitly reveals that our algorithm may obtain a better global model when the local models approach convergence, i.e., with large local iterations large warmup rounds, which are verified by our experimental results in Figure [ and Figure 6.22 Further, we seek to show that minimizing (3) leads to lower global loss than naively aggregating the local models in the newest current round. We define the divergence among local models w" Vn = 1....
1 ll N En- w -w" 11 We denote Similarly, we denote w, as the t-th round global model aggregated from all f-th round local models w w., = I...., N. Subsequently, we demonstrate that the global loss F F(w..) can be assured to be lower than F(w,) under the condition that the divergence Var(w?.) is less than Var(w?) ) by certain value.
: N as Var ()
N Em
We. w n 1...., .N as the solution of objective (3)
Theorem (Impact of Divergence of Local Optima) Let the definition of the local optima w": and distance be the same as Theorem 1. Consider the loss function F.(w) is strongly con- vex with a parameter p within the region spanning from the local optima w n., to the global model w. and the local loss achieves equivalent values on local optima in different rounds, i.e., Fa Fa(w."") = "==""""=" = =F,("";; ".*"" the divergence among selected local models is small enough, i.e. satisfy- ing Varlw?,") ".:") S Var(w""") (4 1)e², then the global loss of using selected global model w. is smaller than that of using t-th round global model wi i.e., F(W..) S F((w.)
Although the conditions of Theorem may be idealized in practical settings, it provides some insights for understanding our method. Smaller divergence among local models leads to a smaller loss of the aggregated model. An ideal case is that the divergence is reduced to 0 where the local optima of all local models across clients are the same. In fact, such an ideal case can widely exist in overparameterized deep neural networks, where a large model may achieve 0 loss in the local dataset of each client and hence is the local optima of all clients. Therefore, our method may prefer large models. The experimental results in Table [] also verify statement, where the improvement is higher for the larger models. Although
The proof can be found in Appendix B.2

---

Published as a conference paper at ICLR 2024
motivation mainly comes from the non-convex functions where there are multiple local optima. our algorithm is also applicable to convex cases. More discussions can be found in Appendix [B.3] Finally, we present the convergence of our algorithm. Noting that even though our algorithm does not achieve faster theoretical convergence using existing optimization analytical tools, our algorithm demonstrates great empirical benefits.
(Convergence on Non-convex Functions) Consider problem ( under Assumption 22 and If the learning rate yj satisfies < 1 S LE then the global model W., solved by (3) achieves asymptotic convergence, i.e., T E I F(
Theorem 3
O
Our proof mainly includes two parts. First, we prove that the difference between the loss of the global model w. obtained by (3) and that of the reference global model w, obtained by aggregating the newest local models is bounded. Then, we prove that the loss of the global model w, achieves convergence, which in turn indicates the convergence of the global model w., Detailed derivations are deferred to Appendix B.4
Ideas of Proof:
6 EVALUATION
6.1 EXPERIMENTAL SETUP
Datasets and Models: We consider three popular datasets in experiments: Fashion-MNIST (Xiao et al. 2017), CIFAR-10 10 (Krizhevsky et al. 2009) and CIFAR- 100 (Krizhessky et al. 2009), which contains 1O, 10, 100 classes respectively. For CIFAR-10 and CIFAR-TO0 datasets, we use ResNet- 18 (He et al. 2016) as the backbone to train and test the performance while for Fashion-MNIST we use a simple CÑN instead. The simple CNN has two 5x5 convolution layers (the first with 32 channels, the second with 64, each followed with 2x2 max pooling), fully connected layer with 512 units and ReLu activation, and a final fully output connected layer.
Data Partition: To evaluate the performance of our work in a heterogeneous scenario, we specify two Non-IID data partition methods called Shards (McMahan et al. (017a) and Dirichlet (Lin al,) 2020). In the Shards setting, the sorted samples are shuffled into N S shards, and assigned to N clients randomly. Each client owns an equal number of pieces. In the second setting, data distribution over clients satisfies the Dirichlet distribution by using to characterize the degree of heterogeneity. We set a of Dirichlet: {0.1, 0.3, 0.5) and shards for each client: {2, 4, 8).
Beside of FedAvg (McMahan al. 2017a), also compare against various types of efficient federated learning approaches with the proposed method in our experiments. The first main type includes typical aggregation methods that speed FL in the local process tuning learning rate, including FedProx (Li et al., 2020). FedExP (hunjhunwala et al.) 2023). and FMdSAM (Qu et al. 2022). The methods of the second type can be divided into three main representative aggre- gation categories: ensemble distillation including FedDF (Lin et al.) 2020) and FedGEN (Chen &
Baselines:
Chao 2021); model fusion including FedMA (Wang et 2020a) and GAMF (Liu et 2022
weights setting including FedLAW (Li et al. 2023).
Implementation: We implement the whole experiment in a simulation environment based on Py- Torch 2.0 and 8 NVIDIA GeForce RTX 3090 GPUs. We use 20 clients in total and randomly choose 20% each round for local training. We set the local epoch 20, batch size to 64, and learning rate to 1e-3. 3. We employ SGD optimizer with momentum of le - 4 and weight decay of le - 5 for all methods and datasets. At the same time, we set the number of global communication rounds to 200. Each experiment setting is run twice and we take each run's 's final 10 rounds' accuracy and calculate the average value and standard variance. For our method, we also need to set the memory size the client K to 3. batch number B to 3, and the number of warmup rounds to 50. Besides, we simply assume L = 1 for all clients to save the computation cost.
6.2 EXPERIMENT RESULTS
Performance Comparison. We report the comparison results with other baselines in Table The results with broader range of hyperparameters can be found in Appendix(D In order to demonstrate the generalization of our method, we compare them on two different Non-IID settings, Shards and Dirichlet distribution. We apply different data distributions our proposed FedCDA achieves the best performance on almost all settings. It demonstrates the effectiveness and benefit of cross-round divergence-aware aggregation. Specifically, relatively
different datasets. We can that

---

Published as a conference paper at ICLR 2024
Table 1: The comparison of test accuracy of different methods. The best results bolded.
Method
Fushion-MNNIS((*))
CIFAR-10(%)
CI0AR-100($)
Shards (S)
2
4
2
4
2
4
FedAvg
64.692568
74.78s455
76.8 23.33
28. 1023.96
59.83x294
70.87=1.91
11. 11
21.9 15.87=1.00
FedProx
64.21=4.1
70.76
72. 19
26.39
53.03e229
40.91=147
10.87
15.37
24. 16
FedExP
65.24-147
69.31
76.66.504
26.84.478
59.31-361
69.53-1.4
11.59.a8:
16.47.0.99
13.58
FedSAM
59.28s0.15
75.19
76.07amm
29.3
57.12z00s
61.56=0031
11.192016
15,95a0.1
22.44.0.16
FedDF
64.72a2.11
74.16+1.52
85,51
32.37
60.08e5.7
71.52a2.67
11.63at07
17.12a1.12
25. 25.84+1.02
FedGEN
63.50-3.37
69.42.4.0
80.17+421
27.21
57.16x271
68,93-1775
10.07-0
15.26.0.19
11.49
FedMA
64.71
74.98a5.00
77.13
28.61
59.97
10.91
11.89
15,90s0.2
22.02x0.22
GAMF
64.97
75.2
77.34
28.92+1.52
60.23a1.93
71.4421775
11. 11.98
16.76
24. 24.15
FedLAW
30.34
73.934491
57.53+3.2
26.32
46.81
61 08-2682
11.57
15. 15.994m.49
22.37
Ours
66.30.007
76.59
78.99
34.97
62.81:024
72.04.0.xx
12.20
19.9
28.16.0.30
Dirichler (c)
0.1
0.3
0.5
0.1
0.3
0.5
0.1
0.3
0.5
FedAvg
71 71-81-961
75.97.1.11
79.73+194
401.43+1.88
61.11-26
67
30.13-0 to
35.73-0.56
38, 88.86
FedProx
70.44.3.87
72.17s4.10
19.24
38.98a5.01
61.6421.22
70.16:2.0
32.9621
a0.81
42.5330.48
FedExP
73.42st.2
76.5741.9
80,22a3.7
60.63s4.32
70.22+2.0
74.37a1.1
16.76.1.1
18.18
47,80
Fed8AM
71 71.65-00r
75.91
77.67-u
49, 29.9640.20
59,534u119
64.54-021
21. 12.54
24.72-0.18
28.59
FedDF
80.03.1.44
84,42.00e
86.84.1.9
$4,28x2.9
69.851sw7
73.76.2.7
34.76.nm7
39.42.1.12
1..3
FedGEN
73.02a1.77
77.48+3.00
81.762421
47.09
64,90s271
68.74.1.75
29.02an19
38.54.0029
1
FedMA
71 77.87
75.89:4.1
80,12a325
49, 98
61.32a217
68 68.42a1.95
30.02ah15
36.21
59,555
GAMF
12.11.5.66
76.24
80.55.20s
11.2
63.45-1m
30.14.1.1
31.12
37.26.0.76
41.25
FedLAW
71.93:8.3
76.88
79.98s1.
48.9
61.502229
67.08a1.75
32.01:261
38.80.220
40.1
Ours
78.63e0.14
84.67
87.01
62.46
70.27sm29
74,96-0.17
39.38a125
45.86
49.31
425
Text Areuma
spasdrrsl
sparal
spoch=10
opocha10
apoch+20
spoch=30
local apoch-30
local spoch-40
looa opochr-00
CIFAN--0 (o)
Communiction
10 190 215
200
25
Communtation
1s 100 15 30
13 19 120
Fourds
(c) Local ep.s (FedCDA) (d) Local ep.s (FedAvg) Figure 2: (a) shows the effect of different aggregation strategies. (b) shows the impact of the memory
(a) Aggregation strategies
(b) Memory size K
size K on FedCDA. (c) and (d) present the impact of local epochs (cp.s) on FedCDA and FedAvg.
larger datasets such as CIFAR-10 Dirichlet 0.1, FedCDA with ResNet- 18 achieves 62.46% accuracy whereas the best baseline method FedExP achieves 60.63% accuracy. In addition, FedCDA with simple CNN also makes improvements on relatively smaller datasets, although the improvement is less than in large models. At the same time, we can also see that the results of our method on relatively small datasets and simple CNN are not the best, which may be because the features of models with different rounds are more similar on small datasets and simple models, and can not provide more aggregation features to accelerate convergence. In conclusion, we can notice our FedCDA makes more improvements on the large model and complex datasets.
More Comparison Results with Different Hyper- parameters. To compare with baselines in a compre-
Table 2: Results of FL with 100 clients.
Method
Dim
Dino.3)
Dir(0.5)
hensive way, we further conduct experiments on differ-
FolAvg
38.89%
42.23%
FedProx
39.86% anar
39.48fuua37
ent hyper-parameters. The number of clients is 100 with
FedEaP
38. 38.01%
44. 14.10%ais99
41.79%:11x2
the sample ratio being 10%. The learning rate is set to be 0.1 with the weight decay being 1e-3, and the number of local epochs is 5. The local optimizer SGD with- out momentum. The experiment is conducted by running the R on the CIFAR-100 dataset. The results are shown in Table 2 As can be seen, our method still performs the best. Specifically, when the data is the most heterogeneous, i.e., with Dirichlet a = 0.1, FedCDA achieves the accuracy of 47.38% which outperforms the best baseline method FedDF by 6.34%.
FedSAM
16.35%
20.5366
25.70%
FodDF
41.04% ausms
47.06f6ua54 39.918ta1.0s
47.63%
FedGEN
41 1.66666.10 105
43, 1ems
FedMA
39.12% anses unssss
40.4266ure
42.89%
GAMF
39.89%
10.989iu..19
FedLAW
40.88%
41.77%6-arss
41.89%
Ours
47.38%
49.96/6u0115
50.04%
Different Aggregation Strategies. We compare five aggregation strategies on the CIFAR-10 datasets. Because the optimal selection for updates method is an exponential method, we only sam- ple 10 clients in each round, where the sample ratio is 0.3 and the client memory size is K = 3. As shown in Figure 2(a) we compare FedAvg with different aggregation strategies in our method. The
8

---

Published as a conference paper at ICLR 2024
clients in FedAvg do not require memory. The newest strategy is that only the newest local model of each client is aggregated during the server aggregation phase. The random strategy is that during the server aggregation phase, we randomly select local model from multiple rounds to aggregate. Finally, the approximate strategy is as shown above and the optimal one is as 3 We can find that the approximate and optimal strategies have huge performance improvement over FedAvg, newest and random strategies with ResNet-18 on CIFAR-IO Dirichlet 0.1, 0.3 and 0.5. At its peak, there is an almost 10% increase over FedAvg. We can also see the performance of approximate performance is about the same as the optimal one, but the convergence time of the former is much smaller than that of the latter. In fact, the former is actually a greedy algorithmic approximation of the latter, so the computation of the solution is greatly reduced. We also compare the average polymerization time of each round of these aggregation strategies. Details are in the Appendix C
We compare the test ac- curacy of client memory size K for 1, 2, 3, 4 on CIFAR-10 Dirichlet 0.1. As K increases, the final test accuracy increases which confirms our theory. The increase of K value gives cross-round polymerization more choices and possibilities.
Hyperparameters Sensitivity. As shown in the following figure 2(b)
Comparison with Different Epochs for FedCDA and FedAvg. By comparing the results in Figure 2(c) and Figure [(d] from the ver- tical perspective,
Tod Bacusecy
FedCDA eventually converges with increasing test accuracy with more local epochs while the final convergence ac- curacy of Fedavg remains roughly unchanged. This proves that our algorithm can tolerate large local interactions to save communication cost. Horizontally, FedCDA: converges rapidly and stably, whereas the convergence curve of FedAvg is very oscillatory. The reason is that our method excludes the negative impact of sampling clients while FedAvg cannot. Therefore, the convergence of our method is more stable than FedAvg.
bagghr4
bygghed
Communieatin Rounds
Figure 3: Impact of Batch
Effect of Batch Number. As we can see in Figure 6.2 different batch numbers have little effect on the final precision result. Our experiment setup 50 clients and 10 sample clients on the CIFAR-10 10 Dirichlet 0.1. We compare the results for batch B = 2, 4, 5, 6, 7. The results show that the approximation selection can keep the accuracy closer to the optimal selection.
Warmup Analysis. The experiments of Figure are conducted on settings of CIFAR-10, 20 clients, and the sample ratio 0.2. FedCDA with no warmup rounds is worse than some with warmup rounds.
Ton Rocd 2
This is because the local models in the early rounds can not ap- proach convergence. The combination with not-well-cowverged local models in old rounds may prevent the training of the global model. Therefore, it is similar to FedAvg in the startup training stages. Its advantages gradually exhibit with the training proceeds and outper- forms FedAvg (warmup=200), Yet, there is a threshold for raising warmup rounds. Specifically, we can see that FL with 150 warmup rounds has worse performance than 100 warmup rounds. The principle behind it is that the local models and the global model have approached convergence in the final training stage. The difference between local models in different rounds is greatly reduced and thus the combination of them gains little benefits.
Efficil pard
Communcatoon Rounds
Figure 4: Impact of Warmup
CONCLUSION
This paper targets aggregation in federated learning, addressing the issue that traditional single- round methods may not preserve locally learned knowledge due to statistical heterogeneity. Recog- nizing clients' convergence post-startup stage and local models' consistent data fitting across rounds, we propose FedCDA- a new method that selectively aggregates cross-round models with minimum divergence. To enhance efficiency, we introduce an approximation selection algorithm. Theoretical convergence is proven and empirical results show our method outperforms state-of-the-an baselines.
This paper addresses the ideal scenario where smoothness value is equal among clients. The goal is to improve our method for cases with varying smoothness by refining objectives and incorpo- rating sharpness in cross-round aggregation, considering the sharpness of their local optima, despite flatter optima often correlating with better generalization.
currently treat all local models equally without

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The research supported under the National Key R&D Program of China (2022ZD016020) and the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contributions from the industry partner(s). This work is supported by National Natural Science Foundation of China under grants U1836204, U1936108, 62206102, and Science and Technology Support Program of Hubei Province under grant 2022BA
REFERENCES
Muhammad ammaddud-din, Elena Ivannikova, Suleiman Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. Federated collaborative filtering for privacy-presente personalized recommendation system. CoRR, abs/1901 .09888, 2019.
Ilai Bistritz, Ariana Mann, and Nicholas Bambos. Distributed distillation for on-device learning. In Proceedings of Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems, NeurIPS, 2020.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gra- dient descent into wide valleys. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. April 24-26. 2017, Conference Track Proceedings, 2017.
Hong-You You Chen and Wei Chao. Fedbe: Making bayesian model ensemble applicable to feder- ated learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Proceedings of Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurlPS, 2020.
Pengfei Guo, Puyang Wang, Jinyuan Zhou, Shanshan Jiang, and Vishal M. Patel. Multi-institutional collaborations for improving deep learning-based magnetic resonance image reconstruction using federated learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 2423-2432.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR, pp. 10200-10029, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778. IEFF, 2016.
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 586-594, 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinsse et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3551-3356, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of 2009.
features from tiny images.
Shuangtong Li, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Learning to collaborate in decentral- ized learning of personalized models. In IEEE/CVF Conference on Computer Vision and Pattenn Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 9756-9765, 2022a.
10

---

Published as a conference paper at ICLR 2024
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020, 2020.
Xin-Chun Li, Yichu Xu, Shaoming Song, Bingshuai Li, Yinchuan Li, Yunfeng Shao, and De-Chuan Zhan. Federated learning with position-aware neurons. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10072-10081, 2022b.
Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pp. 19767-19788, 2023.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2366 2020.
Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learn- ing. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pp. 13857-13869, 2022.
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain gener- alization medical image segmentation via episodic learning in continuous frequency space. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 1013-1023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communicatoon-ffiiien learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS, 2017a.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicatinn-fficient learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017b.
Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pp. 18250-18280, 2022.
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated learning for emoji prediction mobile keyboard. CoRR, abs//906. 04029, 2019.
Yasar Abbas Ur Rehman, Yan Gao, Pedro Porto Buarque de Gusmão, Mina Alibeigi, Jiajun Shen, and Nicholas D. Lane. L-DAWA: layer-wise divergence aware weight aggregation in federated self-supervised visual representation learning. CoRR, abs/2307.07733 2023.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Yan Sun, Li Shen, Tiansheng Huang, Liang Ding, and Dacheng Tao. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023a.
Yan Sun, Li Shen, and Dacheng Tao. Understanding how consistency works in federated learning Anmual Conference on Neural Information
via stage-wise relaxed initialization. Processing Systems (NewrIPS 2023), New Orleans, LA, USA, Dec 10-16, 2023b.

---

Published as a conference paper at ICLR 2024
Chunnan Wang, Xiang Chen, Junzhe Wang, and Hongzhi Wang. ATPFL: automatic trajectory pre- diction model design under federated learning framework. In /EEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR, New Orleans, LA, Jue 18-24, pp. 6553-65622 2022.
Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain-aware federated knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, PP. 20412- 20421, 2023.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopooloss, and Yasaman Khaza- eni. Federated learning with matched averaging. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Informa- fion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020. December 6-12, 2020, virtual, 2020b.
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, 2021.
Lei Wu, Zhamxing Zhu, and Weinan E. Towards understanding generalization of deep learning: Perspective of loss landscapes. CoRR,
2017.
Yingda Xia, Dong Yang, Wenqi Li, Andriy Myronenko, Daguang Xu, Hirofumi Obinata, Hi- toshi Mori, Peng An, Stephanie A. Harmon, Evrim Turkbey, Baris Turkbey, Bradford J. Wood, Francesca Patella, Elvira Stellato, Gianpaolo Carrafiello, Anna lerardi, Alan L. Yuille, and Holger Roth. Auto-fedavg: Learnable federated averaging for multi-institution medical image segmen- tation. CoRR, abs/2104. 10195, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs//108.0777 07747,7777777 2017.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamies: Stochastic gradient descent exponentially favors flat minima. In 9th International Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
An Xu, Wenqi Li, Pengfei Guo, Dong Yang. Holger Roth, Ali Hatamizadeh, Can Zhao, Daguang Xu, Heng Huang, and Ziyue Xu. Closing the generalization gap of cross-silo federated medical image segmentation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 20834-20843.
Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. Fed2: Feature-aligned federated learning. In KDD '21: The 27th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, I, pp. 2066-2074, 2021.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan H. Greenewald, and Trong Nghia Hoang. Statistical model aggregation via parameter matching. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 10964-10964, 2019a. 9a.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan H. Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 7252-7261, 2019b.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yr Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10164-10173, 2022.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning, pp. 22878-12889 PMLR, 2021.
12


Reviews:
[{"review_id": "QmY1lOAeMT", "reviewer": "Reviewer_erYc", "summary": "Traditional federated learning aggregation methods only consider the local model from the current round, failing to account for differences between clients. The proposed FedCDA method selectively aggregates local models from multiple rounds for each client to decrease discrepancies between clients. By aggregating models that remain aligned with all clients' data, FedCDA can better preserve each client's local knowledge in the global model, outperforming state-of-the-art aggregation baselines as shown in experiments.", "questions": "- How to understand that there are varied received global models and they result in local models? (Quote from Introduction: \"each client often converges to different models due to varied received global models and multiple local optima of deep neural networks”?)\n- Do the local models achieve different local minimal only due to their own private local data?\n- Why are there varied received global models?\n- Authors claim that selectively aggregating them can make the global model approach local models more closely and affect the forgetting of local data knowledge. \n- Does Equation (2) have used the approximation (∇wn Fn(wn))’ ≈ 0 ?\n- What is the meaning of N-P+bP/B? why add the #nums unselected N-P?\n- why selectively aggregating them can make the global model approach local models more closely and affect the targeting of local data knowledge? \n- Authors seem to use the random sampling strategy to minimize the eq.7, i.e. selectively aggregating multi-epoch local models, what if P=N?\n- The authors aim to make the global model approach any of the different local models in multiple rounds, how do define the “approach”, and how approach between the multi-local epoch global model and last-epoch local models?\n- What differences between this work “Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization. Dacheng Tao et.al .Nips2023.”", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well organized, and it is mostly easy to follow the different parts.\nThis paper makes the multi-epoch aggregation model problem equivalent to a combinatorial optimization problem.\nThe method seems novel to me, however there may be other methods that include a similar mechanism which I have missed.\nSufficient theory illustrations well explain the objective of the proposed FedCDA.", "weaknesses": "- The variables in the formula are very complex, and a lot of space was spent on deriving the objective function. At the same time, it is still difficult to understand why selectively aggregating them can make the global model approach local models more closely and affect the targeting of local data knowledge? \n- It seem that the caption of Fig(2) is wrong.\n- There seems to be a lack of discussion on memory and communication costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YsURHFjotK", "reviewer": "Reviewer_Pshb", "summary": "This paper introduced FedCDA to solve statistical heterogeneity issues, especially the discrepancies between local models. The principle behind FedCDA is that the local model from each client may converge to different local optima over rounds. Based on this principle, the author utilizes a local model from multiple rounds to minimize the divergence from other clients to make sure that the aggregated global model remains aligned with selected local models. Empirical results show their algorithm’s superiority to other benchmark methods.", "questions": "1.\tWhy not compare with those methods that personalize partial model parameters for each client? Personalized Federated Learning can help local models adapt to local data distribution.\n2.\tFedCDA maintains a consistent global model that will not interfere with those outlier local models as the authors selected the closer local model to do aggregation. The potential problem is that the aggregated model may not adapt to those outliers and be quite sub-optimal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed algorithm addressed the data heterogeneity issues from a novel view, considering the aggregation of cross-round local models. \n2. This paper provides theoretical insights and the convergence bound of the proposed method.", "weaknesses": "1.\tIn FedCDA, each local client needs to store a set of recent K local models, which requires a much higher demand of memory space compared to other FL methods.\n2.\tAlthough FedCDA employed the approximation strategy, communicating the model set is still too heavy and the considerable extra communication may be a severe problem when there exists a large number of engaging clients.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zpHGCpo1ru", "reviewer": "Reviewer_jkW8", "summary": "This paper focus on designing a model aggregation strategy for federated learning. Different from the traditional methods that aggregate local models from the current round only and may cause diversity issues, this paper introduces FedCDA, which selectively combines models from different rounds to maintain knowledge diversity. Experiments show that FedCDA outperforms the existing model aggregation methods.", "questions": "In Theorem 3, the authors claimed that convergence is achievable but did not provide a specific rate of convergence. It would be beneficial for the authors to elucidate the relationship between the convergence rate and parameter $T$ to enhance the assessment of the theoretical contribution when compared to other algorithms.\n\n*Post-rebuttal comments:*\n\nWhile some of my concerns have been addressed, several questions still remian.\n\n1. The author's depiction of motivation in Figure 1 still appears somewhat speculative. The clients trained based on local data, and the rationale behind the considerable variation in the direction of updates across clients in different rounds remains unclear.\n\n2. The proposed method incurs a substantial storage cost, significantly constraining the practicality of the algorithm.\n\nOverall, I will maintain the original score.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is easy to follow.\n2. To the best of the reviewer's knowledge, the approach presented in this paper, which involves the selective aggregation of local models from various rounds to mitigate disparities between them, appears to be relatively novel.\n3. The author conducted an extensive series of comparative experiments.", "weaknesses": "1. First of all, the motivation about Figure 1 is difficult to understand, where the authors attempt to illustrate the usefulness of cross-round local models. But according to the reviewer's understanding, the model aggregation strategy of Fig. 1(b) will cause the global model to overfit to the local node, resulting in extremely unstable global convergence. Further clarification is needed on why it is better to update model via Fig. 1(b).\n2. Why the optimization objective function presented in Eq. (2) is derived based on the effectiveness of cross-round local models remains unclear. The connection between the optimization objective and the effectiveness of cross-round local models requires further elucidation.\n3. Another issue lies in the optimization objective presented in Eq. (7) on the server side, where model derivatives appear to be computed for inactivate nodes. Question then arises as to whether these non-active models are stored on the server side. In federated scenarios characterized by a large number of nodes and typically large models, this could lead to substantial storage overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MDq9X0SdGj", "reviewer": "Reviewer_rxU6", "summary": "Unlike existing federated learning methods that aggregate the weights of local models of clients participating in the current round, this paper proposes an aggregation method that considers local models from recent rounds for the first time. This idea uses the fact that local models in different rounds within a client converge to different local optimal points after the warm up stage, as clients perform local training of multiple epochs to reduce communication costs. This paper also proposes several tricks to simplify the computation of which local model to select and aggregate among local models from multiple rounds. In addition to the theoretical background of the proposed method, experiments show that it has better performance compared to other aggregation methods.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The idea of selectively aggregating local models from several recent rounds is novel. It seems to have successfully overcome the limitations of existing aggregation methods by taking advantage of the fact that deep learning models have multiple local optimal points.\n\nThe feasibility and reasoning of the proposed method were confirmed through several theoretical backgrounds. Also, several tricks were presented to reduce the computational complexity of the proposed method.", "weaknesses": "The effectiveness of the proposed method was not sufficiently proven in the evaluation section, so doubts remain about the proposed method.\n\nConsidering that the proposed method is about a new aggregation method, 20 clients and a participation ratio of 20% are too unrealistic.\n\nAs shown in figure 2 (c), the proposed method is sensitive to local epoch hyperparameter because it assumes that local models converge. However, most experiments are conducted at a local epoch of 20, and such a large local epoch makes comparisons with baselines unfair. It is of course an advantage that the performance of the proposed method is good even when the local epoch is large, but this does not mean that simply increasing the local epoch is always good. \n\nIn addition to the local epoch, I believe that other hyperparameters used in the experiment are not suitable for comparison with baselines.\n\nFor example, the learning rate was set to 1e-3, which helps the local model to converge quickly with a large local epoch. However, it is unclear whether such a small learning rate and large local epoch are suitable for other baselines.\n\nBased on the code included in the supplementary material, I reproduced the FedAvg experiment of CIFAR100 - Dirichlet (0.5) in Table 1. Using the same hyperparameter presented in the paper, FedAvg showed an accuracy of about 38%, which is almost similar to 38.86% recorded in Table 1. However, when lr is 0.1, weight decay is 1e-3, local epoch is 5 and the same experiment was performed with SGD without momentum, the accuracy was about 49%, which is almost similar to the proposed method (49.31%). This does not mean that the proposed method shows similar performance to FedAvg, but I just think that the hyperparmaters should have been chosen a little more carefully. (If possible, can you run the experiment again with the hyperparameters mentioned above?)\n\n\nThe overall writing quality of the paper is not good. There are many incorrect parts in the notation related to formulas and figures in the paper, and many sentences require revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)