Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning
Authors: Yun-Hin Chan
Rui Zhou
Running Zhao
Zhihan JIANG
Edith C. H. Ngai
Venue: ICLR 2024 poster (2024)
Keywords: Federated Learning
heterogeneity
convex optimization.
Primary area: general machine learning (i.e., none of the above)
TL;DR: This paper proposes cross-layer gradients to expand the capabilities of homogeneous methods to tackle system heterogeneity in FL.

Abstract: Federated learning (FL) inevitably confronts the challenge of system heterogeneity in practical scenarios. To enhance the capabilities of most model-homogeneous FL methods in handling system heterogeneity, we propose a training scheme that can extend their capabilities to cope with this challenge. In this paper, we commence our study with a detailed exploration of homogeneous and heterogeneous FL settings and discover three key observations: (1) a positive correlation between client performance and layer similarities, (2) higher similarities in the shallow layers in contrast to the deep layers, and (3) the smoother gradients distributions indicate the higher layer similarities. Building upon these observations, we propose InCo Aggregation that leverages internal cross-layer gradients, a mixture of gradients from shallow and deep layers within a server model, to augment the similarity in the deep layers without requiring additional communication between clients. Furthermore, our methods can be tailored to accommodate model-homogeneous FL methods such as FedAvg, FedProx, FedNova, Scaffold, and MOON, to expand their capabilities to handle the system heterogeneity. Copious experimental results validate the effectiveness of InCo Aggregation, spotlighting internal cross-layer gradients as a promising avenue to enhance the performance in heterogeneous FL.

Published as a conference paper at ICLR 2024
INTERNAL CROSS-LAYER GRADIENTS FOR EXTENDING HOMOGENEITY TO HETEROGENEITY IN FEDERATED LEARNING
Yun-Hin Chan, Rui Zhou, Running Zhao, Zhihan Jiang & Edith C.H. Ngai Department of Electrical and Electronic Engineering. The University of Hong Kong (chanyunhin, zackery, rnzhao, zhjiang cooonnct. hku.hk, chngaa i @eee. hku.hk
ABSTRACT
Federated learning (FL) inevitably confronts the challenge of system heterogeneity in practical scenarios. To enhance the capabilities of most model-homooeneoon FL methods in handling system heterogeneity, we propose a training scheme that can extend their capabilities to cope with this challenge. In this paper, we commence our study with a detailed exploration of homogeneous and heterogeneous FL settings and discover three key observations: (I) positive correlation between client performance and layer similarities, (2) higher similarities in the shallow layers contrast to the deep layers, and (3) the smoother gradient distributions indicate the higher layer similarities, Building upon these observations, we propose InCo Aggregation that leverages internal cross-layer gradients, mixture of gradients from shallow and deep layers within server model, to augment the similarity in the deep layers without requiring additional communication between clients. Furthermore, our methods can be tailored to accommodate model- wodelllhmogeneour FL methods such as FedAvg. FedProx, FedNova, Scaffold, and MOON, to expand their capabilities to handle the system heterogeneity. Copious experimental results validate the effectiveness of InCo Aggregation, spotigghing internal cross-layer gradients as promising avenue to enhance the performance in heterogeneous FL.
INTRODUCTION
Federated learning (FL) is proposed to enable a federation of clients to effectively cooperate towards
global objective without exchanging raw data (McMahan al.] 2017). While FL makes it possible to fuse knowledge a federation with privacy guarantees (Huang al. 2021 McMahan et al.] 2017]]
Jeong& & Hwang 2022), its inherent attribute of system heterogeneity (Li et al. [020a), i.c., varying resource constrants of local clients, may hinder the training process and even lower the quality the
jointly-learned models (Kairouz et al. [021 Li al.
2020a
Mohri al. 2019 Gao et al. 2022).
System heterogeneity refers to a set of varying physical resources { where R; denotes the resource of client i, high-level idea of resource that holistically governs the aspects of computation, communication, and storage. Existing works cater to system heterogeneity through a methodology called model heterogeneity, which aligns the local models of varying architectures to make full use
of local resources (Diao et al. 2021 Baek et al., 2022 Alam et al.] 2022 Huang et al., 2022 Fang
Ye 2022, Lin al.] 2020). Specifically, model heterogeneity refers
of different local
models [M Ji with M, being the model of client i. Let R(M) denote the resource requirement
for the model M. Model heterogeneity is a methodology that manages to meet the constraints
(R(M) < R, >. In the case of model heterogeneity, heterogeneous devices are allocated to common model prototype tailored to their varying sizes, such as ResNets with different depths or widths of layers (Liu et al. 2022 Diao et al. 202 Horvath et al. [021 Baek et al. 2022 Caldas ct al. 2018 Ilhan et al. 2003). strides of layers (Tan et 2022), or numbers of kernels (Alam et al. 2022,, to account for their inherent resource constraints. While several methods have been proposed to incorporate heterogencous models into federated learning (FL), their performances often fall short compared FL training using homogeneous models of the same size (He et al. 2020 Diao et al. 2021). Therefore, gaining comprehensive understanding of the factors that limit the performance of heterogeneous models in FL is imperative. The primary objective of this paper is investigate
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Deeger Layers
Deeper Layers
Rounds
OKA
0.9
0,90
0.6
Lower
0,00
0.85
CKA
0.975
Accuracy
ED with homo
0.75
0.970
0.4
Non-tit IID with home
Non- HID with
0.965
Non with hetero
Non-ll with hetera
Accuracy
0.65
0,960
1
10
20
30
40
Block
(a) The CKA similarity of ResNets.
(b) The CKA similarity of ViTs.
(c) The relations hetween CKA
and accuracy.
Figure 1: CKA similarity in different environments and the relation between accuracy and CKA similarity. (a) and (b): The CKA similarity of different federated settings. between CKA and accuracy during the training process.
(c): The positive relation
the underlying reasons behind this limitation and propose a potential solution that acts as bridge between model homogeneity and heterogeneity to tackle this challenge.
In light of this, we first conduct a case study to reveal the obstacles affecting the performance of heterogeneous models in FL. The observations from this case study are enlightening: (1) With increasing heterogeneity in data distributions and model architectures, we observe a decline in model accuracy and laver-wise similarity (layer similarity) measured by Centered Kernel Alignment
(CKA Korbllihh et al. 2019). a quantitative metric of bias (Luo et al. 2021 Raghu et al. 202 1) (2) The deeper layers share lower layer similarity across the clients, while the shallower layers exhibit greater alignment. These insights further shed light on the notion that shallow layers possess the ability to capture shared features across diverse clients, even within the heterogeneous FL setting. Moreover, these observations indicate that the inferior performances in heterogeneous FL are related to the lower similarity in the deeper layers. Motivated by these findings, we come up with an idea: Can we enhance the similarity of deeper layers, thereby attaining improved performance?
To answer this question, we narrow our focus to the gradients, as the dissimilarity of deep layers across clients is a direct result of gradient updates (Ruder) [016]] [Cen et al.] [202]]. Interestingly, we observe that (3) the gradient distributions originating from shallow layers are smoother and possess higher similarity than those from deep layers, establishing a connection between the gradients and the layer similarity. Therefore, inspired by these insights, we propose a method called InCo Aggregation, deploying different model splitting methods and utilizing the Internal Cross-layer gradients (InCo) in a server model to improve the similarity of its deeper layers without additional communications with the clients. More specifically, cross-layer gradients are mixtures of the gradients from the shallow and the deep layers. We utilize cross-layer gradients as internal knowledge, effectively transferring knowledge from the shallow layers the deep layers. Nevertheless, mixing these gradients directly poses a significant challenge called gradient divergence (Wang et al., 2020) /Zhao et al., (018). To tackle this issue, we normalize the cross-layer gradients and formulate convex optimization problem that rectifies their directions. In this way, InCo Aggregation automatically assigns optimal weights to the cross-layer gradients, thus avoiding labor-intensive parameter tuning. Furthermore, InCo Aggregation can extend to model- nooeel-homgeeneuus FL methods that previously do not support model heterogeneity, such as (eddAvg(eMMalle et al. 2017). FedProx (Li al. (020b), FedNova (Wang et al., 2020), Scaffold (Karimireddy et al. 2020). and MOON (Li et al. 202 to develop their abilities in managing the model heterogeneity problem.
Our main contributions are summarized as follows:
We first conduct a case study on homogeneous and heterogeneous FL settings and find that (1) client performance is positively correlated to layer similarities across different client models, (2) similarities in the shallow layers are higher than the deep layers, and (3) smoother gradient distributions hint for higher layer similarities.
We propose InCo Aggregation, applying model splitting and the internal cross-layer gradients inside server model. Moreover, our methods can be seamlessly applied various model- homogeneous FL methods, equipping them with the ability to handle model heterogeneity.
We establish the non-convex convergence of utilizing cross-layer gradients in FL and derive the convergence rate.
Extensive experiments validate the effectiveness of InCo Aggregation, showcasing its efficacy in strengthening model-homogeneon FL methods for heterogeneous FL scenarios.
'The detailed descriptions for CKA introduced in Appendix A
2

---

Published as a conference paper at ICLR 2024
9.22
Staped cared
0.20
0.30
3 g0.0
0.20
1.05
6M0
1400 3002
1030 1,000 Coo0
0.15
6.4
Stagek.comt
Gradbedt Volee
Gradart vakae
(a) Similarity of gradients (b) Similarity of gradients (c) Gradient distributions (d) Gradient distributions in Stage 2.
in Stage 3.
in Non-IID with hetero. in IID with homo.
Figure 2: Cross-cnvironment similarity and gradients distributions. (a) and (b): Similarity from Stage 2 and Stage (c) and (d): The gradient distributions Non-IID with hetero and IID with homo.
PRELIMINARY
To investigate the performance of clients in diverse federated learning settings, we present a case study encompassing both homogeneous and heterogeneous model architectures with CIFAR- 10 and split data based on IID and Non-IID with ResNets (He et al., 2016) and ViTs (Dosovitskiy et al. 2020). We use CKA (Kornblih et al. 2019) similarities among models to measure the level of bias exhibited by each model. More detailed results of the case study are provided in Appendix
2.1 A CASE STUDY IN DIFFERENT FEDERATED LEARNING ENVIRONMENTS
Case Analyvis. Generally,
find three intriguing observations from Table
and Figure
1
(i) The deeper layers or stages have lower CKA similarities than the shallow layers. (ii) The settings with higher accuracy also obtain higher CKA similarities in the deeper layers or stages. (iii) The CKA similarity is positively related to the accuracy of clients, as shown in Figure Ic These observations indicate that increasing the similarity of deeper layers can serve as viable approach to improving client performance. Considering that shallower layers exhibit higher similarity, a potential direc- tion emerges: to improve the CKA similarity in deeper layers according to the knowledge from the shallower layers.
Table 1: Accuracy of the case study.
Scttings
Test Accuracy
RewNer
IID with homo
81.0
Non-IID with bomo
62.3((18.7)
Non
hetern
52.3(128.77
IDD with homo
81.0
Non- IID with homo
54.8(126.2)
Non: IID with betero
50.1([30.9)
2.2 DEEP INSIGHTS OF GRADIENTS IN THE SHALLOWER LAYERS
Gradients as Knowledge. In FL, there are two primary types of knowledge that can be utilized: features, which are outputs from middle layers, and gradients from respective layers. We choose to use gradients our primary knowledge for essential reasons. Firstly, our FL environment lacks a shared dataset, impeding the establishment of connection between different clients using features derived from the same data. Secondly, utilizing features in FL. would significantly increase communication overheads. Hence, taking these practical considerations into account, we select gradients as the knowledge.
(a) Stage3 convO in Non- (b) Stage3 convl in Non- IID with hetero.
(c) Stage3 convŌ in IID (d) Stage3 convl in IID with homo.
IID with hetero.
with homo.
Figure 3: The gradient distributions from round 40 to 50 in different environments.
Similarity. In this subsection, deeply investigate the cross-enviromment similarity of gradients between two environments, i.e., IID with homo and Non-IID with hetero, to shed light on the disparities between shallow and deep layers the same stage] and identify the gaps between the homogeneous and heterogeneous FL. As depicted in Figure 2a and 2b gradients from shallow layers (Stage2.com convo and Stage3. convo) exhibit higher cross-ent cross-environment CKA similarity than those from deep layers such as Stage2. conv,, and Stage conv2.. Notably, even the lowest similarities (red lines) in Stage2.come
and Stage3.conv0 surpass the highest similarities in deep layers. These findings underscore the superior quality of gradients obtained from shallow layers
We discuss shallow layer (the first layer with the same shape in stage) and deep layers (remaining layers) within stage for ResNets and black for ViTs. The gradient analyses for ViTs are introduced in Appendix G.3
3

---

Published as a conference paper at ICLR 2024
Layer Splitting
Stage Splitting
Hetero Splitting
Central Server
Centrol Server
Central Server
Splt Confa
Clent Resources
Gotel Model
Cieent Resourres
Giobal Mode
Spiit Contie
Client Resources
Goool Mede
Ri
Spift
R
R
Ueeger
Spinting
Spteting
Spiiting
R
Sole
Rg Spill
Spiit
Aggregate
Mg Spit
Spin
Aggregate
Split
Aggrogate
Clients
Resource Constraints: (0R,)
Clients
Resource Constraines (AR,1
Cliems
Resource Constraints: (1R,1
(a) Layer splitting.
(b) Stage splitting.
(c) Hetero splitting.
Figure 5: The system architecture of three different model splitting methods: (a) layer splitting, (b) stage splitting, and (c) heterogeneous (hetero) splitting. (a): Layer splitting divides the entire model layer by layer. (b): Stage splitting separates each stage layer by layer. (c): Hetero splitting partitions the whole model in different widths and depths depending on the available resources R2 of client
relative to those obtained from deep layers, and also indicate that the layers within the same stage exhibit similar patterns to the layers throughout the entire model.
Gradient Distributions. To dig out the latent relations between gradients and layer similarity, we delve deeper into the analysis of gradient distributions across different FL environments. More specifically. the comparison of Figure 2 and Figure 2d] that gradients from shallow layers (Stage3. exhibit greater similarity in distribution between Non-IID with hetero and IID with homo environments, in contrast to deep layers Sttage3.coov/ and Stage3.comv2. Additionally, as depicted in Figure Be and Figure 3d] the distributions of gradients from a deep layer (Figure (3d) progressively approach the distribution of gradients from shallow layer (Figure ( with each round, in contrast to Figure 3a and Figure where the distributions from deep layers (Figure (3b are less smooth than those from shallow layers (Figure Bap in Non-IID with hetero during rounds 40 to 50. Consequently, drawing from the aforementioned gradient analysis, we can enhance the quality of gradients from deep layers in Non-IID with hetero environments by leveraging gradients from shallow layers, i.e., cross-layer gradients as introduced in the subsequent section.
3 INCo AGGREGATION
We provide a concise overview of the three key components in InCo Aggregation at first. The first component is model splitting, including three types of model splitting methods, as shown in Figure 5 The second component involves the combination of gra- dients from shallow layer and deep layer, referred to as internal cross-layer gradients. To address gradient divergence, the third component employs gradient normal- ization and introduces a convex optimization formulation. We elaborate on these three critical components of InCo Aggregation as follows.
Stage 1
02
65
Laser 1, +
Layer @ G
Stage
Layer G. +
3.1 MODEL SPLITTING
Stage f
To facilitate model heterogeneity. we propose three model splitting methods: layer splitting, stage splitting, and het- ero splitting, as illustrated in Figure 5 These methods distribute models with varying sizes to clients based on
Figure 4: Cross-layer gradients for the server model in InCo.
their available resources, denoted as R, In layer splitting, the central server initiates a global model and splits it layer by layer, considering the client resources R, as depicted in Figure 5a In contrast, stage splitting separates each stage layer by layer in Figure [b] For instance, Figure 5b ilustratess how the smallest client with R1 resources obtains the first layer from each stage in stage splitting, whereas it acquires the first three layers from the entire model in layer splitting. Furthermore, hetero splitting, depicted in Figure 5cc involves the server splitting the global model into distinct widths and depths for
4

---

Published as a conference paper at ICLR 2024
afou of
of
Ple e'as
dient opdiium
optimum
dient opdmium
aptmuu
octimun
cleba apemum *
*
wfte
82
-5.6
(a) Gradient divergence.
(b) Normalized gradients.
(c) Normalized and optimized.
Figure 6: A depiction of gradient divergence, as shown in Figure 6a along with its solutions. Despite the normalization portrayed in Figure [b] the impact of gradient divergence persists. To mitigate this issue, we propose convex optimization problem that is restricting gradient directions, as demonstrated in Figure and supported by Theorem 3.1
different clients, similar to the approaches in HeteroFL (Diao et al. 2021) and FedRolex (Alam et al. 2022). Layer splitting and stage splitting offer flexibility for extending model-homogeneour methods to system heterogeneity. while hetero splitting enables the handling of client models with varied widths and depths. Finally, the server aggregates client weights based on their original positions in the server models.
3.2 INTERNAL CROSS-LAYER GRADIENTS
Deploying model splitting methods directly in FL leads a significant decrease in client accuracy, as demonstrated in Table [ However, based on the findings of the case study, we observe that gradients from shallow layers contribute to increasing the similarity among layers from different clients, and CKA similarity exhibits positive correlation with client accuracy. Therefore, we enhance the quality of gradients from deep layers by incorporating the utilization of cross-layer gradients. More specifically, when server model updates the deep layers, we combine and refine the gradients from these layers with the gradients from the shallower layers to obtain appropriately updated gradients. Figure T provides visual representation of how cross-layer gradients are employed. We assume that this stage has N layers. The first layer with the same shape in a stage (block) is referred to as Layer 0, and its corresponding gradients time step the same stage, the cross-layer gradients are given by G + G Despite a large number of works on short-cut paths in neural networks, our method differs fundamentally in terms of the goals and the operations. We provide a thorough discussion in Appendix B
Gū. For Layer k. where & E 21,2. .N. within
3.3 GRADIENTS DIVERGENCE ALLEVIATION
However, the direct utilization of cross-layer gradients leads an acute issue known as gradient and weight divergence (Wang et al.] 2020 Zhao et al. 2018), as depicted in Figure To counter this effect, we introduce gradient normalization (Figure 6b) and the proposed convex optimization problem to restrict gradient directions, as illustrated in Figure 6c.
Cross-layer Gradients Normalization. Figure (b) depicts the benefits of utilizing normalized gradients. The normalized cross-layer gradient 96. + 9k directs the model closer to the global optimum than the original cross-layer gradient gó + + In particular, our normalization approach emphasizes the norm of gradients, i.e., s6' = / /l and of of ll The normalized cross-layer gradient is computed as (g6' gf x (l1sóll
11)/2 in practice.
Convex Optimization. In addition to utilizing normalized gradients, incorporating novel projective gradients that leverage knowledge from both şú and gf serves to alleviate the detrimental impact of gradient divergence arising from the utilization of cross-layer gradients. Moreover, Our objective to find the optimal projective gradients, denoted as Jopt. which strike a balance between being as close possible to Os while maintaining alignment with yo- This alignment ensures that ys is not hindered by the influence of go while allowing fopt to acquire the beneficial knowledge for ge from 9o. In other words, aim for gopt capture the udvantagcous information contained within go without impeding the progress of gt. Pursuing this line of thought, we introduce constraint aimed ensuring the optimization directions of gradients, outlined (9ó- of) 2 0, where (:,)) is the dot product. To establish a convex optimization problem incorporating this constraint, we denote the projected gradient as 9opt and formulate the following primal convex optimization problem,
min
Sépel 8.1. Wopt g6) 2 0,
(1)
Sopt
where we preserve the optimization direction of gi in g'pe while minimizing the distance between Sopt and ok. We prioritize the proximity of J of over gf since of represents the true gradients of
5

---

Published as a conference paper at ICLR 2024
layer k. By solving this problem through Lagrange dual problem (Bot al. 2009), derive the following outcomes,
Theorem 3.1. (Divergence alleviation). If gradients vectors, for the layers that require cross-layer gradients, their updated grudients can be expressed as,
gh.
3 2 f < 0,
Sopt
0+ gui
where 0¹ = a. a = (s6)) 96 and 3
of
Remark 3.2. This theorem can be extended to the matrix form.
We provide proof for Theorem 3.1 and demonstrate how matrix gradients are incorporated into the problem Appendix [ Our analytic solution Equation qqquationaautmmattiin automatically determines the optimal settings for parameter 8" eliminating the need for cumbersome manual adjustments. In our practical implementation, we consistently update the server model using the expression of 8' gó. irrespective of whether 3 0 or B O. This procedure is illustrated in Algorithm iin Appendix H
Communication Overheads. According to the entire process, the primary process (internal cross- layer gradients) is conducted on the server. Therefore, our method does not impose any additional communication overhead between clients and the server.
4 CONVERGENCE ANALYSIS
In this section, we demonstrate the convergence of cross-layer gradients and propose the convergence rate in non-convex scenarios. To simplify the notations, we adopt L; to be the local objective. At first, we show the following assumptions frequently used in the convergence analysis for FL Tan al. 2022 Li et al. Karimiirddy et al.) 2020).
Assumption 4.1. (Lipschitz Smooth). Each objective function I2 is L-Lipschitz smooth and satisfies that ( (y))ll s L||x poc-ypi yll. (xx y) D E I,.,, K.
Assumption 4.2. (Unbiased Gradient and Bounded Variance), At each client, the stochastic gradient is an unbiased estimation of the local gradient, with TE [ggg (x))) = V (x)),, and its variance is bounded by 0². meaning that (x) v [=) S 0², Vi E 1, ..... K, where o² 2 0.
Assumption 4.3. (Bounded Expectation of Stochastic Gradients). The expectation of the norm of the stochastic gradient at each client is bounded by p. meaning that [Ellg, (=)]ll S P. E I,... K. Assumption 4,4. (Bounded Covariance of Stochastic Gradients). The covariance of the stochastic gradients is bounded by T, meaning that Cou(get 96,1) S T, E 1.., K, where 1,,1y are the layers belonging to model at client i.
Following these assumptions, we present proof of non-convex convergence concerning the utilization of cross-layer gradients in Federated Learning (FL.) We outline our principal theorems as follows.
Theorem 4.5. (Per round drift). Supposed Assumption 4.1 to Assumption 4. are satisfied, the loss function of an arbitrary client at round t + 1 is bounded by.
Lnj²
E-
LEd"
E[ B.L - -n-
2n(T + Lnp² +
(3)
e-U
The Theorem 4.5 demonstrates the bound of the local objective function after every communication round. Non-convex convergence can be guaranteed by the appropriate Theorem 4.6. (Non-convex convergence). The loss function is monotonously decreased with the increasing communication round when,
E /L 1(T
(4)
| 2 Ep² + 2(2p² + o² +))
Moreover, after prove the non-convex convergence for the cross-layer gradients, the non-convex convergence rate is described as follows.
Theorem 4.7. (Non-convex convergence rate). Supposed Assumption 4.1 Assumption 4.4 are satisfied and k = Lo for an arbitrary client, given any > 0, after
T En((2 Lnjee 3Lno² -((2 + Lr) 1(1 Lm)pp)
(5)

---

Published as a conference paper at ICLR 2024
Table 2:
Test accuracy of homoe--omogeneous methods with 100 clients and sample ratio 0.1. We shade in gray the methods that are combined with our proposed method, InCo Aggregation. We bold the best results and denote the improvements compared to the original methods in red. See Appendix H.5 for the error bars of InCo methods.
SVHN
CIFARIO
CINICIO
Methods
<k 0.5
a 1.0
0.5
1.U
0.5
a 1.0
0.5
a 1.0
HeteroAng
8788+111
1600100
25.
2.9
64
36.7=3.3
4.66
56.1-166 1.6
ReaNer (Snge spEiing)
HeicroProx
86.34-15
$3.9:118
87.8=2.1
199
72.54221
13.1=119
26.44-2.0
009---13
BittenScaffold
85.01:0.
5641077
80.6:2.3
2633277
65.5230
69.7228
$08129
57.8134
LieteruNova
81.91:13
16.711
14
1.001:7
60.11:37
68.0:13.5
46.11:2.
52.1122
HetesMMOON
K7.02044
58.12033
83.022.3
16
65.1229
68.422.
S0 20.1
$4.71: 188
InCoAug
90.2(12.4)
88.4(12.4)
87.0
19.0121
67.8113.0)
70.7074.0)
43.0014.4)
57.5(11.0)
InCoProx
S8.8(12.0)
86.4(12.5)
19.2
90.9
74.5(12.0)
76.77
19.1(127)
62.5(11.))
InCoScaffooa
38.3113.1)
90.1113.7)
(5.4(14.8)
67
73.8(14.1)
53.50127)
617(1399
InCoNova
1.76(1
87.4(10.7)
86.A(12.0)
S3 ((104)
62.8(12.7)
69.7(12.7)
1.
$4.1(12.0
IsCoMOON
10.1((1122)
89.5(11.2)
68-2(13.))
$7.6(12.9)
HeteroAvg
92:24:06
92000066
919=1.
93
1.
04.1=09
14
13
HeeteroProx
90.9::0.88
91.7:1066
91
92
9..0:1155
1.36:133
84.01.
$4.8.10
VIT Layer spiiting)
HeteenSerffeld
91.90:66
92.1104
92.5=099
93
818
94.3204
13
85.111
HetzroNova
92.1::99
92.4404
12
94.1
93.6
94.2206
1.7
1.7
HereroMOON
92.0+:4
92.140.3
92.7=111
9400209
93.5008
946=01
14
85.6 1.4
InCoAvg
93.0(10.))
13.111
1..3("13))
1.
94.6171.00
95.0010.9)
35.9(11.3)
86.8(11.5)
InCofrox
1..6(11.7)
92.5(10.8)
93.9(12.7)
94.0(12.))
91.0(12.0)
94.8(12.2)
(111
86.0(11.2)
InCiScnntiodd
1.09
93.0(10.9)
94
1.3(
94.6(10.8)
95.011077
85 (1119
86.5(1
InCoNowa
93.1((1.0)
93.6(11.2)
94.7(72.)
95.6(111.5)
94.8(71.2)
95.7(1.22)
86.2(109)
88.2(11.2)
IsCsMOON
92. (28())88)
93.0(10.7)
94.7012.0)
95. ((11))))
94.2(10.7)
(70.5)
86.0(11.))
86.8(11.2)
communication rounds, we have
T-1E-1
2c 4(T p²) EII n T( + Fpp + 2(2p² + 0² + r))
T TE T=U c=u
(6)
Following these theorems, the convergence of internal cross-layer gradients is guaranteed. The proof is presented in Appendix
5 EXPERIMENTS
In this section, we conduct comprehensive experiments aimed at demonstrating three fundamental aspects: (1) the efficacy of InCo Aggregation and its extensions for various FL methods (Section (.2). (2) the robustness analysis and ablation study of InCo Aggregation (Section (3) in-depth analyses
of the underlying principles behind InCo Aggregation (Section(5.4). Our codes are released on GitHub
S More experimental details and results can be found in Appendix H
5.1 EXPERIMENT SETUP
Dataset and Data Distribution. We conduct experiments on Fashion-MNISS Xiao et al., (017). SVHN (Netzer et al. 2011). CIFAR-10 (Krizhevsky et al. (009) and CINIC-10 (Darlow et al. 2018 under non-iid settings. We evaluate the algorithms under two Dirichlet distributions with a 0.5 and a = 1.0 for all datasets.
Baselines. To demonstrate the effectiveness of InCo Aggregation, we use five baselines in model- homogeneous FL: FedAvg (McMahan al. 2017), FedProx (Li et al. 2020b). FedNova (Wang et al.) 2020). Scaffold (Karimireddy et al. 2020), and MOON (Lietal, 202 a for ResNets and ViTs. the context of model heterogeneityy we extend the training procedures of these baselines by incorporating model splitting methods, denoting the modified versions with the prefix "Hetero". Furthermore, by incorporating these methods with InCo Aggregation, prefix the names with "InCo". Moreover, also extend our methods to four state-of-the-a methods in model- hondel-hetroogeeouus FI.: HeteroFI. al. 2021), Inclusive (Liu al.) 22022), Folex|l et al. 2022)) and F L(Ihhan et al.
2023 for ResNets. We take the average accuracy of three different random seeds.
Federated Settings. In heterogeneous FL, we consider two architectures, ResNets and ViTs. The largest models are ResNet26 and ViT-S/12 (ViT-S with 12 layers). We deploy stage splitting for ResNets and obtain five sub-models. which can be recognized as ResNet10, ResNet14, ResNet18. ResNet22, and ResNet26. For the pre-trained ViT models, we employ layer splitting and result in five sub-models, which are ViT-S/8, ViT-S/9, ViT-S/10, ViT-S/11. 1, and ViT-S/12. Moreover, we consider five different model capacities 3 = /2, //4, 4 1/8 1/16} in hetero splitting, where for instance, 1/2

---

Published as a conference paper at ICLR 2024
Table 3: Test accuracy of hnter--hhterogeeeitty methods with 100 clients and sample ratio 0.1. We shade in gray the methods that are combined with our proposed method, InCo Aggregation. We denote the improvements compared to the original methods in red. See Appendix H.5 for the error bars of InCo methods.
Base
Splintng
Methods
MiNIion-MNIST
SVHY
CIFARIO
Contt overheads
FLOPs
0.5
a 1.0
a 0.5
1.0
0.5
a 1.U
HeteroFL.
88.9+10
89.7407
90.+16
92.211.3
65.2=33.2
6844:36
4.6M
33.4M
Hetero
-InCo
90,0(11))
904(10.7)
92.1(11)
93.5(11.3)
68.0(13.0)
71.2(12.5)
4.6M
33.8M
InclusiveFL.
89.1:111
89.81:0
65.7::3.5
75.2M
Stage
-InCo
90.1(1.0)
90.5(10.7)
90.6(12.)
90.9(10.9)
69.1(13.)
72.3(13.9)
12.3M
75.6M
ResNer
FedRolex
88.221.0
90.22088
909113
9..6
64.72411
72.32300
4.6
33.4M
Hetero
90.4(12.2)
91.3(111))
92.8(199)
93.4(118))
67.973.2)
75.6(1).3)
ScakFL
90,9:005
91
9264:10
92.9:099
21.1-22.9
74.74:3.1
9.5M
51.9
InCCo
91.811000)
91.70107)
92,4(10.))
93.6/10.7)
73.872277
76.1(12.4)
9.5M
52.3M
N/A
AllSmall
$3.5
72.1133
$1.0:119
39.2::2.0
44.912233
0.07M
3.7M
arge
9188203
92.5208
934400
938200
7..6:2.9
82:51:10
112.4M
72.5
70.0
Hetemee
Habershig
56.3
52.5
Hettrrdyy
67.5
244
Bokn
Hobe
(a) Different batch sizes in (b) Different batch sizes in (c) Different noise pertur- (d) Different noise pertur- bations in CIFAR- 10. bations in CINIC-10.
CIFAR-10.
CINIC-10.
Figure 7: Robustness analysis for InCo Aggregation.
(ccohe wi
wia
RAO
Vatwrlle
weo
NAO
Incolin with
NAG
IncoRee wis
NAO
(a) Fashion- -NIST.
(b) SVHN.
(c) CIFAR-IO.
(d) CINIC-10.
Figure 8: Ablation studies for InCo Aggregation. The federated settings are the same as Table
indicates the widths and depths are half of the largest model ResNet26. Our experimental setup involves 100 clients, categorized into five distinct groups, with a sample ratio of 0.1. The detailed model sizes shown in Appendix H.4
5.2 INCO AGGREGATION IMPROVES ALL BASELINES.
Table and T present the test accuracy of 100 clients with a sample ratio of 0.1. Table 2 provides compelling evidence for the efficacy of InCo Aggregation in enhancing the performance of all model- homogeneous baselines. demontadessssstttes the improvements of deploying InCo Aggregation in the hetroe-heteoogeeeouus methods. Moreover, Table 3 higghlights that InCo Aggregation introduces no additional communication overhead and only incurs 0.4M FLOPs, which are conducted on the server side, indicating that InCo Aggregation does not impose any burden on client communication and computation resources.
5.3 ROBUSTNESS ANALYSIS AND ABLATION STUDY.
We delve into the robustness analysis of InCo Aggregation, examining two aspects: the impact of varying batch sizes and noise perturbations on gradients during transmission. Additionally, we perform an ablation study for InCo Aggregation. We provide more experiments in Appendix [
Effect of Batch Size and Noise Perturbation. Notably, when compared to FedAvg depicted in and Figure 7b) our method exhibits significant improvements while maintaining comparable performance across all settings. Furthermore, illustrated in Figure 7c and Figure (7d) we explore the impact of noise perturbations by simulating noise with standard deviations following the gradients.
Figure aal
Ablation Study. Our ablation study includes the following methods: (i) InCoAvg w/o Normalization (HeteroAvg with cross-layer gradients and optimization), (ii) InCoAvg w/o Optimization (HeteroAvg with normalized cross-layer gradients), (ii) InCoAvg w/o Normalization and Optimization (Het- eroAvg with cross-layer gradients), and (iv) HeteroAvg (FedAvg with stage splitting). The ablation study of InCo Aggregation is depicted in Figure & demonstrating the efficiency of InCo Aggregation.
8

---

Published as a conference paper at ICLR 2024
433
00 300 403
(a) in all layers. (b) / in Layer 11, (c) in Layer 13. (d) FedAvg.
(e) HcteroAvg.
(f) InCoAvg.
Figure 9: Important coefficients of Theorem 3.1 and t-SNE visualization of features. (a): 0 in all layers. (b): 5 in Layer 11. (c): 3 in Layer 13. (d) to (f): t-SNE visualization of features learned by different methods on CIFAR-10. 10. We select data from one class and three clients (client O: ResNet10, client 1: ResNet14, client 2: ResNet26) to simplify the notations in t-SNE figures.
1.0
Stage
CID
InCoArg
HeteroAug
0.9
0.8
6.7
0
$4.4
54.8
8 6.6
20
70.7
67.8
6.
40
75.3
72.6
Haterohep
Stagee
60
73.3
69.9
80
72.0
68.5
(a) CIFARIO with a = 0.5. (b) InCoAvg.
(c) HeteroAvg.
(d) FedAvg.
(e) Client Accuracy.
Figure 10: CKA layer similarity. Heatmaps, and accuracy of different clients. (a): The layer similarity
of different methods. (b) to (d): Heatmaps for different methods in stage and stage 3. (e): Accuracy
of each client group. (0: ResNet10, 20: ResNet14, 40: ResNet18, 60: ResNer22, 80: ResNet26)
5.4 THE REASONS FOR THE IMPROVEMENTS
We undertake comprehensive analysis to gain deeper insights into the mechanisms underlying the efficacy of InCo Aggregation. Our analysis focuses on the following three key aspects: (1) The investigation of important coefficients 0 and 33 in Theorem 3.1 (2) An examination of the feature spaces generated by different methods. (3) The evaluation of CKA similarity across various layers. Moreover, discuss the differences between adding noises and InCo gradients in Appendix H.6
Analysis for and B. In our experiments, we set 0 1 for InCoAvg w/o Optimization, the blue dash line in Figure 9a However, under Theorem [3.]] we observe that the value of 0 varies for different layers, indicating the effectiveness of the theorem in automatically determining the appropriate values. > denotes the same direction between shallow layer gradients and the current layer gradients. Furthermore, Table provides empirical evidence supporting the efficacy of Theorem 3.1 in heterogeneous FL.
Table 4: The Percentage of B > 0
Methods
Pereentage 0
Layer 11
Layer 13
InClaAg
83.8
InCoAvg wlo
t-SNE Visualizations. Figure and Figure 9e provide visual evidence of bias stemming from model heterogeneity in the FedAvg and HeteroAvg. In contrast, Figure 9/ demonstrates that InCoAvg effectively addresses bias. These findings highlight the superior generalization capability of InCoAvg compared to HeteroAvg and FedAvg, indicating that InCoAvg mitigates bias issues in client models.
Analysis for CKA Layer Similarity. Figure reveals that InCoAvg exhibits significantly higher CKA layer similarity compared to FedAvg. Consistent with the t-SNE visualization, FedAvg's s heatmaps exhibit block-wise patterns in Figure model architectures. Notably, the smallest models in InCoAvg (top left corner) exhibit lower similarity (more black) with other clients compared to HeteroAvg in stage 3. This discrepancy arises because the accuracy of the smallest models in InCoAvg is similar to that of HeteroAvg, but the performance of larger models in InCoAvg surpasses that of HeteroAvg, as indicated in Figure T0e Consequently, a larger similarity gap emerges between the smallest models and the other models. Addressing the performance of the smallest models in InCo Aggregation represents our future research direction.
to its inability extract features from diverse
CONCLUSIONS
We propose a novel FL training scheme called InCo Aggregation, which aims to enhance the capabilities of model-homogeneouu FL methods in heterogeneous FL settings. Our approach leverages normalized cross-layer gradients to promote similarity among deep layers across different clients. Additionally, we introduce convex optimization formulation to address the challenge of gradient divergence. Through extensive experimental evaluations, we demonstrate the effectiveness of InCo Aggregation in improving heterogeneous FL performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the RGC General Research Funds No. 17203320 and No. 17209822 and seed project grant from HKU-TCL Joint Research Center for Artificial Intelligence from Hong Kong.
REFERENCES
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. FedRolex: Modee-betcrogeneouus federated learning with rolling sub-model extraction. Advances in Neural Information Processing Systems, 35:29677-29690, 2022.
Sergio A. Alvarez. Gaussian rbf centered kernel alignment (cka) in the large-bandwidth limit, IEEE Transactions on Putten Analysis and Machine Intelligence, 45(5)66587-6593, 2023. doi: 10.1 T..100/PPMM.222.226665
Hankyul Back, Won Joon Yun, Yunseok Kwak, Soyi Jung, Mingyue Ji, Mehdi Bennis, Jihong Park, and Joongheon Kim. Joint superposition coding and training for federated learning over multi- width neural networks. In IEEE INFOCOM 2022-IEEE Conference on Computer Communicutions, pp. 1729-1738. IEFE, 2022.
Radu loan Bot, Sorin-Mihai Grad, and Gert Wanka. Duality vector optimization. Springer Science & Business Media, 2009.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. university press, 2004.
Cambridge
Sebastian Caldas, Jakub Konečny, H Brendan McMahan, and Ameet Talwalkar, Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.072 2018.
Yun Hin Chan and Edith Ngai. Fedhe: Heterogeneous models and communication-effcienn federated learning. IEEE International Confer- ence on Mobility, Sensing and Networking (MSN 202 I), 2021.
Yun-Hin Chan and Edith C-H Ngai. Exploiting features and logits in heterogeneous federated learning. arXiv preprint arXiv:22 10. 15527, 2022.
Chen Chen, Hong Xu, Wei Wang, Baochun Li, Bo Li, Li Chen, and Gong Zhang. Communication- efficient federated learning with adaptive parameter freezing. In 2027 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pp. 1-11. 1. IEEE, 2021.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3514-3522, 2019.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):995-828, 2012.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey. Cinic-10 10 is not imagenet or cif arXiv preprint 1rXiv:1810.03500 2018.
Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL.: Computation and communication efficient feder-
ated learning for heterogeneous clients. In International Conference on Learning Representations, 2021.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Xiuwen Fang and Mang Ye. Robust federated learning with noisy and heterogeneous clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10072-10081, 2022.
10

---

Published as a conference paper at ICLR 2024
Dashan Gao, Xin Yao, and Qiang Yang. A survey on heterogeneous federated learning, 2022. URL wttp://.wwr..bb////////////
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):188--188) March 2021. ISSN 1573-1405. doi: 10. 00.1007/11223-001-014444 01263-021-01453-2
URL. /ostp://dx.o. doi. wttp://ddoooooooo.o//ooo/oo////e or.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns the edge. Advances in Neural Information Processing Systems, 33: 14068-14080, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. In Computer Vision-ECcV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 630-645, Springer, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. NIPS Deep Learning and Representation Learning Workshop, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 12276-118888 2021.
Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous federated learning. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10143-10153, 2022.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses federated learning. Advances Neural Information Processing Systems, 34:7232-7241, 2021.
Fatih Ilhan, Gong Su, and Ling Liu. Scaleft: Resource-adaptive federated learning with heterogeneous clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24532-24541, 2023.
Wonyong Jeong and Sung Ju Hwang. Factorized-fl: Personalized federated learning with parameter factorization similarity matching. Advances in Neural Information Processing Systems, 2022.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.

---

Published as a conference paper at ICLR 2024
Daliang Li and Junpu Wang. FedMD: Heterogenous federated learning via model distillation. NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality, 2019.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 1713--0722, 2021a. la
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 965-978. IEBE, 2022.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 3703):50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of the 3rd MLSys Conference. 2020b.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv: 1511:::/111.0543 2015.
Yiying Li, Wei Zhou, Huaimin Wang. Haibo Mi, and Timothy M Hospedales. FedH2L: Federated learning with model and statistical heterogeneity. arXiv preprint 1rXiv:2101.112% 2021b.
Tao Lin, Lingjng Kong, Sehastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-255 2020.
Ruixuan Liu, Fangzhao Wu, Chuhan Wu, Yanlin Wang, Lingjuan Lyu, Hong Chen, and Xing Xie. No one left behind: Inclusive federated learning over heterogeneous devices. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3398 3406, 2022.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communicationn-effccient learning of deep networks from decentralized data. In Arlificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional Conference on Machine Learning, pp. 4615-4625, PMLR, 2019.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. shoo knowledge distillation deep networks. In International Conference on Machine Leurning, pp. 4743-4751. PMLR, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011. 1.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 14:12116-12122 2021.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint 1rXivv1609.044 04747, 2016.
12

---

Published as a conference paper at ICLR 2024
Felix Sattler, Tim Korjakow, Roman Rischke, and Wojeiech Samek. Fedaux: Leveraging unlabeled auxiliary data in federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Tao Shen, Jie Zhang, Xinkang Jía, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXiv:2006. 16765, 2020.
Yue Tan, Guodong Long, Lu Liu, Tinyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8432-8440, 2022.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. Advances neural information processing systems, 31, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for bench- marking machine learning algorithms, 2017.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. 12th Annual Workshop on Optimization for Machine Learning, 2020.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10523, PMLR, 2020.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1803-1811, 2019.
Running Zhao, Jiangtao Yu, Hang Zhao, and Edith C.H. Ngai. Radio2text: Streaming speech recognition using mmwave radio signals. 7(3), sep 2023. doi: 10.1145/3610873 URL https 1ww.://///11111111111111
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:/806.005:: 2018.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou Data-free knowledge distillation for heterogencous federated learning. In International Conference on Machine Learning, pp. 12878-12889 PMLR, 2021.
13

---

Published as a conference paper at ICLR 2024
A CENTERED KERNEL ALIGNMENT
Centered Kernel Alignment (CKA) originally serves as a similarity measure for different kernel functions (Cortes al. 2012). Later, its purpose has been extended to discovering meaningful similarities between internal representations of neural networks (Kornblith et al. (019). Compared with alternative methods to monitor representation learning, such as Canonical Correlation Analysis- based methods (Raghu et al. 2017 Morcos et al. 2018) and neuron alignment methods (Li et al.) 2013; Wang et al., 2018). CKA achieves the state-of of taee--fftthheeeeee the-art performance in measuring the difference between representations of neural network. This is based the fact that CKA reliably identifies correspondences between representations from architecturally corresponding layers in two networks trained with different initializations
Denote X E R and Y E < as two representations of n data points with possibly different dimensions (i.e., p o Q). These two representations fall into the following three categories: (1) internal outputs at two different layers of an individual network, (2) internal layer outputs of two architecturally identical networks trained from different initialization by different datasets, or (3) internal layer outputs of two networks with different architectures possibly trained by different datasets. The application of CKA in our paper belongs to the third category, where we examine the CKA similarities of a corresponding layer output between every pair of local client models in the context of federated learning.
Let li (:,-) -) and kg (+.-) be the kernel functions for X and Y respectively. Then the resulted kernel matrices of k. and ky with respect to xI,... Xn and y :I..... yn are K₂ and Ky, whose -entries are K2 (i, = kis (xi, xj) and (t,1,) = (y), Then CKA is defined as
tr( Hr(,HH,,F) KH HK,
CKA(KZ, Ky)=
Vi ln(K, H
HHH,H
where - 11T T is the centering matrix.
As for the kernels in CKA, we select linear kernel (i.e., K2 -XT, K, YYT over Radial Basis Function (RBF) kerne from common kernels for the following reasons. First, experiments in Kornblith et al. (2019) manifest that linear and RBF kernels work equally well in similarity measurement of feature representations. Furthermore, it is recently validated that CKA based on an RBF kernel converges to linear CKA in the bangebbandwidth limit (Alvarez 2023). Hence, in
investigation we stick with linear CKA for computational efficiency, where the resulting linear CKA
CKA (X, Y) = CKA(XXT. .YT)
u(XXTTYYTTH)
HXXTXX MIX ||YT
(8)
|XXXX||||X*XXX]]
In our design, we measure the averaged CKA similarities according to the outputs from the same batch of test data. The range of CKA is between and 1, and higher CKA score means more similar paired features.
B COMPARISONS WITH RESIDUAL CONNECTIONS
Remarks self-mixture approaches in neural networks. The goal of residual connections is to avoid exploding and vanishing gradients to facilitate the training of single model (He et al., 2016). while cross-layer gradients aim increase the layer similarities across group of models that are jointly optimized in federated learning. Specifically, residual connections modify forward passes by
*We refer the readers to Section 6.1. in Kornblith et al. 2019 for complete sanity check of representational similarity measures.
$1 Linear kernel k(x,,x) x;) = x]xx
RRdiaa Basis Function (RBF) kernel k(x,,,) x;) exp(->|lx; x,113) with 7
14

---

Published as a conference paper at ICLR 2024
adding the shallow- layer outputs to those of the deep layers. In contrast, cross-layer gradients operate on the gradients calculated by back pacckpropagation. We present the distinct gradient outcomes of the two methods in the following.
Xi-2
x--2
Wi-1
Wi-1
g-1-1 1-
Xt-1
X
W;
W
X
x
W(+1
X
W(+1
M++1
Xi+1
x/+1
(a) Cross-layer gradients
Residual connections
Figure 11: Comparison of cross-layer gradients and residual connections Consider three consecutive layers of a feedforward neural network indexed by - 1, i 1. With slight abuse of symbols, input X; to layer i 1, the output from the previous layer becomes the input to the next layer, thus generating X X1, x sequentially.
use f(-; W&) denote the calculation in the k-th layer. Given the
Xi = f(xi- W:-1)
Xi f(x-11; Ws)
XI+1 f(x;; W++1)
In the case of residual connections, there is an additional operation that directs x; Xi+1, formulated as x x Xi+1 + x5. The gradient of W; is
Øloss
gw:
OW;
Öloss
0x(+1
0xi
0x(+1
0x;
oW;
(10)
Aloss
[0x++1
Ix:
+11
0(x1+1 + xi)
0xi
Ow;
Yloss
0xi+1
Ox;
+
0(x++1 + xx)
aw;
aw,
In the case of cross-layer gradients, the gradient of W; is Ōloss
Mloss
gw. ow,
aW,-1
(11)
Öloss
[0x++1
0xi+1
0x,+1
ow,
aW,11
We note that both residual connections and cross-layer gradients are subject to certain constraints. Residual connections require identical shapes for the layer outputs, while cross-layer gradients operate on the layer weights with the same shape.
C PROOF OF THEOREM 3.1
This section demonstrates the details of the proof of Theorem 3.1 We will present the proof of Theorem 3.1 in the vector form and the matrix form.
15

---

Published as a conference paper at ICLR 2024
C.I VECTOR FORM
We state the convex optimization problem Theorem Iin the vector form in the following.
min
119a
Dopt l2.
Sope
(12)
s.t. (gopt. go) 0.
Because the superscript would not influence the proof of the theorem, we simplify the notation g to We use Equation 12 instead of Equation T to complete this proof. The Lagrangian of Equation T2 is shown as,
Lgopt, 2) =(g - 9opt) (gk= (opt) Aopt
T gr Uo T
Hopt + PepsSopt A
(13)
gh 9k 2 + Popt A
Let (I((oop(s))
O, we have Sgopt
9opt = + 2go/2,
(14)
which is the optimum point for the primal problem Equation 12 To get the Lagrange dual function L(A) =
inf Sopt L[Oopt X), we substitute Jopt by Ok + >go/2 2 in L(gopt, A). We have
L(A) = gh 2( Aga yT 9u (9x +
Ago ((
)
+
Ago
A(gn +
Ago )T go
2
2
Ago 9k
Ag g0
X go go
22 go
gk gk
29h 9k
Ago 9k + gh 9k +
+
Agh 90
2
4
X
g0 90
X² 9o 90
-9% 9k
29 T + o% 9s - bbk + Ago
T
+
Age 90
4
2
9 90 X² 9h 90d.
(15)
Thus, the Lagrange dual problem is described as follows,
max L(A) = 9 90
X2 of gud,
4
(16)
s.t. A 2 0.
L(A) is a quadratic function. Because 96 90 2 0, the maximum of L(A) is at the point À where a 90 gu and = 9% 9u if we do not consider the constraint. It is clear that this convex optimization problem holds strong duality because satisfies Slater's constraint qualification(Boyd et al. 2004). which indicates that the optimum point of the dual problem Equation [6] is also the optimum point for the primal problem Equation 12 have
26
We substitute by in Equation 14 and we
gk 19k
if 2 0,
9opt
0t go,
if O,
(17)
where 6¹ h (go)Ton gn and b gb.90. We add the superseript to all gradients, and finish the proof of Theorem 3.1
9o.
C.2 MATRIX FORM
The proof of the matrix form similar Appendix C.1 as follows,
We update Equation 12 to the matrix form
min
|| - Gopt
Gopt
(18)
s.t.
(Gopt, Go) 2 0.
Similar to Equation 13 the Lagragian of Equation
18 is,
I(Gopt, A) ir Ir G+) Ir(G Gk) tr(G Gopt) + GTT Xtr(GT (19)
16

---

Published as a conference paper at ICLR 2024
where (r() means the trace of the matrix A. We can obtain the optimum point for Equation 18
according to (L(Gopt-A) 3Gopt
We have
Gopt G5 + XGa/2.
(20)
Similar the analysis in Appendix C.1 and Equation [5] we get the Lagrange dual problem as follows,
max L()) tr(G] Go)
Gr(
4
(21)
8.L. X 2 0,
where tr(G] Go) 2 0. Following the same analysis in Appendix C.1 we have
Gk: Gk O'Go,
if 2 0, if b < 0,
Gapt
(22)
where 0¹ = tr(G] Go) and Ir(GT last, we have finished the proof of the matrix form of Theorem 3.1
D PROOF OF CONVERGENCE ANALYSIS
We show the details of convergence analysis for cross-layer gradients. W the weights from the layers which need cross-layer gradients at round of the local step e. To simplify the notations, we use Wee instead of W
Lemma D.1. (Per Round Progress.) Suppose our functions satisfy Assumption 4.1 and Assump- tion 4.2 The expectation of a loss function of any arbitrary clients at communication round after E local steps are bounded as,
Lnj
E-1
LEn
S (n
| L
o².
(23)
c=0
Proof.
Considering arbitrary client, omit the client index in this lemma. Let Wi,+1 Wae 199t,e. we have
L
Lt,e11 S Li,e + LL We,t1 We,e ||W Wo,ell
2
(24)
L
L VV It,e) 2 |ng.,ell.
where Equation
24 follows Assumption 4.1 We take expectation on both sides of Equation 24 then
Le, < [E[L,e] nB es se
1]+
Blllngeell
Ln E1llrr
ELL V L, .
2
(a) E[L,e] V. r. 112
Tun2
+ (Billgt,e111 + Var(Ils,eel))
(25)
2
Tmp
E[L,.l VL
(lIV,,,e 12 1i-Vand Var(Ilgte11)
2
(b)
Ln
Tnp
E E
- (n
V +
o².
2
2
where (a) follows Var(X) = E[X - [x] and (b) is Assumption 4.2 Telescoping local step
E
have
Lai E-
LEn² c=0
EL [T. (1)
0
(26)
then finish the proof of Lemma D.I
17

---

Published as a conference paper at ICLR 2024
Lemma D.2. (Bound Client Dirft.) Suppose our functions satisfy Assumption 4.2 Assumption 4.3 and Assumption 4.4 After each aggregation, the updates, AW, for the layers need cross-layer gradients have bounded drift:
= (20" + +1).
(27)
Proof.
We have W1+10 WLE-- AW = 7799 + 91 44, need cross-layer gradients. Because all gradients in the aggregation round, we omit the time subscript in this proof process. Since a constant, we also simplify it. Mt and Jt, the gradients from the same client, indicating that they dependent, then
|awI] 196. + tr,1 11
W&
(28)
11% 112 + (9t. 9t..)....)111 11,,,11
where (c) Schwarz inequality. We take the expectation on both sides, then
E
[AW]] v Bll [2] |B[] 9t))1 + E(Ila,
@ 1llla 1| + Var(I19.11 II) + Bill,, |1] + Var(Ila,11) + ( 9t,)
2(p² + + 2E[l| 91.
(29)
[ 2(p² + +02) 2(Cor(CCo..o..ooo 0c)) + E El1 [[]
2 2(p² + 0²² + 2[ + p²)
= 4p² + +
where (d) follows assumption Assumption
4.2 and Assumption 4.3, (e) follows the covariance formula,
and (f) follows assumption Assumption 4.4 we complete the proof Lemma D.2.
We put back 17² to the final step of Equation 29 At last,
D.1 PROOF OF THEOREM 4.5 AND THEOREM 4.6
We state Theorem 4.5 again in the following.
(Per round drift) Supposed Assumption 4.1 to Assumption 4.4 are satisfied, the loss function of an arbitrary client round is bounded by,
Tnp
E
L++1,l S EELL, - -nn 2
V VL 12+
c=0
(30)
LFE 2
σ² + 2n + + (2p² + +1).
Proof.
Following the Assumption 4.1
we have
L Li+1,0 & L1,-1 (VL,E-1, - Wi,E-1) + = 11W - Wr,--112
(31)
-I,-- + (VL, + + + "" 11
Taking the expectation both sides, obtain
[T++1o R[ + (V Mo. + 91)))= 5 (WW
(32)
The first item is Lemma D.I and the third item is Lemma D.2 (L 1,91, + m.)) in the following, then,
We consider the second item
( Sto 99, ELVL, E I E
(c) (CoV(VL,EE_,,
B || I|}+
(33)
Cou(VVn,,E-,99)) + F 1,,
2(T + p2). 5
18

---

Published as a conference paper at ICLR 2024
Combining two lemmas and Equation 33
we have
Lap
E-1
LEn²
[[4++1,0] S [[ (1)
al +
0² 27 + x²) - (2p + +),
(34)
then we finish the proof of Theorem 4.5
For Theorem 4.6
consider the sum of the second term to the last term in Equation 34 be smaller
than 0, i.e.
Lnp
E-1
LE
(1-
1
o² +
2n + + Ln +0² +1) < 0,
(35)
2
2
c=
then, have
2
> 0-=0
E-1
/L - p²)
(36)
c VL 12 + Epp + 2(2p² + 0² +1)
E
We finish the proof of Theorem 4.6
D.2 PROOF OF THEOREM 4.7
Telescoping the communication rounds from 0 to t T - 1 with the local step from = to
E on the expectation on both sides of Equation 34 we have
T-1E-1
TE Di (E[L,,o) [[L++1,0)) + Lnjo 4n + P²) 2L o² +1)
2
V
TE
21 Lap
1=0
=
(37)
Given any > 0, let
ΣT- TE
( Lc,o L[(+))))) + Ln + 4n(T + +3² +2) + +T) +
1=
Se.
(38)
2n Lnp
and we denote = To
L,, then Equation
38 becomes
o= 2= 4n (T + p²) 22p2 + +T)
TE
S c,
(39)
2n - Lip²
because
ΣT- [B( E[1411,0)) S K. We consider T Equation 39 i.e.
2x
T
(40)
En - Lnje 3Lmp²
2(2 + Ln)l - 4(1 + Ln)p²)
then, we have
T-1E-1
Σ Σ
|L,
Sc. Sc.
(41)
TE
when
2 4(T + 4(T p²)
(42)
L(e Ep + 2(2p² + 0 +T)
We complete the proof of Theorem 4.7
MORE RELATED WORKS
E. FEDERATED LEARNING.
In 2017, Google proposed a novel machine lcarning technique. i.c., Federated Learning (FL). to organize collaborative computing among edge devices or servers (McMahan et 2017). It enables multiple clients to collaboratively train models while keeping training data locally, facilitating privacy protection. Various synchronous or asynchronous FL schemes have been proposed and achieved good performance in different scenarios. For example, FedAvg (McMahan et al. 2017) takes a weighted average the models trained by local clients and updates the local models iteratively. FedProx (Li al.) 2020b) generalized and parameetrizedd FedAvg, guaranteeing the convergence when learning over non-in iid data. FedAsyn (Xie et al., 2020) employed coordinators and schedulers to achieve an asynchronous training process.
19

---

Published as a conference paper at ICLR 2024
E.2 HETEROGENEOUS MODELS.
The clients in homogeneous federated learning frameworks have identical neural network architec- tures, while the edge devices or servers in real-world settings show great diversity. They usually have different memory and computation capabilities, making it difficult deploy the same machine- learning model in all the clients. Therefore, researchers have proposed various methods supporting heterogeneous models in the FL environment.
Knowledge Distillation. Knowledge distillation (KD) (Hinton et al. al., aiming to train student model with the knowledge distilled from
2015) was proposed by Hinton teacher model, which
2021, Zhao et al.) (023). Inspired by the knowledge distillation, several studies(Li & Wang] 2019 Li et al. al.aaal22200 2021b] He et al. 2020) are proposed to address the system heterogeneity problem. In FedMD(Li & Wang] (211)). the clients distill and transmit logits from a large public dataset, which helps them learn from both logits and private local datasets. In RHFL (Fang& & Ye, 2022), the knowledge is distilled from the unlabeled dataset and the weights of clients are computed by the symmetric cross- cross-entrops loss function. Unlike the aforementioned methods, data-free KD is new approach to completing the knowledge distillation process without the training data. The basic idea is to optimize noise inputs to minimize the distance to prior GoooledeeNNayal et al.) 2019). Chen et al. alenn et al.) 2019) train Generative Adversarial Networks (GANs) Goodffeloww et al. 2014) to generate training data for the entire KD process utilizing the knowledge distilled from the teacher model. In FedHe & Ngail 2021), a server directly averages the logits transmitted from clients. FedGen(Zhu et al. (021)) adopts a generator to simulate the prior knowledge from all the clients, which is used along with the private data from clients in local training. In FedGK et al. 2020), neural network is separated into two segments, one held by clients, the other preserved in a server, in which the features and logits from clients are sent to the server to train the large model. In Felo (Chan & Ngail 2022), the representations from the intermediate layers are the knowledge instead of directly using the logits.
becomes an important research area in Machine Learning (Gou et al.
Public or Generated Data. In FedML. (Shen et al., 2020), latent information from homogeneous models is applied to train heterogeneous models. FedAUX (Sattler et al. (02T) initialized heteroge- neous models by unsupervised pre-training and unlabeled auxiliary data. FCCL (Huang al., 2022)
calculate a cross-correlation matrix according to the global unlabeled dataset to exchange knowledge. However, these methods require a public dataset. The server might not be able to collect sufficient data due to data availability and privacy concerns.
(Diao et al. 202 derives local models with different sizes from one large model, the architectures of local and global models still have to share the same model architecture, and it is inflexible that all models have to be retrained when the best participant joins or leaves the FL training process. Federated Dropout (Caldas et al. (018)) randomly selects sub- models from the global models following the dropout way. SlimFL (Back et al. 2022) incorporated
Model Compression. Although HeteroFL
width-adjjstable slimmable neural network (SNN) architectures/ Yu & Huang 2019) into FL which can tune the widths of local neural networks. FjORD (Horvath al. [2021] tailored model widths to clients' capabilities by leveraging Ordered Dropout and a self-distillation methodology. FedRoLex (Alam et al., (022) proposes rolling sub-model extraction scheme to adapt to the heterogeneous model environment. However, similar to HeteroFL., they only vary the number of parameters for each layer.
F PROBLEM FORMULATION
In this section, we introduce federated learning with model heterogeneity. Federated learning aims to foster collaboration with clients to jointly train shared global model while preserving the privacy of their local data. However, in the context of model heterogeneity, it becomes challenging to maintain the same architectures across all clients. Specifically, we consider a set of physical resources denoted as [2 where R; represents the available resources of client i. For each local client model (M, (M)}² )- the resource requirement R(M,) must be smaller than or equal to the available resources of client i, i.e. (R(M.) S R To satisfy this constraint, the client models have varying sizes and architectures. Let w, denote the weights of the client model Md. and f(, w;) represent the forward function of model M, with input v. Moreover, each client has a local dataset D; = (( bks)lk E (1, [. where [D,] signifies the size of a dataset D,.. The loss
20

---

Published as a conference paper at ICLR 2024
function 4 of client i is shown follows,
D.1 (f uk),
min 4(w,) Dil
(43)
where lesE is a cross-entropy function. Moreover, if we denote K = -i== Di| as the total size of all local datasets, the global optimization problem is,
Dal Dell L(W1,-.. Wn) Σ k(w:))
n
min W1 M2 Win
(44)
where the optimized model weights (wi, W2, ..., w/ are the parameters from M In our case, {W1, u2,,..,, Wn) } are split from the server model weight w, from the server model M, The goal of our paper is to propose a method that can effectively optimize Equation 44
G CONFIGURATIONS AND MORE RESULTS OF THE CASE STUDY
G.1 CONFIGURATIONS
In the case study, we have five ResNet models which are stage splitting from ResNet26, resulting in ResNet10, ResNet14, ResNet18, ResNet22, and ResNet26. Five ViTs models are ViT-S/8, ViT-S/9, VIT-S/10, VIT-S/I1, ViT-S/12, the results from the layer splitting of ViT-S/12. The model prototypes are the same as the experiment settings. To quantify a model's degree of bias towards its local dataset, we CKA similarities among the clients based on the outputs from the same stages in ResNet (ResNets of different sizes always contain four stages) and the outputs from the same layers in Vision Transformers (ViT') (Dosovitskiy al. 2020). Specifically, we measure the averaged CKA similarities according to the outputs from the same batch of test data. The range of CKA is between O and 1, and higher CKA score means more similar paired features. We train FedAvg under three settings: IID with the homogeneous setting, Non-IID with the homogeneous setting, and Non-IID with the heterogeneous setting. FedAvg only aggregates gradients from the models sharing the same architectures under the heterogeneous model setting (Lin et al., (020). For ResNets, we conduct training 100 communication rounds, while only rounds for ViTs. The local training epochs for clients five for all settings. We use Adam(Kingna settings for all client models, and the batch size is 64. We use two small federated scales. One is ten clients deployed the same model architecture (ResNet18 for ResNets and ViT-S/12 for ViTs), which is called a homogeneous setting. The other ten clients with five different model architectures, which is heterogeneous setting. This setting means that have five groups whose architectures heterogeneous, but the clients belonging to the same group have the same architectures.
& Ba, 2015) optimizer with default parameter
Marit 7 morml
3.08
9.07
Dlocta Lá sarmi
CIAA
0.028
9.00
Aceency
9.040
20.15
5oa5s
0.04
0.02
Razoda
(a) Similarity of gradients (b) Similarity of gradients (c) The relations hetween (d) The relations between in Block 7.
in Block 11.
CKA and accuracy in CKA and accuracy in stage2.
stagel.
Figure 12: Cross-environment similarity and more results between accuracy and CKA similarity. (a) and (b): Cross-enviromment similarity from Block 7 and Block 11 from ViTs. (c) and (d): The positive relation between stagel and stage2.
G.2 RELATIONS BETWEEN CKA AND ACCURACY.
In this subsection, we continue to describe more results about the relations between CKA similarity and accuracy. Similar to Figure Ic from stageO, Figure 12c and Figure 12d show the positive relations between CKA and accuracy from stagel and stage2.
G.3 GRADIENT ANALYSIS FOR VITS
Similar to the gradient analyses conducted for ResNets, have performed the analysis of gradient distributions for ViTs. In our investigation, we have analyzed the outputs from the norm I and norm2 layers within the VIT blocks and have also applied InCo Aggregation to these layers. The selection
21

---

Published as a conference paper at ICLR 2024
(a) Block7 norml in IID (b) Block? norm2 in IID (c) Block1 norml in IID (d) Block11 norm2 in IID with homo.
with homo.
with homo.
with homo.
Figure 13: The gradient distributions from round 10 to 20 of ViTs in IID with homo.
(a) Block7 norml I in Non- (b) Block7 norm2 in Non- (c) Block norml in Non- (d) Block11 11 norm2 in Non- IID with hetero.
IID with hetero.
IID with hetero.
IID with hetero.
Figure 14: The gradient distributions from round 10 to 20 of ViTs in Non-IID with hetero.
of norml and norm2 layers is motivated by the significance of Layer Norm in the architecture of transformers (Xiong al. 2020). Additionally, have chosen Block7 and Block11 for analysis as, in the context of heterogeneous models, Block7 is the final layer of the smallest ViTs, while Block11 represents the final layer of the largest ViTs.
From Figure 2a and Figure 12b, we observe that the cross-environment similarities derived from the shallow layer norm (norml) are higher compared those from the deep layer norm (norm2). Moreover, similar to the analysis conducted for ResNets, we discover that the distributions of norml ViTs exhibit greater smoothness compared to norm2, as depicted in Tigure[3]ane Figure [4 These findings reinforce the notion that InCo Aggregation is indeed suitable for ViTs.
wddes
CATD 2000
wides
2.008
+2.00
Cradl
2208 CO08
(a) Stage2 convÓ in Non- (b) Stage2 convl in Non- IID with hetero.
(c) Stage3 convŌ in Non- (d) Stage3 convl in Non- IID with hetero with dif- IID with hetero with dif- ferent seed.
IID with hetero.
ferent seed.
Figure 15: The gradient distributions from round 40 to 50 of ResNets in Non-IID with hetero. (a) and (b) Stage2 convO and convl. (c) and (d) Stage3 convO and conv 1 with different seed.
G.4 GRADIENT DISTRIBUTIONS FROM STAGE 2 AND DIFFERENT SEED.
Figure 15a and Figure 15b demonstrate the gradient distributions in Stage 2. In contrast to the gradient distributions of Stage the differences in gradient distributions across different layers less evident for Stage 2. This can be observed from Figure Tal where the CKA similarity for Stage 2 is considerably higher than that of Stage The higher similarity indicates that Stage is relatively less biased and more generalized compared to Stage 3, resulting in less noticeable differences in gradient distributions. This observation further supports the relationship between similarity and smoothness, as higher similarity leads to smoother distributions. Moreover, Figure 13c and Figure [3d]lluustaat illustrate that the gradient distributions still keep the same properties in different random seed, indicating that the relations between similarity and smooth gradients are not affected by SGD noise.
G.5 OTHER RELATIONS BETWEEN CKA AND THE STATISTICS OF GRADIENTS.
Figure 16 provides an overview of additional relationships between layer similarity and the cross- environment gradient statistics derived from IID with homo and Non-IID with hetero. We calculate the difference between gradients from the same layer across these two environments. To clarify the tendency of similarity for each stage, normalize the results according to the smallest value within each stage. As shown in Figure [6 none of these gradient statistics exhibit stronger correlations with the similarity of gradients compared to the smoothness, discussed in Section 2.2 and Appendix G.3
22

---

Published as a conference paper at ICLR 2024
1.
1.30
31.0
22
23
21
(a) CKA with mean.
(b) CKA with variance.
(c) CKA with covariance. (d) CKA with difference.
Figure 16: Other relationships between CKA and the cross-environment statistics of gradients from IID with homo and Non-IID with hetero. We use abbreviations for "stage" and "conv," represented as "s" and "c" respectively. For example, "s2c0" represents stage 2, convO.
(a) The CKA similarity of (b) The CKA similarity (c) The CKA similarity of (d) The CKA similarity
IID with homo for stage of IID with homo for IID with homo for stage of IID with homo for stage I.
0.
stage
(e) The CKA similarity of (f) The CKA similarity of (g) The CKA similarity Non-IID with homo for Non-IID with homo stage O.
(h) The CKA similarity Non-IID with homo for stage 3.
of Non-IID with homo for stage 2.
stage 1.
(i) The CKA similarity Non-IIl with hetero for Non-III with hetero for of Non-IID with hetero stage 0.
(j) The CKA similarity of (k) CKA similarity
(l) CKA similarity of Non-IID with hetero for stage 3.
stage 1.
for stage 2.
Figure 17: The CKA similarity of IID with homo, Non-IID with homo and Non-IID with hetero for ResNets.
G.6 HEATMAPS FOR THE CASE STUDY
In this part, we will show the heatmaps for all stages of ResNets and layer layer of ViTs in Figure I7 and Figure 18 These heatmaps are the concrete images for Figure [ We can see that the CKA similarity is lower with the deeper stages or layers no matter in ResNets and ViTs. However, it is notable that the different patterns for CKA similarity between ResNets and ViTs from the comparison between Figure 17 and Figure 18 To get a clear analysis, we focus on the last stage of ResNets and layer 7 of ViTs, which are the most biased part of the entire model. Like Figure I7] in ResNets, almost clients are dissimilar, while only a part of clients has low similarity in ViTs from Figure [8] Along with the experiment results from Table the improvements in ViTs from FedInCo are modest. One possible reason is that we neglect more biased clients and regard all clients having the same level of bias in ViT', which is possible improvement for FedInCo.
23

---

Published as a conference paper at ICLR 2024
(a) CKA similarity of (b) The CKA similarity (c) CKA similarity (d) The CKA similarity
IID with homo for layer IID with homo for IID with homo for layer of IID with homo for layer 7.
4.
layer 5.
6.
(e) The CKA similarity of Non- with homo for Non-IID with homo for layer
(f) The CKA similarity of (g) The CKA similarity
(h) The CKA similarity of Non-IID with homo for layer 7.
of Non-lID with homo for layer 6.
layer 4.
(i) The CKA similarity of Non-IIl IID with hetero for layer
(j) The CKA similarity of (k) The CKA similarity
(I) The CKA similarity of Non-IID with hetero for layer
Non-IID with hetero for of Non-IID with hetero layer 5.
for layer
Figure 18: The CKA similarity of IID with homo, Non-lID with homo and Non-lID with hetero for ViTs.
H MORE DETAILS OF THE EXPERIMENTS
H.1 PROCEDURE FOR INCO AGGREGATION
The pseudo-codes for InCo Aggregation shown in Algorithm [ server model, indicating that the methods focused on the client can be aligned with InCo Aggregation, shown our experiments.
InCo Aggregation is operated in
Algorithm InCo Aggregation (InCoAvg as the example)
Require: Dataset Da. (1..., K}, K
else
clients, and their weights Wi.-.., WK-
11:
end 1/1 = gi,
Ensure: Weights for all clients W1, WK.
12:
1: Server process:
13: w//
[+1 wf, + gh 1+1
2: while not converge do
14: end for
3: Receives 9',, from the sampled client.
15: Sends the updated w to sampled
4: Parameter aggregation for gum.-
clients.
5: for each layer I in the server model do if bt needs cross-layer gradients then gi st. Normalizes gi and
end while
6:
17:
Client processes:
while random clients i, 1,..., K do
7:
19: Receives model weights w/
gi..
8:
0r, a, from Theorem 3.1 9 x
20:
Updates client models w to wf.
of (0), of,
1l9f, II)
21:
Sends y!., = we w (-1
to the server.
9:
222
end while
24

---

Published as a conference paper at ICLR 2024
Table 5: Test accuracy of 100 clients and sample ratio 0.1. We shade in gray the methods that are combined with our proposed method, InCo Aggregation. We show the enror bars for InCo Aggregation in this table.
SVHN
CIPARIO
CINICIO
Bose
Methods
1.0
0.5
1.0
1.0
0.5
1.0
FodAvg
86.71 1.0
87.72206
3.2
81.6225
523114
61.3232
43.1227
40.23.1
FedProo
73.14:18
1..6:::55
32.04:28
43.7
1924:2
23:4:124
17.4
198:114
Scaffold
07.950.5
88.020.3
3.
82443.1
54.1336
6L8430
43.524
49.43.1
FedNova
12.7:10.2
13.6:02
1344 0.4
1533203
1041003
14,30022
12.0:0.3
1400:02
ResNet
MOUN
57.74004
87.5-003
47.24277
58.8226
InCoAvg
9.22
88.42:
57 628
89.0226
678232
70.74::4
$3.02322
$7.5233
IoCoProx
$8.8-42.33
86.4433.2
19.04-13
90.8-11.2
74.5
76.8-1-888
59.1#332
6254-24
InCaScaffold
1.
90.12:12
4:
37.823
673116
738429
$3.533
61.7130
InCoNeva 1..6.11.4
87,41:33
86.11:25
884118
62.81399
69.7:1422
48.0::277
11.11117
InCoMOON
1.31
89.5
35.6
893320
68223311
7188223
$4.323.0
57.6227
FodAvg 92.0
91.9:005
92.4109
93.9:088
95.7110
94.240088
83.8114
$5.1
FedProx
89.8:1.5
89.7:20.5
71 814
81.1229
82.6133
84.7::333
67.82288
713310
Scazfold
92.040044
92.0:90.5
FedNova
70.320.5
76.7204
27 44
49.8:2.
50.72033
$4.400.5
31.62:15
507213
vir
MOON
92.11003
92.1:10.2
92.5:1122
93.91:99
93.61:088
94.6:10.3
84.3:1:66
85.3-12
LoCooAng 98.09:06
93.12005
95.0
94.62:077
95.0%b0.6
15.9
16.113
InCoProx 92.6:0033
92,5:10,3
91 9:7
94.41006
94.01110
94.8:10077
85.1
$6.0::88
InCoScatfold
92.9403
93.0+-0.2
94.64-0.5
95.0:+022
InCaNowa
93.12003
93.620.3
94.72:09
V5.610.55
94.82:044
95.7..0.3
86.22 1.8
InCoMOON
92.8:103
93.0:10.3
94.7:10.88
95.1
94.2
95.1:10.5
36.0:30.9
36.8+11.
H.2 DATASETS
We conduct experiments on Rashion-MNISS SVHN, CIFAR-10, and CINIC-10. CINIC-10 is dataset of the mix of CIFAR-10 and ImageNet within ten classes. We use 3x224x224 with the ViT models and 3x32x32 with the ResNet models for all datasets.
H.3 HYPER-PARAMETES
We deploy stage splitting for ResNets and obtain five sub-models, which can be recognized as ResNet10, ResNet14, ResNet18, ResNet22, and ResNer26. For the pre-trained ViT models, we employ layer splitting and obtain five sub-models, which are VIT-S/8, VIT-S/9, ViT-S/10, VIT-S/I1, and ViT-S/12 from the PyTorch Image Models (timm) Our implementations of FedAvg, FedProx, FedNova, Scaffold and MOON are referred to (Li al. 2022). We use Adam optimizer with a learning rate of 0.001, 31 = 0.9 and 82 = 0.999, default parameter settings for all methods of ResNets. The local training epochs are fixed 5. The batch size is 64 for all experiments. Furthermore, the global communication rounds are 500 for ResNets, and 200 for ViTs for all datasets. Global communication rounds for MOON and InCoMOON are 100 to prevent the extreme overfitting in -NIIT. The hyper- hyper-paramen parameter y for FedProx and InCoProx is 0.05 for ViTs and ResNets. We conduct our experiments with 4 NVÍDIA GeForce RTX 3090s. All baselines and their InCo extensions are conducted in the same hyper-parameters. The settings of hetero splitting for ScaleFI. followed the source codes
H.4 MODEL. SIZES
We demonstrate the model sizes for each client model in Table 7 Table 8 and Table 9
H.5 ERROR BARS OF INCo AGGREGATION
We illustrate the error bars InCo Aggregation and the results from model-homogeneeus baselines (not use stage or layer splitting in the model heterogeneous environment) in Table For the model- heterogeneity methods, we demonstrate the error bars of InCo Aggregation in Table These results show the stability of InCo Aggregation. In all cases of ResNets and many cases of ViTs, the worst
25

---

Published as a conference paper at ICLR 2024
Table 6: Test accuracy of notee--tetry methods with 100 clients and sample ratio 0.1. We shade in gray the methods that are combined with our proposed method, InCo Aggregation. We show the error bars for InCo Aggregation in this table.
Fashion- MNIST
SVHN
CIFARIO
Splitring
Methods
0.5
1.0
or 0.5
a
1.0
0.5
a 1.0
HeteroFT.
88.911.0
89.740.7
10.54::.6
92.241.3
65.2433.
68.41366
Hetern
+lnCo
90.0:11.2
90.41 1.1
92.11110
9..5.1
68.2.13.8
71.2.13.4
InclusiveFI.
89.1
89.8 1.0
88.6422000
90.0:22.2
65.7.3355
68.411.3
Stage
+InCa
90.1..1.5
90.5.1 1.3
9..6.1..7
90.9.11.9
69.1
72.3.13.1
ResNet
FedRolex
88.21:10
90.22-0.8
1.
91.6
64.7:t:41
72.3+3.00
Hetern
+lnCa
90.4.1.4
91.3.1 1.1
9..811.55
93.4.11.6
6..911..9
7..6.1
ScaleFL
90.9:0055
91.0=20.4
92.611
92.9:1099
71.1
74.7#3.1
Hetero
+lnCo
91.51100
91.7+1111
93.4:00.9
93.6::0.9
73.8+:322
76.1+:2.6
N/A
AllSmall
83.54:177
84.0+1.7
72.1433.5
81.0+2.9
39.2::200
44.922.3
N/A
AllLarge
91.8:0055
92.5+0088
93.4+:0.8
93.8:00.5
79.6:2299
82.5+100
Table 7: Model parameters for different architectures of ResNets (Stage splitting).
ResNets (Stage splitting)
ResNer10
ResNet14
ResNet18
ResNet22
ResNer26
Params
4.91M (x0.281)
(x0.
(x0.611)
17.08M (x0.979)
17.45M (x1)
Layer 6
0.0
TedAup
Layer?
(a) CIFARIO with a = 0.5.
(b) InCoAvg.
(c) HeteroAvg.
(d) FedAvg.
Figure 19: CKA layer similarity and Heatmaps of ViTs. (a): The layer similarity of different methods.
(b) to (d): Heatmaps for different methods in layer 6 and layer 7.
results of InCo Aggregation are better than the Averaging Aggregation, demonstrating the efficacy of InCo Aggregation.
H.6 DIFFERENCES BETWEEN ADDING NOISES AND INCO GRADIENTS.
The convergence speed of InCo gradients surpasses that of the other two methods, as illustrated in Figure 20a Table 200 demonstrates that InCo gradients outperform other methods across different datasets. The primary distinction between InCo gradients and adding noises lies in their ability to determine the precise gradient modification for each node in the model. Gaussian noise lacks the capability to specify the exact modification required for each node, leading to less controlled and targeted adjustments. This is evident in Figure 20b where the Frobenius norm of noises is larger and exhibits greater instability compared to InCo gradients.
H.7 LAYER SIMILARITY AND HEATMAPS OF VITS
Figure 19 illlustrates the layer similarity of the last four layers, along with the corresponding heatmaps for Layer 6 and Layer 7. Furthermore, Figure 19a demonstrates that InCo Aggregation significantly enhances the layer similarity, validating the initial motivation behind our proposed method. Addition- ally, since the disparity in layer similarity between Layer and Layer is minimal, the heatmaps for these layers do not exhibit significant differences, depicted in Figure 19h through Figure 19d
26

---

Published as a conference paper at ICLR 2024
Table 8: Model parameters for different architectures of ViTs (Layer splitting).
WT's (Layer splitting)
Sizes
VIT.S/8
VIT-S/9
VIT-S/10
VIT-S/11
VIT-S/12
Params
14.57M (x0.672)
16.34M 754)
18.12M (x0.836)
19.90M (10.912)
(1.77((x1)
Table 9: Model parameters for different architectures of ResNets (Heterogeneous splitting).
ResNets (Hetero splitting)
Sizes
ResNer26
Parums
0.07M (x0.004)
0.28M (x01))
1.10M 0.06)
4.37 (x0.25)
17.45M (x1)
H.8 MORE ABLATION STUDIES AND ROBUSTNESS ANALYSIS.
We conduct additional experiments on different baselines to demonstrate the effectiveness of InCo Aggregation. Figure 21 to Figure 23 present the results of the ablation study for FedProx, FedNova, and Scaffold, incorporating InCo Aggregation. These results highlight the efficacy of InCo Aggrega- tion across different baselines. Additionally, Figure 24 and Figure 25 illustrate the robustness analysis for FedProx and Scaffold. In Figure 24 and Figure 25 InCoProx and InCoScaffold consistently obtains the best performances across all settings. These experiments provide further evidence of the efficiency of InCo Aggregation.
I LIMITATIONS AND FUTURE DIRECTIONS
The objective of this study is to expand the capabilities of momoe--hmmoesseus methods to effectively handle model- heteroe-eeto FL environments. However, the analysis of layer similarity reveals that the smallest models do not derive substantial benefits from InCo Aggregation, implying the limited extensions for these smallest models. Exploring methods to enhance the performance of the smallest models warrants further investigation. Furthermore, research mainly focuses on image classification tasks, specifically CNN models (ResNets) and Transformers (ViTs). However, it is imperative to validate conclusions in the context of language tasks, and other model architectures such as LSTM Hochreiter & Schmidhuber (1997). Additionally, is important to consider that the participating clients in the training process may have different model architectures. For example, some clients may employ CNN models, while others may use Vision Transformers (ViTs). We believe that it is worth extending this work to encompass wider range of tasks and diverse model architectures that hold great value and potential for future research.
27

---

Published as a conference paper at ICLR 2024
FashionMNNSST
SVHN
CIFARIO
Methods
0.5
1.0
0.5
1.0
0.5
1,0
HeteroAvg
86.7
gradients
(c) Accuracy for different datasets. Figure 20: Convergence speeds, Frobenius norm adding noise and InCo gradients (InCoAvg), and accuracy results for different datasets. (a): Convergence speeds of HeteroAvg. adding noise and InCo gradients. (b): Frobenius norm of noise and InCo gradients. (c): Accuracy for different datasets.
(a) Convergence speeds.
(b) Frobenius
28
77
210.
HAD metechrss
wit
MestrsPrott
Intalrat
MeatrePron
(a) Reshion-MNIST
(b) SVHN.
(c) CIFAR-IO.
(d) CINIC-10.
Figure 21: Ablation studies for InCo Aggregation for FedProx. The federated settings are the same as Table 2
725
70.0
477
13.0
62.
600
246.5
575
55.0
200.5
we
wee
HBO
150
u
(a) Fashion- -NISS..
(b) SVHN.
(c) CIFAR-IO.
(d) CINIC-10.
Figure 22: Ablation studies for InCo Aggregation for FedNova. The federated settings are the same as Table 2
75
we NAD
N60 Seterncctthe
BAG
vEe HAC
(a) Fashion- Fashion-MNIST.
(b) SVHN.
(C) CIFAR-IU 10.
(d) CINIC-10.
Figure 23: Ablation studies for InCo Aggregation for Scaffold. The federated settings are the same as Table
2
76
InCoProx
HeteroProx
72
Accuracy
72
Accuracy
70
70
68
66
66
InCoProx
64
62
HeteroProx
32
64
128
256
1*std
2*std
3*std
4*std
5tstd
BatchSize
Noise Std
(a) Different batch sizes.
(b) Different noise perturbations.
Figure 24: Robustness analysis for InCo Aggregation for FedProx in CIFAR-10.
28

---

Published as a conference paper at ICLR 2024
InCoScaffold
70.0
InCoScaffold
HeteroScaffold
67.5
HeteroScaffold
Accuracy
65.0
Accuracy
62.5
60.0
57.5
55.0
60
52.5
128
256
1*std
2'std
3*std
4*std
5*std
BatchSize
Noise Std
(a) Different batch sizes.
(b) Different noise perturbations.
Figure 25: Robustness analysis for InCo Aggregation for Scaffold CIFAR-10.
29


Reviews:
[{"review_id": "zsKl8xrL5r", "reviewer": "Reviewer_vLVW", "summary": "The authors observe that layer similarities are related to accuracy for some FL models. They also observe that layer similarities are related to gradient distribution smoothness. Based on these observations, they modify the learning approach to increase similarities between shallow and deep layer gradients. This modification results in improved accuracy for a variety of FL methods and datasets. They also provide for layer splitting and other engineering necessities to evaluate their ideas, but I found the observations and modification above to be the most interesting and novel.", "questions": "1) Is there a fundamental justification for the form of expression 1, or might other expressions perform as well or better?\n\n2) Why did you decide to simply constrain gopt from opposing g0 (with an inequality) instead of imposing a cost that increases with decreasing dot product?\n\n3) What is the relationship between your findings and those regarding the contribution of residual connections in ResNets?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ This is an interesting paper to read.\n\n+ The paper combines experimental approaches to scientific discovery and system engineering to use these discoveries to produce improved accuracy.\n\n+ The visualizations of findings and diagrams are clear.\n\n+ The writing is well organized.", "weaknesses": "- The reasons for the relationships observed by the authors are not well explained, although Section 5.4 makes a mostly unsuccessful attempt at doing so. The authors find that high-accuracy models tend to have particular properties and push the learning process to produce those properties without well explaining why the properties result in accuracy. They do, however, demonstrate that their approach works so it's a question of depth of understanding, not merit.\n\n- Section 3.3 seems central to enabling improvement but it is relatively short, without much justification for design decisions. It states what is done but now why this is the most appropriate approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sJ08D96EM1", "reviewer": "Reviewer_e8EV", "summary": "The paper highlights the positive correlation between client performances and layer similarities in a federated setting, specifically how similarities are higher in shallow layers of the network and steadily decrease as one gets to the deeper layers as well as how smoother gradient distributions correspond to higher layer similarities. \nKeeping these ideas in mind, the paper proposes InCo Aggregation, which leverages cross-layer gradient similarities to improve the similarity across clients in deeper layers, without the need for additional communication between clients.\nOverall, InCo Aggregation is validated on both model-homogeneous and heterogeneous methods to highlight its impact in improving performances across the board.", "questions": "- Could the authors define the notion of system level heterogeneity, using model and data heterogeneity, earlier and provide connection points to how the proposed method would address these issues. Having these points described earlier would allow the reader to grasp the importance of the observations described later on.\n- Could the authors provide gradient plots like in Figs 2 and 3 for Stage 2's layers as well? Drawing a parallels to behavior across Stages would be helpful in establishing the consistency of the observations on smoother gradients and how they relate to similarity values.\n- Could the authors provide an explicit definition of which layers can be considered deep vs.shallow? If the nomenclature implicitly defined in \"Cross-environment Similarity\" is to be maintained, could the authors provide an explanation of whether this pattern carries over to ViTs as well?\n- Could the authors discuss further about similarity patterns in ViT's and how this impact the observations and InCo aggregation as a whole?\n- Could the authors provide more detail explanations for the exact setups used to generate Fig. 1, 2, and 3?\n- Could the authors provide more insight into how Fig. 3 would vary when tested across multiple trials? This could help remove the uncertainty caused by SGD noise.\n- Could the authors provide the standard deviation values for InCo-based methods?\n- Given the variation in performances in Table 3 and the original values cited under FedROLEX, ScaleFL, etc., could the authors provide a detailed breakdown of how the experimental settings differ from the original works?\n- A discussion on how and where it isn't applicable (e.g., Complete model heterogeneity, where models are restricted to be within the same family of backbones or in cases where weight matrices do no align) is critical to understand and apply the proposed method.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The observations on similarity at varying levels of deep networks across clients could potentially establish an observation mechanism for federated settings to analyze the impact of changes across multiple axes like data settings, model heterogeneity, etc.\n- Figures 4 and 6 do a good job at conveying concepts surrounding various model splitting methods and gradient divergence, respectively.", "weaknesses": "- \"System level heterogeneity\" is mentioned multiple times and is loosely defined through the early portion of the manuscript (Pg. 1, Paragraph 1). Over the course of reading the paper, one can figure out data and model heterogeneity are the relevant axes along which system level heterogeneity is defined. Defining these ideas earlier and more concisely would allow the reader to contextualize the problem domain and understand how the solution being proposed fits within its scope.\n- While the paper discusses the relationship between gradient similarity and performance/accuracy, and gradient similarity with smoother gradients, the key justifications for choosing to use smoother gradients are (a)  lack of a shared database, (b) Features would increase communication overheads, and (c) correlation between similarity of gradients and smoothness. Are there stronger correlations between the level of similarity (actual value) to the peak density or other statistics of the gradients?\n- There doesn't seem to be a an explicit definition of \"deep\" vs. \"shallow\" layers. Implicitly, within each stage there seem to be shallow and deep layers (Pg. 3, Cross-environment similarity). In general, there are certain experimental settings necessary to fully understand the figures plotted through the course of the manuscript that seem to be missing.\n- Given the specific model heterogeneity settings under which InCo Aggregation is applicable, a discussion on how and where it isn't applicable (e.g., Complete model heterogeneity, where models are restricted to be within the same family of backbones) would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dWZYK3GtsU", "reviewer": "Reviewer_ibVy", "summary": "The paper tackles the challenge of heterogeneous models within federated learning, uncovering a notable pattern where shallow layers exhibit similar gradient distributions, in contrast to the disparate distributions in deeper layers. The authors further observed that higher gradient similarity corresponds to improved accuracy. Capitalizing on these insights, a novel aggregation method is proposed and substantiated through comprehensive experiments, demonstrating the efficacy of the proposed approach in navigating model heterogeneity.", "questions": "1. I wonder if the proposed approach can be applied to complex models. \n2. I wonder how cross-layer gradient adjustments ameliorate the effects of model heterogeneity.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The author presents intriguing findings regarding the relationship between layer similarity and model performance.\n2. By refining the direction of deep layers, which exhibit lower gradient similarity compared to shallow layers, the author enhances model performance.\n3. The proposed method demonstrates versatility, adaptable to various Federated Learning (FL) schemas.\n4. The effectiveness of the proposed method is substantiated through extensive experimentation.", "weaknesses": "1. The utilization of a model with merely three convolutional layers is unconvincing; larger models should be employed in primary experiments to validate the findings.\n2. The manuscript could benefit from a more coherent narrative, including background on previous works, an introduction to the CAK similarity metric, and a discussion on why the proposed method outperforms state-of-the-art (SOTA) methods, especially in handling model heterogeneity.\n3. Figure 6 lacks clarity; indicating the positions of client and global optima could elucidate the depicted concepts.\n4. It is imperative to delineate the problem definition and notations before introducing the method, ensuring a logical flow and better comprehension.\n5. The manuscript does not adequately explain how cross-layer gradient adjustments ameliorate the effects of model heterogeneity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6SLD93JGIo", "reviewer": "Reviewer_ky42", "summary": "this paper discover three interesting observations based on the exploration between homogeneous and heterogeneous FL settings. Then the authors propose InCo Aggregation methods, inpried by these observation and demonstrate the proposed method can be tailored to accommodate model-homogeneous FL methods and achieve better performance.", "questions": "1. where is CKA from in Fig. 1 (c). Is it the average value of all stages or deep/shallow stage?\n2. Can you provide a more detailed comparison of InCo Aggregation with other state-of-the-art methods for handling system heterogeneity in FL? How does InCo Aggregation compare in terms of performance, communication overhead, and computational complexity?\n3. The paper does not provide a detailed analysis of the computational and communication overhead of InCo Aggregation, which could be a significant factor in large-scale FL applications. This limits the practical applicability of the proposed approach.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easily understandable, effectively communicating the research in a clear and coherent manner.\n\nThe discovered observations are interesting and valuable for future research, providing a foundation for further investigations and potential advancements in the field.\n\nThe experiments conducted in the paper are sufficient, with an appropriate and comprehensive setup that collects relevant data to support the claims and conclusions.", "weaknesses": "1. CKA is an important metric in this paper. The authors should explain it in more details.\n2. Model splitting is proposed to facilitate model heterogeneity. What if the layer-wise gradient sizes of different models are not the same, how do you conduct cross-layer gradients mergence?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)