Heterogeneous Personalized Federated Learning by Local-Global Updates Mixing via Convergence Rate
Authors: Meirui Jiang
Anjie Le
Xiaoxiao Li
Qi Dou
Venue: ICLR 2024 poster (2024)
Keywords: Personalized Federated Learning
Heterogeneous Data
Feature Distribution Shift
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose a novel approach to achieve model personalization on heterogeneous features by local-global updates mixing via the convergence rate induced by NTK.

Abstract: Personalized federated learning (PFL) has emerged as a promising technique for addressing the challenge of data heterogeneity. While recent studies have made notable progress in mitigating heterogeneity associated with label distributions, the issue of effectively handling feature heterogeneity remains an open question. In this paper, we propose a personalization approach by Local-global updates Mixing (LG-Mix) via Neural Tangent Kernel (NTK)-based convergence. The core idea is to leverage the convergence rate induced by NTK to quantify the importance of local and global updates, and subsequently mix these updates based on their importance. Specifically, we find the trace of the NTK matrix can manifest the convergence rate, and propose an efficient and effective approximation to calculate the trace of a feature matrix instead of the NTK matrix. Such approximation significantly reduces the cost of computing NTK, and the feature matrix explicitly considers the heterogeneous features among samples. We have theoretically analyzed the convergence of our method in the over-parameterize regime, and experimentally evaluated our method on five datasets. These datasets present heterogeneous data features in natural and medical images. With comprehensive comparison to existing state-of-the-art approaches, our LG-Mix has consistently outperformed them across all datasets (largest accuracy improvement of 5.01\%), demonstrating the outstanding efficacy of our method for model personalization. Code is available at \url{https://github.com/med-air/HeteroPFL}.

Published as a conference paper at ICLR 2024
HETEROGENEOUS PERSONALIZED LEARNING BY LOCAL-GLOBAL UPDATES MIXING VIA CONVERGENCE RATE
FEDERATED
Meirui Jiang
Anjie Le
Department of Computer Science and Engineering The Chinese University of Hong Kong mr jiangecse cuhk. edu. hk
Department of Computer Science and Engineering The Chinese University of Hong Kong a j.je@echkk. edu
Xiaoxiao Li
Qi Dou"
Department of Electrical and Computer Engineering The University of British Columbia xiaoxiac li@ece. ubc. ca
Department of Computer Science and Engineering The Chinese University of Hong Kong qidouêcuhk. edu.hk
ABSTRACT
Personalized federated learning (PFL) has emerged as a promising technique for addressing the challenge of data heterogeneity. While recent studies have made notable progress in mitigating heterogeneity associated with label distributions, the issue of effectively handling feature heterogeneity remains open question. In this paper, we propose a personalization approach by Local-Global updates Mixing (LG-Mix) via Neural Tangent Kernel (NTK)-based convergence. The core idea is to leverage the convergence rate induced by NTK to quantify the impor- tance of local and global updates, and subsequently mix these updates based on their importance. Specifically, we find the trace of the NTK matrix can mani- fest the convergence rate, and propose an efficient and effective approximation calculate the trace of a feature matrix instead of the NTK matrix. Such approx- imation significantly reduces the cost of computing NTK, and the feature matrix explicitly considers the heterogencous features among samples. We have theoret- ically analyzed the convergence of our method in the over peee-paameterize regime, and experimentally evaluated our method on five datasets. These datasets present heterogeneous data features in natural and medical images. With comprehensive comparison existing state-of-the-ard approaches, our LG-Mix has consistently outperformed them across all datasets (largest accuracy improvement of 5.01 %), demonstrating the outstanding efficacy of our method for model personalization. Code is available at httpt:://////o /ttt::///ghhuue com/ med aeddaairrHeeeeee
INTRODUCTION
Personalized federated learning (PFL) aims to leverage the aggregated knowledge from other clients
to learn client-specific model that best fits its own data distribution (Smith et al. 20188 Arivazha-
gan et al. 2019: Li et al., 2021b Hanzely Richtárik 2021 Tan et al.) 2002).. Although many PFL
methods have tackled heterogeneity issues regarding label distribution shift (Jeong & Hwang 2022 Zhang et al. (002).. 2). computation limitation (Diao et al., 2021] (Setayesh et al. (023). model archi- how effectively address the feature 2021 Hsieh et al. 2020) still an open question. In practice,
tecture design MMakhija al., 2022 Wang et al. 2022).
heterogeneity issue (Kairouz et al.
feature distributions across client data often heterogeneous due variations in acquisition or generation conditions (Xu et al. 2021 Rieke et al. 2020 Li et al. 2021c). For instance, in health- care domain, medical images collected from different hospitals exhibit appearance misalignment as imaging protocol changes. Training one common global model for tackling various feature distribu-
"Corresponding author.

---

Published as a conference paper at ICLR 2024
tions can be challenging. Therefore, personalization of the global FL model to each individual client becomes imperative, but how to solve the feature heterogeneity is still unclear.
There have been many approaches proposed to improve the global model in order to tackle het- crogeneous features. Despite the promising performance, the converged global model may not be optimal for all clients (Kairouz al., 2021: Li et al. 2020a; /Sattler al. 2020). Another way is to train personalized models to overcome the feature distribution shifts. It is worth noting that deal- ing with heterogeneous features under PFL setting is largely different from doing it under standard FL. Specifically, PFL aims to fit clients' individual feature distribution, whereas the standard FL focuses on utilizing various feature distributions to enhance the global model. For instance, there are some FL methods proposed to tackle the heterogeneous data by aligning the feature distribu- tion over clients to promote global aggregation (Li et al. 2021c a), and rectifying client gradient direction to avoid aggregated gradients being distracted (Gao et al. 2022, Karimireddy et al. 2020)_ These methods mainly focus on global updates. However, for PFL., is essential to consider both local and global updates and their interplay relations for achieving personalization.
To investigate the relations between local and global updates, we consider their effects during fed- erated training. The global update typically is the aggregation (e.g., weighted averaging) of local updates, while the local update solely relies on the client's local data. In federated training, the global update contains common knowledge from all clients' data, thereby aiding in minimizing the error of joint data distribution. However, it may not precisely align with the direction of the local update when the data exhibits heterogeneity. For local update, it specifically minimizes the local error, but suffers from limited local data size. From a personalization standpoint, the key incentive to minimize the local error with the help of other clients' data (i.e., the common knowledge). In this case, an ideal solution would be leveraging local updates to rectify the potential adverse effects induced by data heterogeneity in global updates. promising solution is mix the local and global updates, while the question is how to determine the mixing ratio.
One related work has proposed to mix the local and global models and select the ratio based on the differences observed between the mixed model and the global model (Deng et al. 2020). How- ever, considering that the local/global update strongly correlates with the input data, we argue that attention must be given to the data aspect in the context of feature heterogeneous PFL. To answer this question, we draw insight from the Neural Tangent Kernel (NTK) (Jacot et al. 2018: Huang et al. 202I), which leverages the dot product of input data for measuring the convergence of neural network training. By calculating the NTK matrix when employing local and global updates, we can take the NTK-based convergence rate as guiding factor adaptively adjust the mixing ratio.
In this paper, we propose to achieve model personalization by Local-Global updates Mixing (LG- Mix). Our key contribution is determine the mixing ratio via NTK-based convergence. Specifi- cally, we theoretically show that the trace of NTK is an effective measurement of the convergence rate. By leveraging the convergence rate induced by NTK, we can assess the importance of local and global updates, and perform mixing based on their importance. We propose an efficient and effective approximation calculate the trace of a feature matrix instead of the NTK matrix. This approximation not only reduces the computational cost, but also explicitly considers heterogeneous features among samples. We conduct comprehensive experiments to demonstrate the efficacy of our method, including both performance comparisons and analytical studies. We also theoretically analyzed the convergence of our method in the over- wwer-parameterized regime. Our experiments include three computer vision datasets with heterogeneous features (diverse image styles/appearaacce)) and two real-world medical image datasets. Our method consistently outperforms state-of-the-an ap- proaches across all datasets, showing its effectiveness in personalization.
2 RELATED WORKS
Federated Learning on Heterogeneous Data. To promote the training of global model on hetero-
geneous data, various techniques have been proposed, such as regularizing local model training (Li
et al.
2020b
Durmus et al.] 2021 Li et al.) 202 facilitating model optimization (Karimiredly
jet al.
2020;
Reddi et al.| 2021: Tran Dinh et al.) 2021). enhancing aggregation algorithm (Wang
le al.
2020; Pillutla al. 2019). improving feature normalization (Li et al. 2021c Reisizadeh
ct al.
2020). etc. For instance, SCAFFOLD (Krrmireedy et al. 2020) proposed a new optimization
algorithm reduce variance in local updates. Later on, FedNova (Wang et al. 2020 suggested
2

---

Published as a conference paper at ICLR 2024
using normalized stochastic gradients for global model aggregation, and MOON (Li et al. 202 proposed using contrastive learning on latent feature representations to enhance the aligmment be- tween local and global models. FedDC (Gao et al.] 2022) improved SCAFFOLD by dynamically updating the client objective function. However, when aiming to maximize the performance for each local client, learning one common global model may not be optimal solution.
Personalization in Federated Learning. The PFL aims to utilize data from multiple clients to learn a personalized model for each client. Existing methods have performed personalization by leveraging: meta-learning (Fallah et al., 2020) Acar et al. (02T). multi-task learning (Smith et al.,
2018
Li et al., 2021b), model parameters decomposition (Collins et al.) 202T Oh al.) 2022). model mixture Deng et al. 2020 Hanzely et al. [220 Hanzely & Richtárik 2. Bayesian treat-
ment Kotelevski et al. 2022 Ozkara et al., 2023), etc. For example, PerFedAvg (Fallah et al.
2020
proposed to seek meta- model that adapts to each client's 's local dataset. APFL. (Deng et al.,
2020 and L.2GD (Hanzely et al., 2020) proposed to mix the local and global model for personal- ization. FedAlt (Pillutla et al., 2022) proposes to personalize partial model layers. FedBABU (Oh ctt al. 2022) and FedRep (Collins et al. 202T) propose a similar idea of decoupling the learning model into a model body and a local head, and the head is personalized. Recently, FedHKD (Chen et al., 2023) proposes to use knowledge distillation to train local models and share hyper-knowledge methods (e.g.. APFI., L2SGD)
instead of parameters. However, although some optimization-base can be extended for heterogeneous features, they are not specifically designed for feature hetero- geneity problems. The issue of feature distributional shifts in PFL remains under-explored.
3 LG-MIX: LOCAL-GLOBAN UPDATES MIXING VIA CONVERNGENCE
In this section, we begin by describing the model training process involving local and global updates. Then we present our proposed method that performs update mixing guided by the convergence.
3.1 LOCAL AND GLOBAL UPDATES MIXING
Assume we have N clients joining the PFL for T communication rounds, and we use e to denote the client index. Each client will perform K local update steps. We denote the aggregated server model at the 1-th communication round as vu() and the client model in round and local step k as we(t, k). Denote Se as the data index of client c, assuming there are n input data and label pairs [E., } E Fd x [R which follow the global distribution D. Note that S₁ U... U Sx = [n],
and S; n ns; = Φ. The data of client e is {(r,, y/s) : i E Se), and it follows distribution De.
The local update is defined as the weight changes before and after local client training. Consider the local model W.. the local update and global update at t-th round can be expressed as:
(w, = (t, K)- (.((,,), 0), Au(t) = n(t + 1) - m(t).
(1)
Here, the global update takes the format of aggregation using local updates in practice, i.e., Au(() Σ. P.A.w(t)), where Pe denotes the client importance (c.g., proportional to client sample number) and the sum over all clients equals to 1. In FL, typical paradigm is that each client receives the global update during communication and then use it to update the local model. However, clients may experience covariate shift concept drift in the real world, i.e., given the joint probability Pc y) of input x and label y. we have (z) varies even if Po (z) is the same or (x(l)) varies across clients while Py()) is unchanged. Therefore, D does not necessarily represent the actual distribution of a client c's data D. well. That is, the global update may not concisely align with the direction of the local update, solely using the global update is not sufficient to minimize the error of each client.
To overcome the potential shift inside the data distribution. we propose to perform the PFL by in- corporating the local update into the global update to fit D... The insight is that the global update contains common knowledge from all clients and conveys global information, while the local update specifically encompasses local information. By mixing local and global updates, we can make the best use of both local and global knowledge to maximize the performance for each client. Specifi- cally, we consider the following update regime for model personalization:
we(t + 1) + wc(() + (tt + (1 Ae(t))
(2)
where de is coefficient of choice to balance the local and global update. When de approaches 0 or 1, the update regime corresponds to the vanilla FedAvg and local gradient descent respectively. Next, we aim to find an optimal ratio to mix these two updates.
3

---

Published as a conference paper at ICLR 2024
3.2 MIXING RATIO CALCULATION BY WTK-CONVERGENCE
To find the optimal mixing ratio, the key point is to quantify which update is more important for model optimization. In other words, the optimal mixing should minimize the loss faster than other mixing ratios. Therefore, we propose to measure the convergence of using local/global updates, and use the convergence rate determine the mixing ratio. We measure the convergence by using the tool NTK, which has been widely adopted to analyze the convergence of modern neural networks.
To study the convergence of the gradient descent at time step we can measure the evolution of the error between the ground truth and model prediction at the t-th step with the help of the Gram matrix (t). The Gram matrix (t) is a function of parameter w and input z, which characterizes the optimization process of gradient descent (Du et al. 2019). This Gram matrix is often used as an empirical approximation to the NTK, which describes the dynamics of gradient descent in the infinite wide neural network (Arora et al. 2019a] Lee al.) 2020), and the spectral property of the Gram matrix also governs convergence guarantees for networks in the finite width case (Huang & Yau 2020; Zhang et al. 2020; Brand et al. 202I). For instance, consider two-layer neural network with ReL. activation, the Htt) for input data pair 2i, can be denoted as: Ht) = m Ere(m) (x."
where m is the number of hidden nodes, r is the node index. We omit the footnote e in w for ease of notation. Intuitively, the Gram matrix captures the correlations between the training samples in the network evolution dynamies. Following Du et al. 2019), we express the evolution of prediction error for one step of gradient descent as:
1
(3) f ; Rd + IR is the model function. Based on (3), we have the following observation, which illustrates the significance of between the convergence rate and the trace of the Gram matrix.
y - (t + 1) = (I TH - y(t + 1)),
where y denotes the label, y()) = (ww()), æ)
Proposition 1. With the assumption that the error vector E(t) y y(t) can be regarded random vector distributed uniformly in the space, by decomposing (t)) and E(t) into the eigenbasis of H()), we note that in gradient descent, the prediction error has an approximate convergence rate of (1 - 2n tr( (t))nn), where r is the learning rate and is small, and tr( (H)))) is the trace of Htt).
Proof Sketch: Consider the cigen-decomposition H(t) = (f)uu (t). then E(t) st(t),,(t)u)
(of (t). Then we have E(t + 1)
(I (H(())((t)
Σl= (1
(s.)(u (t)(t()))) which implies [|E(t 1))12 Ei= ns:) (o.
Then for n small and error vector distributed uniformly in space, we have 1|E(t + 1)11 = (1- ()(((((()))m)))l((())
This proposition means that the trace of the Gram matrix can serve as an effective metric for as- sessing the convergence rate of the model on the data samples. Consequently, the convergence rate can be quantified by leveraging the trace of the matrix. Let us denote the Gram matrix of the model obtained through global updates H, and the Gram matrix acquired through local updates as He. Intuitively, if the convergence rate using local updates surpasses that of global updates, a greater emphasis on local updates becomes necessary. According the proposition, for the same amount of sample, if tr( (H+))= > tr( (H), then this means the local update converges more rapidly than the global update. This proposition aligns with our intuitive understanding. Following this idea, we would like to give the update with higher convergence rate a greater weight in the mixing ratio. To be more specific, for client at t-th step, we can calculate the mixing ratio as:
At tt) / (t)) + (t)) By taking the ratio back to Eq. we have completed the regime to generate the personalized model.
(4)
3.3 ALGORITHM IMPLEMENTATIOE IXPLEMENTATION
In the preceding section, we have introduced the utilization of the trace of the Gram matrix for determining the mixing ratio. However, the computation of the Gram matrix H in practical networks poses significant challenges and is often computationally infeasible (Fort et al. 2020 Mohamadi et al. 2023, Holzmüüler et al. 2023). Specifically, the size of the H matrix increases rapidly as the number of samples grows, making the computation impractical, particularly for large datasets. In this case, we propose to approximate the calculation of tr( ()) by two steps.
First, we draw upon previous studies (Seleznova et al.) 2023 Kirichenko et al. 2023) that empha- size the significance of the last layer in the dynamics and performance of neural networks. In our
4

---

Published as a conference paper at ICLR 2024
Algorithm 1: LG-Mix: Local-global updates mixing algorithm
Input: communication rounds T, number of clients N, local update steps K, client learning rate n/, global learning rate lg.. mixing ratio (A.) feature matrix { Σ. for f 0, .....
do for c 1,2, .N in parallel do for k = 0, ....;. K do
sample a batch of data pairs {(z,, w:) E S. (1ri wr)) // also output feature he(s)) += (xx))((;;)
we(l, 1) - SGD(walls Ee += (c(
update feature matrix
Awa(t) we(t, K) (t,t, 0)
send client update server // global model update
u(t + 1) = =() + Au(t). where Au(() = Ee (1)
for 1, 1,2,
in parallel do
(e) (t)) (t) + tr(((t))
// Eq.
11
w.(t + + w.(t) (o + (1 Ao(())au(())
// Eq.
Output: The personalized models [we)om J== and the global model u.
approximation, we consider the evolution of the last layer. For the matrix H. we use (1)
m Ere(m (h, (1;)) as a approximated representation, where h denotes the parameters up to the penultimate layer. This approximation also justified by the fact that h, (,,(z)),,,z,,) is always factor of H, (V fwo (x)), V (I;)), irrespective of the activation function employed. Here, w the parameter of the last layer, and ff gives the prediction of the whole model. Notably, an exist- ing work (Seleznova et al.] 2023) has also shown that the empirical NTK with entries of last-layer features highly aligns with the original NTK in terms of training dynamics.
Second, since our objective is to compute the matrix trace, it is unnecessary to calculate the complete Н. By leveraging property of linear algebra, namely, (r(h(x)²(ha)) simplify the trace calculation as follows:
tr(b(z))(())²). we can
1 tr( ur(i((t) = - Eieto E hy (x = - x tr()(tt),
(5)
where Σ(t) (.=n)(x;)haa))) is feature matrix in Rmxm considering features among all samples. Note that H()) has size of x n, while E m m. As result, the computational cost is significantly reduced since the number of samples is typically much larger than the last-layer feature dimension (e.g., 256 or 512).
With the aforementioned approximation, we can approximate the trace of the H matrix obtained through local updates as tr( (1)) R (()).()),, where Ex(() = Ei (a)))))a(x))) the case of the global H matrix, since the data is distributed and not directly accessible, we propose employing features from local data in conjunction with the global model u. Consequently, we approximate tr(H(t) (t)) tr (t)() where (t) Σ h(I;)h(I,))
Finally, the mixing ratio for client c at time step can calculated as follows:
(t tr) / (ur(E.(t) + tr(E(t)))-
(6)
To further stabilize the ratio, propose consider the history information, that is, Ac(t) (t). We have also studied the effects of such stabilization in our experiments.
CONVERGENCE ANALYSIS
We prove the following theorem that describes the convergence of our proposed PFL algorithm.
Theorem 1. For m = Ω( -4n log(n/))), randomly initialized parameters (i.e. w(0) ~ N(O, T)), and Tn O(\<kr'² kn)) ) Ml = (1), then with probability least 8 over the random initializa-
tion, we have for Vl.
lly ( 1) < lyy
- (1 (H) vt)]
> (He) (H))) - (t)))
where Ç == m and smin denotes the smallest eigenvalue of H matrix.
2N
5

---

Published as a conference paper at ICLR 2024
Note that lyy s(t)ll = Ee 11 (t) and v(t) (f(wn (1), x1 ,..,.,,........ (wi (1), onn )ny), (w2(1) 21); ... sw (4), Kny Ve(t) = (0, 0, f(wo(t), x (wo (n) ......
x
The proof is rather involved including bounding each of the terms in the recursion relation:
|y y(t + 1)ll lly =(t) 2 y(t))² (y(t + 1) v(t)) l + 1) - v(t]l1
We defer the details of the proof to Appendix B
Discussion on the convergence result. To gain intuition into the convergence result, we consider the case when À is not homogeneous among clients, and we consider sub-optimal inequality as can be deduced from Appendix [8]
lly y(t + 51y 1- u(t)l1 (ny (1 Amar (t)) (t)l))matttt)s (H) 11
v(t)]1 - Σ, s
(Ha) 11 (t]l
Then by rewriting (1Ha) Smin + 8c, and 11 vo(()]12 + Byy - =(1)]1 2 we have
(H)
(1 where A₂ denotes the average value of {te). With the first two terms in the square bracket being relatively fixed, we observe that with c. = o(1), larger value of Xa is preferred for larger value of &
llv (|v-v(++1)] 1 (ly- (1)) 77 (1 Xmmx (t))s
+X.s + N Σ [x
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets. We evaluate our approach on four classification datasets, including (1) Digits-5 (Zou et all, 2020 Li et al. 2021c) with digits images showing drastic differences in font style, color, (Gong et al., 2012) with images acquired in different cam- eras environments; (3) DomainNer (Peng et al. 2019) with different image styles; (4) Came- Iyon?7 (Bandi et al., 2018) with histology images with different stainings from 5 hospitals, and one segmentation task the Retinal dataset which contains retinal fundus images from 6 different sources (Fumero et al.) 2011 Siiaawamy et al. 2015;; Almazroa et al. 201 & Orlando et al., 2020).
and background. (2) Office-
As shown in Fig. I each client represents data source, and data are heterogeneous across clients.
Digltss 2
Compared methods and evaluation metrics. We com- pare our approach with state-of-the-art (SOTA) PFL. methods, including APFL (Deng et al. (0200) and L2SGD (Hanzely et al., (020)) for personalization by mix-
3
Offke-Coltech20
ing local and global models, which shares similar idea of ours; FedAlt (Pillutla et al. (202)) for personalizing partial model layers; PerFedAvg (Fallah et al. (2220) for learning a meta- modal that adapts to each client's local data; FedBN (Li et al.] 2021c) for personalizing hatch normalization layers; FedFomo (Zhang et al., (021] for aggregating certain client models based on client contri- bution; FedBABU (Oh et al. 2022) and FedRep (Collins) et al. 202T) which propose to personalize the last model layer; FedHKD (Chen et al. 2023) for using knowledge distillation to personalize local models. For evaluation metrics, we report the accuracy for all classification tasks, and the Dice coefficient (Dice) and Hausdorff Distance (HD) for the segmentation task. All results are reported with mean and standard deviation across three independent runs.
DomoinNet
A
Comelyon
Retinal
Figure 1: Samples of five datasets show various styles and appearances.
Implementation details. In our implementation, all methods use the same training settings. We use the SGD optimizer with a learning rate of 0.01 and CrossEntropy loss for classification tasks and use Adam optimizer with learning rate of 1e with / (0.9. 0.99), dice loss (Milletari al. 2016) for the segmentation task. For more experimental results and training implementation details, please refer Appendix. D

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparison with SOTA PFL methods on classification datasets of Digits5 (different shapes and view angles).
(different strokes and colors) and
Datasct
Digits5
Office-Caltech10
Client
A
D
Avg.
A
B
C
Avg.
FodAvg (PMLR
98.85
89.95
95.8
99.28
88,70
94.52
78.18
56,99
51.04
61.58
61.95
(0.09)
(0.37)
(2.36)
(1,69)
(6:51)
(391)
(1.13)
APFL (Arsis)
96.62
90.07
96.94
99.12
91.17
94.78
78.05
53.48
64.76
57.91
63.55
(2.47)
(0.53)
(0.99)
(0.15)
(2:31)
(0:43)
(3211)
(0.56)
(17.94)
[13.46]
(0.42)
L2SGD (NeuzIPS 2020)
98.87
89.99
96.00
99.29
88.84
94.60
18.18
56.99
51.04
68.25
63.62
(0.05)
(0.07)
(02)
(0.01)
(0.3))
(0.3)
(2.36)
(1.09)
(6.51)
(1.36)
(1.18)
FedAIt (CML, 2022)
99.20
90.51
98.26
99.32
92.36
95.93
77.31
55.95
44.79
75.14
63,30
(0.05)
(0.27)
(0.38)
(0.03)
(8.88
(0.08)
(2.69)
(2.01)
(1.80
(4.27)
(0.37)
PerFedAvg (NeuriPS 2000)
99.05
89.55
96.11
99,23
89,53
94.69
71.73
56.55
61.46
74.01
65,94
(0.00)
(0.12)
(0.11)
(0.02)
(0.27)
(0.1)
(1.39)
(2:23
(4.77)
(3.5)
(0.61)
FedBN (ICLK 2021)
99.22
91.48
96.20
99,32
91,14
95.47
80,10
58,18
79.17
83.05
75.13
(0.2)
10.01)
(0.58)
(0.91)
(2.46)
(3.61)
(6.48)
(1.75)
FedFomo (CCLR 2021)
98. 83
90.49
95.95
99.33
89.19
94.76
74.76
$4.69
54.58
67.34
62.84
(0.04)
(0.44)
(0.22)
(0.04)
(0.36)
(0:15)
(0.9)
(1.09)
(2.01)
(2.84)
(0.99)
FedRep (CMI,
98. 86
90.35
95.99
99,53
89.15
94.78
78.53
57.74
56.25
67.23
64,94
(0.09)
(0-47)
(0.01)
(0.19)
(1.05)
(0:90)
(541)
(5.5)
FedBABU (ICLR 2022)
98. 85
90.15
95.75
99,53
88.74
94.60
77.84
57.74
56.25
67.23
64.76
(I.04)
(0.17)
(0.39)
(0.01)
(0.49)
(0.17)
(0.6)
(1.29)
(6:25)
(7.06)
(2.74)
IK (ICLR 2023)
98.11
90.38
95.41
99.47
90.06
94.69
77.14
56.52
53.98
65.48
63,28
(0.48)
(0.19)
(0:38)
(0.09)
(087)
(0:56)
(1.31)
(1:46)
(0.34)
(3.04)
(0.72)
LG-Mix (Ours)
99.29
92.35
98.66
99, 44
95.77
97.10
80.45
56,55
86.46
93,79
79,31
(0.08)
(0.05)
(0.02)
(0.05)
(0.04)
(0.6)
(0.68)
(1.8)
(2.59)
(1.05)
Table 2: Performance comparison with SOTA PFL methods on classification datasets of DomainNet (different styles) and Camelyon17 (different stainings).
Dataset
DomainNet
Camelyon 17
Client
B
E
Avg.
B
D
Avg.
FodAvg 2013
58.17
35.06
53.61
51.97
67.05
54.15
53.34
95.44
92.20
93,96
97.21
97.71
95.30
(0.0)
(1.69)
(2.52)
(0.90)
(195)
(0.72)
(0.5)
(1.8)
(0.67)
(088)
(0:37)
(0:33)
(028)
APFL (Arii)
58.11
36.87
53.29
46.60
69.02
58.00
53.65
96.27
92.94
96.70
70
97.95
98.25
96.42
(1.35)
12.00
(188
(7.54)
(3.02)
(0.51)
(.60)
(0.29)
(1.23)
(0.4)
(0.13)
(0.15)
(0.25)
L2SGD (NewIPS 2020)
59.00
34.60
53.88
50.33
69.82
56.02
53,94
96.67
93.02
94.71
97.55
97.68
95.93
(2.13)
(1.28)
(2.35)
(0.15)
(0.13)
FedAIt (CML)
59.82
37.14
58.47
58.57
72.45
54.27
56.79
98. 10
95.51
98.41
98.80
98.74
97.91
(0.48)
(0.97
(1.47)
(3,65)
(2.0)
(0.73)
(0.99)
(0.18)
(0.13)
(Q.1))
(0.01)
(0.11) (0.06)
PerffedAgg (NeutPS 2020
59.57
35.42
55,99
48.47
67.60
56.08
53.85
96.71
93.05
95.06
97.65
97.92
96.08
(1.52)
(0.99)
(1.06)
(0.31)
(0.92)
(2.64)
(0.39)
(0.32)
(0.41)
(OUIT)
(04)
(0.2)
(0.3)
FedBN (CC) 2021)
58.17
36.94
55.61
69.10
73.25
53.31
57.73
96.65
92.84
94,22
97.55
97.60
95.77
(0.87)
(1.14)
(1.75)
(1,91)
(361)
(4.68)
(0,89)
(0 49)
(0.45)
(0.13)
(0.30)
(021)
FedForn (00))
59,28
36,38
56.43
49.40
69.33
54.69
96.67
92.25
95.18
97 60
96,42
95.62
(2.37)
(1.07)
(1.58)
(4.5)
(2.19)
(1.09)
(1.38)
(0.34)
(0,51)
(O.1))
(0.16)
(0.47)
(0.13)
FedRep (CML.
60.08
36.33
56.36
47.80
67.98
58.78
54,56
97.07
93.64
96,79
98.12
98.28
96.78
(2.05)
(0.15)
FedBABU (CC) R 2012)
60.71
37.14
56.26
44.63
68.50
59.12
$4.40
96.69
92.93
94.30
97. 53
97.41
95.77
(3.17)
(0.72)
(1.65)
(1:00)
(3.05)
(1311)
(0:55)
(0006
(0.53)
(0.65)
(0. (9)
(0.2)
(0:2)
FedHKD DCLR
59.04
36.78
54.01
48.81
67.82
56.84
53.88
96.32
93.91
94.75
96.91
97.56
95.89
(0.46)
(1.37)
(1.25)
(1.25)
(1.35)
(1.29)
(0.41)
(0.3)
(0.76)
(0.62)
(0.72)
(0.25)
(0.24)
LG-Mix (Ours)
60.84
37.20
61.49
78.07
78.62
60.23
62.74
98.77
97, 99
98.93
99.13
98,95
98.75
(0.38)
(0.95)
(0.74)
(1.94)
(0.67)
(1.15)
(0.12)
(002)
(0.29)
(0U7)
(0.09)
(0.04)
(0.05)
5.2 PERFORMACNE COMPARISON RESULTS
We first present the classification performance on four datasets, which cover the heterogeneous fea- tures regarding covariate shift and concept drift. Table [ and report all results, including each client and the average performance. It can be observed that most PFL methods outperform the common global model learned by FedAvg. As FedBN is specifically designed for heterogeneous features, it presents larger improvements than other methods on most tasks. Interestingly, we find the FedAlt, which personalizes partial layers (i.e., the output layer), also clearly outperforms other PFL methods on most tasks. This may be owing to its alternating update strategy and the importance of the output layer in classification tasks. Compared with all methods, our approach shows signif- icant improvements on 18 over 20 clients on four datasets, with the largest average performance improvements of 5,01 % and least average performance improvements of 0.84%. This demonstrates the effectiveness of our strategy specifically designed by considering the data heterogeneity.
We further perform the comparison on the retinal fundus image segmentation. The fundus image varies with different machines, illumination conditions, field of view, etc. The results are shown in Table. Note that we do not include FedBABU, FedRep, and FedHKD because they are specifically designed for classification and suffer significant performance drop on segmentation. The data
7

---

Published as a conference paper at ICLR 2024
Table 3: Comparison with SOTA PFL methods the real-world retinal fundus image segmentation.
Retinal Fundus Image Segmentation
Client
A
B
C
D
E
F
Avg.
A
B
C
D
E
F
Avg.
Dice Coeflicient (Dice)
Hausdorff Distance (IID)
L
FedAvg (PMLR 2017)
83,95
83,00
81.15 87.98 67 60 91.07
82.46
7.71
4.77
8.35
4.19
68.14
2.38
15.92
(0.66) (0.32)
(0.12)
(0,02)
(0.9)
(5.04)
APFL (Acxiv)
74.03
72.76
70.03
81.30
66.27 90.56
75.83
27.91
52.47
43.07
18.00
76.69
5.73
37.31
(10.16)
(9.3g) (1043) (663) (3,33) (0.75) (6:55)
(182)
(0.11
(27.7))
(15.1)
(42.1)
(5.75)
(22.22)
L2SGD (NeurIPS 20200
84,46
83,73
82.20
88.40
68.90 91.12
83.14
7.55
4.81
7.95
4.18
57.76
2.48
14.12
(1.57) (0.19) (0,86) (0.28) (0.39)
(0:10)
(2.59)
(1:3)
(15.5)
(2.48)
(2.84)
FedAIt (ICML 2022)
85,01
85,19 84,06
88.98
64.48
91.05
83.13
6.20
5.07
5.25
3.59
75.60
2.58
16.38
(2,08)
(0.91) (1.93) (0.7) (2,46) (0.36) (0.57)
(0.39)
(0,02)
(0.5%)
(0.3)
(43025
(2.55)
(7.22)
PerFedAvg (NearIPS 2020)
85.87
84.55
84.74
88.75
67.91
91.10
83.82
7.32
4.54
7.79
3.82
73.58
2.47
16.59
(0-4)
(0:75) (1.53)
(0.18)
(1.77) (012)) (0:79)
(0.23)
(0.02)
(4:11)
(0.81)
(99.5))
(1.47)
(6.04)
FedBN (KCL 2021)
84.77
83,26
83,88
88.45
67.03
91.07
83.08
7.60
4.82
5.69
2.95
63,34
2.70
14,52
(0:29)
(0.61)
(0.5) (1.55) (0.09)
(0.4
(0.01)
(0.15)
(1831)
(2.7) (2.9%)
FedFomo (CCLR (021)
71.48
80.47
76.62
86. 19 55.10
89.87
76.62
12.29
4.71
12.08
5.10
154.01
2.72
31.82
(5.94)
(0.75)
(5.67)
(1.01)
(2.5)
(0.71)
(2,66)
(2.57)
(0.06)
(3,07)
(0.4)
(25.3))
(2.72)
(3,47)
LG-Mix (Ours)
89.25
86,76
85,86
89.79
83,95
90.86
87.75
4.43
3.63
4.53
3,57
6,38
2.34
4.15
(1.11)
(0U0))
(0.37)
(0.09)
(0.22)
from client E (fifth column of Retinal in Fig. T shows different appearances due to its different image settings (dual) from others (mono). This scenario further illustrates the necessity of model personalization. Our approach consistently outperforms all compared methods. Specifically, for client E with a unique imaging setting, our method shows very large improvements (15.05%) on Dice compared with the second best, while most PFL methods fail to present a high performance.
5.3 ANALYTICAL STUDIES
We further analyzed the key properties of our method, including (a) the personalization weight change during training, (b) why our personalization is helpful, (c) the client scalability of our method, (d) the distance between the final personalized model and global model, and (e) the ef- fects of considering history personalization weights.
Trend of the personalization weight. We compare our personalization weight with APFL, which finds the optimal mixing ratio based on differences between mixed and global models. We present the weight curves on the Digits5 dataset in Figure 2 (a). APFL tends to mix local and global models evenly (in the range of 0.5 - 0.6). In contrast, our method tends to use more local updates at the beginning and gradually decreases the personalization weight. It finally leads a stable value. This trend fits the intuition that the global update might be distracted by heterogeneous features and may not be very informative at the early training stage. Client D is always higher than 0.6, we speculate that client D has more data samples than others, which contribute most to global updates.
Feature value distribution. We further study the feature value distribution. This helps further vali- date the quality of the learned personalized model. If a model learns confident representations, then the related neurons should be highly activated Zhou et al. 2018). i.e., the values are higher than the activation criteria (0 for ReLU in analysis). The results five clients from the Digits5 dataset are shown in Figure 2 (b). Our analysis shows that our personalized model achieves significantly higher feature values than learning FedAvg global model on heterogeneous features. Moreover, our method also outperforms client standalone training on some clients, such as clients A, B, and E. This demonstrates the effectiveness of our method in learning accurate and effective representations by incorporating both local and global knowledge.
(4)
(b)
(d
EERFA
Taclleg
Prstralacionegg weggl
25
Aff. Oum
Diiee
Chonce
07
10
Communiatiionnoo
Rumber
Figure 2: Analytical studies on key features of our method, including (a) the personalization weight
changes; (b) the benefits of personalization via observing feature values; (c) client scalability.
8

---

Published as a conference paper at ICLR 2024
Digits Clowsfication (Convnlutional Neural feetwork)
Retinol Fundus Image Segmentation (U-Ner)
0.40
Parameter afffereice
0.02
c
Client
Client
0:00
ec00e 00000 so...
0.20
0.07
0.03
Client A Client
Clent Clent
Ciint E
0.05
Lojer
10 11
3
Loyoz 10
12
24
Figure 3: Distance between local and global model on the classification and segmentation model.
Client scalability study. We perform the client scalability analysis on our proposed method by comparing it with the two better-performing methods, FedAlt and FedBN, on the Digits5 dataset. We report the average test accuracy by increasing the number of training clients, which also results in different training feature distributions. The results are shown in Figure 2 (c), it can be observed that all PFL methods show better scalability than the baseline FedAvg. In particular, our method shows a stable increasing trend with a lower standard deviation among clients (shaded area), while performance improvements of FedBN are saturated by involving more clients, and FedAlt presents an unstable trend with larger standard deviation.
Distance between personalized and global model. We also analyze the distance between the per- sonalized model and the global model, in order to investigate how updates mixing reflect in model parameters. Specifically, we report the differences (i.e., We u in layer-wise) of each layer in the dassificationsseementttion model in Figure 3 Our analysis shows that the output layer varies sig- nificantly from the global model in classification, which validates the design of personalizing the classification head in FedBABU, FedRep, and FedAlt. Note that layers 2,4,6,8 are batch normal- ization layers, which also supports the idea of FedBN. However, our method does not specify any target layers but effectively personalizes these important layers identified by other methods. As for the segmentation model, all layers show variance, and the differences become larger in the de- coder and output layer. This explains the failure of methods like FedBABU and FedRep, which only personalized the output layer. Overall, our mixing of local and global updates can effectively personalize the proper model parameters across all layers.
Ablation study us- ing history personal- ization weights. We conduct ablation studies to evaluate the effective- ness of our strategy for stabilizing personalization weights. We compare our method with the second-best method on each
Table 4: Ablation study for effects of stabilizing personalization weights by considering the history weights.
Task
Digits5 Ottice-Cltechl0 DomainNet Camelyon17]
Retinal (Dice)
Second best
95.93 (n.0))
75.13
57.73 (0.9)
97.91 (0.06)
83 83.82(0.29)
Ours w/o history
97.10 (005)
78.25 (1.1))
60.332.20
98.70
87.73 c0 26)
Ours
97.10 (000)
79,31(109)
62.74.0.22)
98.75
87.75.0.17)
dataset and report the results in Table. E The results show that utilizing historical personalization
weights improves performance on all classification tasks and has lower standard deviation, which validates the effectiveness of our stabilization strategy.
CONCLUSION
In this work, we proposed a novel approach to address the challenge in PFL for heterogeneous features. Our approach mixes local and global updates by measuring the NTK- convergence during training. Specifically, we take the trace of NTKs using local/giobal updates as the hint to perform the mixing, and approximate the NTK calculation putation efficiency. Besides the empirical solutions and significant performance improvements, we also theoretically analyze the convergence rate of our method using NTK. Our approach has no strict restrictions on model architectures and can be applied to a wide range of PFL applications. For fu- ture work, it is promising to investigate the effectiveness of our method other data heterogeneity, such as the class distributional shift, and the larger model, such as the transformer.
feature matrix calculation for com-
Acknowledgement. This work was supported in part by National Natural Science Foundation of China (Project No. 62201485), in part by Hong Kong Research Grants Council Project No. T45- 401/22-N, in part by Science, Technology and Innovation Commission of Shenzhen Municipality Project No. SGDX202205301 1201008. in part by Canada NSERC Discovery Grant (RGPIN-202) 05316).

---

Published as a conference paper at ICLR 2024
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ruizhao Zhu, Ramon Matas, Matthew Mattina, Paul What- mough, and Venkatesh Saligrama. Debiasing model updates for improving personalized feder- ated training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 21-31. PMLR, 18 Jul 2021.
Ahmed Almazroa, Sami Alodhayb, Essameldin Osman, Eslam Ramadan, Mohammed Hum- madi, Mohammed Dlaim, Muhannad Alkatee, Kaamran Raahemifar, and Vasudevan Lakshmi- narayanan. Retinal fundus images for glaucoma analysis: the riga dataset. In Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications, volume 10579, pp. 55- 62. SPIE, 2018
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary, Fed- erated learning with personalization layers. arXiv preprint arXiv:1912.00818. 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019a.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. CoRR, abs/1901.08584, 2019b. URL
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
Jan van den Brand, Binghui Peng, Zhao Song, and Omri Weinstein. Training (overparametrized) neural networks in near-linear time. Innovations in Theoretical Computer Science (ITCS), 2021.
Huancheng Chen, Chaining Wang, and Haris Vikalo. The best of both worlds: Accurate global and personalized models through federated learning with data-free hyper-knowledge distillation. In The Eleventh International Conference on Learning Representations, 2023.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International Conference on Machine Learning. pp. 10089-2090 PMLR, 2021.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, 2020.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. International Conference on Learning Represen- tations, 2021. http:httts///ppeeeeeeeee iew net./ forum? (at.ruummm.ddNNh>BBBB.......
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterizze neural networks, 2019.
Alp Emre Durmus, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. Federated learning based on dynamic regularization. International Conference on Learning Representations, 2021.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3568. 2020.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kemel learning: empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861, 2020.
10

---

Published as a conference paper at ICLR 2024
Francisco Fumero. Silvia Alayón, José L Sanchez, Jose Sigut, and M Gonzalez-Herandee Rim- one: An open retinal image database for optic nerve evaluation. In 2011 24th international symposium computer-based medical systems (CBMS), pp. 1-6. IEEE, 2011.
Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In CVPR, 2022.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp- 2066-2073. IEEE, 2012.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models, 2021.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and opti- mal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33:2304-2314 2020.
David Holzmüller, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart. A framework and bench- mark for deep batch active learning for regression. Journal of Machine Learning Research, 24 (164):1-81. 2023.
Kevin Hsieh, Amar Phanishayee, et al. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning. PMLR, 2020.
Baihe Huang, Xiaoxiao Li, Zhao Song, and Xin Yang. Fl-ntk: A neural tangent kernel- based frame- work for federated learning convergence analysis, 2021.
Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hier- archy. In International conference machine learning, pp. 4542 4551. PMLR, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernl:: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, 31, 2018.
Vinay Jayaram and Alexandre Barachant. Moabb: trustworthy algorithm benchmarking for beis
Journal of neural engineering. 15(6):06601,, 2018.
Wonyong Jeong and Sung Ju Hwang. Factorized-fl: Personalized federated learning with parameter factorization similarity matching. In Advances in Neural Information Processing Systems, 2022.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-21) 1200, 2021.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, et al. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient for robustness spurious correlations, 2023.
Nikita Yurevich Kotelevskii, Maxime Vono, Alain Durmus, and Eric Moulines. Fedpop: A bayesian approach for personalised federated learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL net forum? https: htttps://openreve www..eeeeteooteeeeteteeeme wtw.rrmmmmidkKE.tiieiemee.
Jason Lee, Ruoqi Shen, Zhao Song, Mengdi Wang, et al. Generalized leverage score sampling for neural networks. Advances in Neural Information Processing Systems, 33:10775-10787, 2020.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In CVPR, pp- 10713-10722, 2021a.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEFE signal processing magazine, 37(3):50 60, 2020a.
11

---

Published as a conference paper at ICLR 2024
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and sys- tems, 2:429 450, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization, 2021b.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. FedBN: Federated learn- ing non-IID features via local batch normalization. In International Conference on Learning Representations, 2021c.
Disha Makhija, Xing Han, Nhat Ho, and Joydeep Ghosh. Architecture agnostic federated learn- ing for neural networks. International Conference on Machine Learning, pp. 14860-14870. PMLR, 2022.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 2016 fourth international conference 3D vision (3DV), pp. 565-571. IEEE, 2016.
Mohamad Amin Mohamadi, Wonho Bac, and Danica Sutherland. A fast, well-founded approxi- mation to the empirical neural tangent kernel. In International Conference on Machine Learning, pp. 25061-25081. PMLR, 2023.
Jachoon Oh, SongMook Kim, and Se- Young Yun. FedBABU: Toward enhanced representation for federated image classification. In International Conference on Learning Representations, 2022.
José Ignacio Orlando, Huazhu Fu, João Barbosa Breda, et al. Refuge challenge: A unified frame- work for evaluating automated methods for glaucoma assessment from fundus photographs. Me- dIA, 59:159911110 2020.
Kaan Ozkara, Antonious M. Girgis, Deepesh Data, and Suhas Diggavi. A statistical framework for personalized federated learning and estimation: Theory, algorithms, and privacy. In The Eleventh International Conference on Learning Representations, 2023. URL/https:
Beet/Eorrumid==7UDiie W4o
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1406-1415, 2019.
Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. arXiv preprint 1912: 2019.
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. International Conference on Machine Learming, pp. 17716-1778 PMLR. 2022.
Sashank Reddi, Zachary Charles, Manzil Zaheer, et al. Adaptive federated optimization. In International Conference Learning Representations, 2021.
Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts. arXiv preprint arXivv2006.08907 2020.
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyri- don Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):177, 2020
Felix Sattler, Klaus-Robers Müller, and Wojeiech Samek. Clustered federated learning: Model- agmostic distributed multitask optimization under privacy constraints. IEEE transactions on neu- ral networks and learning systems, 32())3370-3322, 2020.
Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, and Hung-Hsu Chou. Neural (tan- gent kernel) collapse. arXiv preprint 164.22 2023.
12

---

Published as a conference paper at ICLR 2024
Mehdi Setayesh, Xiaoxiao Li, and Vincent W.S. Wong. Perfedmask: Personalized federated learning with optimized masking vectors. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Jayanthi Sivaswamy, S Krishnadas, Arunava Chakravarty, G Joshi, A Syed Tabish, al. A compre- hensive retinal image dataset for the assessment of glaucoma from the optic nerve head analysis. JSM Biomedical Imaging Data Papers, 2(1):1004, 2015.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task
learning, 2018.
Zhao Song and Xin Yang. Quadratic suffices for over-parametrrization via matrix chernoff bound, 2020.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Quoc Tran Dinh, Nhan H Pham, Dzung Phan, and Lam Nguyen. Feddr-randomized douglas- rachford splitting algorithms for nonconvex federated composite optimization. Advances in Neu- ral Information Processing Systems, 34:30326-30338 2021.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33, 2020.
Tianchun Wang, Wei Cheng, Dongsheng Luo, Wenchao Yu, Jingchao Ni, Liang Tong, Haifeng Chen, and Xiang Zhang. Personalized federated learning via heterogeneous modular networks. In 2022 IEEE International Conference on Data Mining (ICDM). pp. 1197-1202. IEEE, 2022.
Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 5(1):1-19, 2021.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized fed- erated learning with first order model optimization. In International Conference Learning Representations, 2021.
Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning. pp. 26293-2631 10. PMLR, 2022.
Yi Zhang, Orestis Plevrakis, Simon S Du, Xingguo Li, Zhao Song, and Sanjeev Arora. Over- parameterized adversarial training: An analysis overcoming the curse of dimensionality, Ad- vances in Neural Information Processing Systems, 33:679-688 2020.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9): 2131-2145, 2018
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European Conference Computer Vision, pp. 561-578. Springer, 2020.
13

---

Published as a conference paper at ICLR 2024
APPENDIX A NOTATION
Table 5: Major notations occurred in the paper.
Notations
Dimension
Description
k, K
N
number and total number of local update steps client index, total number clients
G, N
Teey
N
number of samples of client and overall number of hidden neurons client-level learning rate, global learning rate local model client c, global model sample pair of index i model prediction at time t with local model, global model collection of samples of client
m
N
mh. Ml
R
Wey
RP
(Iri, Vs)
R
(e)t), v(t) S.
R
of size The
APPENDIX B GLOBAL CONVERGENCE
B.
FORMULATION
B.1.1 THE NEURAL NETWORK
For the following theoretical analysis, follow Du al. [209 & Yang (2020 Huang et al. 2021 to consider an one-hidden-layer neural network with ReLU activation:
m
f(u, x)
where m is the total number of neurons in the hidden layer, a,'s are Rademacher random vari- ables (take values {+1} with equal probability) and é is the activation function. Each client aim to optimize its MSE loss:
L.ne Lm (u, E (f(,,x;) -1,)
iES.
where S. represents the collection of data of client c. The global loss is taken as the average of the loss of each client:
L(u,z) =
1
Σ L,(u,
N
cE[N]
B.1.2 OUR ALGORITIM
We first formulate our proposed algorithm mathematically establish consistent notations and fa- cilitate analysis.
In FL, each client alternate between performing local updates on its local data and communicating with the central server for global aggregation. Each client takes K local updates between each communication round. The local update is performed using vanilla gradient descent with local learning rate n7. and (t, k) represents the weight parameters of client e at global round and local step k:
(w..
(.,t, k 1) (t, k) T7 Ow. (1,
After each communication, the global aggregation procedure is conducted by taking the average of local updates of all N clients, and learning rate 1/ added for the global update:
Au(t)
7lg N
Σ (w.(t)
CE(N]
where Aw.(!) (t,(, K) (1, Ee 1n (t.k)
(L.(., is the cumulative local updates of
client at global round For the local step, a combination of local update and global update is taken, with X(t) E [0, 1] being the combination factor:
we(t 1,0) (c(1,)) (1 A(1))a((1) + A(1)Aw.(1)
14

---

Published as a conference paper at ICLR 2024
B.1.3 GRADIENT UPDATES
With the above setting, we can explicitly write out the gradient updates:
aL (we,r(t,R)
Awer (1) Σ 1/
Σ
Σ (sW,,r
vis)
Vi 1 wT, 2,0
ower (t,
Vm
ke[k]
kE|K]iES.
(1,
Tel
E E
ΣV
ye 0, u,7,0
N
m
eE(N iES.
B.2 CONVERGENCE ANALYSIS
We analyze the convergence behavior of all clients collectively. That is, consider the dynamics of lly - y(t)l12 = E. li (e(2) where
v(t) (f(wi (t), x1), .... f(w (t), (n, f (wz(t), ..... (e(1) (0, 0, [W](((()) x ,,,,,, (we(()) (n.),, ,),
Iny)
F.68.
the stacked vector of predictions and y, Me are the corresponding ground truth.
Note first the following recurrence relation (t):
ll ytt + 1) = lly y(t)] - lu(t + 1) (t]]ll?
2(t)l12 2(y -v(t)) (2(t 1) v(t) + 1ly(t 1) v(t)ll² |y u()112 - > (vee - (t)) (vc(t + - (e())) +I 1) r())
the cross term
We will express lyy - y(t+ + 1) in terms of yy y(2) with a shrinking factor, by bounding each of these terms, and hence prove the convergence of the algorithm.
B.2.1 THE CROSS TERM
We first investigate the cross term. Note that the difficulty in the analysis mainly comes from the non-linear activation pattern. However, this is overcame by a key observation in classical NTK theoryDu al. (2019 Huang et (202T) that the activation patterns stay the same for most of the neurons.
We follow their approaches to define
Q = (r E [m] : Vm E RRs.ttl|ww w,)(l] S R, m.. L T2,20
which represent the set of neurons whose activation pattern does not change during training for sample z4, and let Q, denote its complement. Then for each sample i E Sc,
v. 1 (t(
Σ ar [o( (t 1)xx)) o(w) (t)z;)
Vm rE[m]
1
a, (u, (0)
Vm
rEQ.
Σ
(1) 10 +-3,
m
rEQ:
V2,4
15

---

Published as a conference paper at ICLR 2024
where
(1 )e(t))
E
E
(u(t, w/)x ",, 2,0,,
01,
Nm
rE| JES
Xa(t)m
Σ
Σ (Moll, Wos)a wi (00
22
rE[ JES
1
Σ or p(w, (t+ 1)x;) - 6(w (t)zi)
33
reQ:
We can already notice the almost symmetric kernel factor in the terms above. We give the formal definitions here.
Definition (Global Gram matrix). For l E [T], eK], E [N E S. and j E Se, we define the global gram matrix as:
H(t, k)., ==
Σ
d 2z
wT wT, 20, ,, (1,),,200
E R
rE/m]
H(t, k) = m
Σ rQQ
x
E R
Note that this definition is similar to, but not exactly the same as, the definition in FL-NTTHHunng et al. (202). This is because they considered the vanilla FedAvg with no personalization of model parameters.
Definition 2 (Local Gram matrix). Fort [T], E [K],e e [N] and i,j Se,, we define the local gram matrix as:
He(t,k)s.j
1
Σ
2j Øz wT (1:20000000
RR. X
m
He k)
Σ
o σ. 1 wT (0):::
E B
r
However, in order to maintain consistent dimensions and correspond to our definition of ye, we can extend the dimension of H. to by adding zeros to the undefined entries, i.e.,
H. :
R
From now the symbol H. will refer to this matrix. Note that (Ho)). = HigLisjs.. We will show that the convergence can be governed by the spectral property these Gram matrices. Substitute them into the cross term, we get:
Σ (Ye - (a(1))T((a + ve(()))
Σ
(Je,s - ve(()))
Σ
(u(t,k), - (t, - H(t, k)),,)
N
teS.
kE(
Aem
(Ve, (t)aa)
Σ
(ve(t,)), (H,)(HH(t,H))). Hc(t.k)
iES.
(ye, - ye (t)s)
JES.
16

---

Published as a conference paper at ICLR 2024
Let
Xc(t))mm0g
C -
Σ
Cuc,s-eell (Vc.s - ve(s)s)
Elv(t, k)5 - ()),H( - (tt,
N
iES
c = dc(())
Σ
(Va, vc(()))
Σ
(uelt, k)3 ve. (H,(t, - He(t, k))3)
iESo
C2 Σ c2
(c)
C2 = (Be, - ye vs(
C iES
Then by substituting them back into the recursive relation (t), we get:
1 v(t + 1)]2 - 2(C C2) + 11u(t+ + -((1)1
We will bound each of these terms and hence prove the result.
B.3 CONVERGENCE ANALYSIS MAIN THEOREM
We first restate the main convergence theorems.
Theorem For uniform Ae(t) X(t). Ve, for m 2n==nn=* log(n/6)), randomly initialized
parameters (i.e. w(0) ~ ()). and m = O(A/kKn²) 7g = O().. then with probability at least 1 over the random initialization, we have for Vt.:
lly ut + 1)11 $ 11 y(t) (1 a(t))s M by y(t)l
((t) min llye (t)l
m. where Ç = 2N
Theorem 2. For non-uniform Xe(t), let Amin (1) = min, Aolt) and Amar(t) = max. Aolt),
For m Q0==nn (og(n/))), randomly initialized parameters (i.e. w(0) NO0, I)), and
= O(1/kKn²), Tlg O(1), then with probability at least 1 - 8 over the random initializa- tion, we have for Vt:
(1)) llv 11--1(++1111 < 1w- (1-Amaa (1))s K 11))(111
Σ
Amin (1) (1)s (HHe) ( 11vo-1 Wo (1)l22
K where := 2N
We will give the proof of theorem M in the subsequent sections, and we note that theorem is natural extension of theorem [ so the proof also naturally extends.
B.4 USEFUL LEMMAS
Before giving the proof of the theorem, we state two useful lemmas.
The first lemma gives bounds on the norm of the local and global updates.
Lemma 1. With || = 1, we have
(1)(1) 210 K (1 + 2mmkK) /n
I v()]l2
Nmm
2m (( (1+ + 2mm K) nhc
(
|1 vc(())22
vm
17

---

Published as a conference paper at ICLR 2024
Proof. The first inequality follows from FL-NTK. For the second inequality, consider:
A (
( = m11
r (t)l22
ar
Σ
Σ
[v(t, k), - Aw
V
kE[K]
F.S.
m
Σ
Σ
1y
v((
v'm
kc[K]
iES.
Σ
11ye
V
kE(K]
TK + 2mm K) The
11 -
1 (t)ll
m
The second lemma bounds the sum of client prediction error by that of the global error. Lemma 2.
Σ llve ya(1)l S VN s(O1l
Proof. By Jensen's inequality, and since the square root function is concave,
1 N
Ilve vo(1)112
11 yyc(4)
B.5 PROOF OF THEOREM
We provide here a detailed proof of theorem [ and note that the same proof naturally extends to prove theorem 2
We will use À to represent X(t) for ease of notation.
Firstly, here global and local updates respectively.
results that directly follow from FL-NTK. They provide bounds on the effect of
Proposition 2. With probability at least 1 - nexp( mexp(mmii mmR) over random initialization, we have
11gg(1 A) N lly (-K)) min + 400/nKK(1 + 2nj + 27j (H) K² vn))
C1 S
(H)
8n A) K(1 + 2mmK) RRly yt))
N
Proposition 3. With probability at least 1 n exp( -mR) over random initialization, we have
che0
AMR
1lye (o)()) K& (Ha) + 40/ + 23p Kv (He) K² Vn)
8Am K(1 + 2nm 2c(2)
N
For the following two propositions, we assume that all clients possess the same number of samples, ie., ne n/N, Vc. Additionally, let n denote max(1,ng)
The following proposition aims to bound the effect of updates on neurons whose activation pattern changed during the algorithm.
Proposition 4. With probability at least 1 n exp(-mRR) over random initialization, we have
Smiig
C <
(1 + 2mpnK)RRl y(t)l12
N
18

---

Published as a conference paper at ICLR 2024
Proof. Consider
|
2 v
1-11 m
Σ
Σ
(0%
2
+
Σ iE(m)
Σ
(1,(())
2
m
(E(n)
rEQ.
r
A
B
and
8(11
M N
(1 + (nnnK)nRRly -v(0)) 2
A
As for B,
B= m
ΣV Σ
11 1, (A,,(1)T xr)))))))
iES.
TE mm)
X
2 (1 2mm. K) p) no
11 11 -yc(t))2 -nc4mRR)
m
m
8Am K
(1 + 2nnn u(()))
v(t)))
N
where for the second inequality we used the assumption made above. Then
C3 : - Σ (yi y (1))
TE(m)
S 1ly (ttlllal
S K
(1 + 2nnnK v(t]l
N
Now we have bounded the cross term. For the last term, we have the following inequality: Proposition 5. We have
4n 7 K2 2mmnK22
11 + +1 11)((11 <
11 - y())
N2
Proof.
l + 1111-((())) u(1)122 < -ーン Σ Σ x.
2
A Σ
Σ (t) ))
+
m
m
iEin)
rE[m)
rE(m)
(1 Au
(2 - 2mmnk
|18 - v(())) nnm?
N Nmm
Ank 2K + 2mm /TC
Il vc(t)1)) mnmm?
m
Vm
4m 4m 1 K2 + 2m
11 - 21)1112
N2
where we have used the assumption that n. n/N.
19

---

Published as a conference paper at ICLR 2024
Now by substituting the above results to the recursion equation, we get:
ly v(t 1)]1 11 y(t)11
2n (1 X) X) 11y (())((())((( s 400/nKK(11 + 21. Vn)
N
2mp (M x² vn) + 16 (1 A) K(1 2nmm vt))
N
2
1i yc(t))l((-Ks Ks) min + 400TRKK(1 - 2m Kn N
(M.)
(Ho) K2 vn)) + 16Am
+ 2mp mas
K(1 + 2mmK) Σ vc())
N
16 K N 4xp 1 2² + 2)²
(1 + 2 K y Rlyy x(1)1
Ily vt)]
N2
(Ra) 1000kn K minel 1000 K }} where == Smaz Smin and T] S
Then by the choice of Tp min
min
(H)
((8))
min
1000xn K minal 1000M
}} and R mmmee//11000 (1000m), we have
y(t 5 fly v(t)11
(1 2) min
(H) K
Ami (He)
K
11y 1000((012- Σ
min
1 (1)11
N
N
myg KnR N
+ 40
11 1 -(((t) x2
n n K2
11y y()1112
N2
(L
A(t)) ( K
S fly v(t)|
11 - 2())1
2N
X(t)mssmn
(Ho)
K
11y. vo(())12
2N
by substituting in the condition on and R. Quod erat demonstrandum.
APPENDIX C GENERALIZATION
In this section, we prove the generalization bounds. That is, we aim to find a bound on
CD(f) := Bn v)]
where if refer to the prediction function we consider. Note that, in practice, this is approximated by the empirical loss Ls(f) = 1. Eie(n) ((())))))). We also consider a more general initialization
scheme ~ ~N(0
C.1 SETUP
We follow Arora et al. 2019b): Huang al. (2021) to consider non-degenerate data distribution.
Definition 3 (Non-degenerate Data Distribution). A distribution 'D over Tbb x TR is (A, nn-nn degenerate, with probabiliyy at least 1 1-8. n iid samples {1) v: from D, s (H-) 2
> O.
20

---

Published as a conference paper at ICLR 2024
We also state here the definition of the dynamic matrices which can be used to describe the evolution of the neural network: Definition (Global Trajectory Matrix).
01 oz (r,8)m120
a1 I (t.k),.0
k(,
E Rmdxx
vm
(m wT (1.))120
am Fennlwy wt (1.
Definition 5 (Local Trajectory Matrix).
01 we (1.))2100
ai an 1 Tw (1.k) (4.)) 20
Je(t,k) k)
Vm
(1m31 w m (t.k)m:
Om 1 wT in (t,k)an
for 's sample of client e, and where appropriate, we fill in the undefined entries with form matrix of dimension md n.
Note that H = J and He = JeJ Jc. We also give some useful notations following the above definitions.
Notation
J(t,k) = (Jor (t, Jos (1, (),
.. .. Sew (t, k)) E gddxn
Notation 2.
H1
H
E R m=x
H
We also use a notation vec(A) to express the vectorization of a matrix in column-firs order. Then the gradient update rule can be expressed as:
vec(Wo(t, k 1)) vec( rec(Wo(t,k))) (w) ye)
vec(U(L++ 1)) vec(U(())- 7ill Σ ((L,)((((L))) -
vec(W.(t- 1)) vec(Wo Am Σ (e(t, y00
(1 - A) 7M
Σ ((, N
C.2 SOME USEFUL RESULTS
We first quote a result from Huang et al. (2021) which will be used later.
Lemma 3. For (1,,k) k) as defined above, with probability at least 1 n 6)/10)), we have
[] ((, 0) 2n(oo 5)
The following lemma give an approximation on the dynamics of the global model.
Lemma 4. For A(A) (1 A) K H + Am K H and ((A) (1 2) N K
Am, K, we have
v(t) y -(I A())) +e(t)
where
[ee(t)l2 s (0(1- (1-3(A)smin)) (
Smin vmm poly(log(m/6)
t/(2)n2/2
21

---

Published as a conference paper at ICLR 2024
Proof. Recall that from Appendix B, we have (y(t) y) - [u(t - 1) y) 01 U2 23 and that
(1
A)mons K
(1y(t) U3)
11s
N
jE/m
(1
7)m
(Ug(t, k) - y/ (t((t)
N
JEIn]
X)mo
Σ
(us(t, k) - (H k) H()
N
jEin],k
(1
X)mm
Σ
- 35) k))(t,k)
k))s)
N
jE(n],k
and similar for v2,; except that i, So for some client
Let
(.()) (()) + (t, (t)
+ A)mm, K
(1)(1) 31 H33
jE
+ Am/
(t)(t) y1 ) (H )as
jESc
Note that by Appendix B, ||ug(t]]| 16
K (1 + 2m y(t)ll. [lue(t) bc(t, =
2m K [ya - Yells lllllyyll yt, k)ll2 I 2(1 + 22)K))l Ily - y(t) 12, H(w, w) - Hww, W2 W2)llF 4n
and | (1, -)-I|r 4n/ etc. By taking the maximum order among the terms, have that
IE(t)]l2 I 0 B())n² smar / /log(mm) (n//()))
lly - v(t)l2)
odvm
where B(A):- = (1-AmmT N
Am..
Then
v(t) y (1 - A(A))(u(t 1) v) + E(tt 1)
= (I - A(A))' y) + Σ (I- (1-A(N)E(--17
rE(t-1)
= -(I A(A))'++ee)
where
e(t) (1 - (())) + (()
rE[t-1]
and since
[lv(0)]] < no² 2 log (2mmmmm))))) log (4n/6))
we have
[ee(t)l2 2
- (1-B(m)smmn)) ( /0 (og(8n/5) + 3())n712 log(m/ ((mmmm)))) hogn//))ll(31)) log((1/))) (n//1) (0)088(1100))
o
Smin m
(Vio+ + Smin /m poly(log(m/6))
t3(
s 0
(1 3()))mmn)
22

---

Published as a conference paper at ICLR 2024
C.3 AVERAGE GENERALIZATION
When examing a new OOD sample, we would use the average of all current parameters for predic- tion. Therefore, first examine the generalization performance of the average of all paramters:
Σ Wc(t) W(t) = N
1
By 77. we have:
An (tt+)()) = vec(W(()))
Σ
Je(t, Kb) Mc) 10)--(1-1) -1- 1 N
Σ ((rt, k)
N
A,C.
&
Lemma 5. For A(A) -1 N H Am K H. and 7() (1- A) N
x mn
N we have
[W((L) (())l S (0" (w"A(A) A(A) - A())-1y)112
n
o pol)))og(m//))) + 01/2m114
poly(log(m/0)))
8min
Proof.
vec(W(()) - vec(W(())
-Σ (1 X) nim
Σ ((, V)
An
ΣΣ Jett, (t),(tt) - ve)
N
N
1E[T-1]
-2(1)
Σ
J(L,))o((,))) - v)
K
LE[T 1
T())
7(A) ((,,))(o(t,)) v(t) + c(k))
Σ
-AA))---- E
K
K
NE l
1E[T-11,
Σ (TT - A(1))'y
-E[
2(1) (J(1,)) - (10,0))((-A(AA -
Σ
K
K
EETT 1).R
7())
Σ
((,))((((1))) - vV() +c())
K
t.k
=Bi + B2 + B3
where
B1 Σ (1) -
1E[T-1]
B2 Σ 7(1)
(J(t,)) - -1000)((-A(AA)'
11,
B3 Σ 7(1) K ((t v(t ee())
t.k
Then by substituing in the following claims, we have (W(T) ((
5(8 0 -(A)-11 1/20)/++/((()))) +0( na
-poly(log(m/66) + 1/2 poly(log(m//6))
n
8min
23

---

Published as a conference paper at ICLR 2024
Claim 1.
With probability at least
8 over random initialization, as t - have
B 11 S A(A) O0 10 "i2oooo Vloooo///))
min Vm.
Claim 2. With probability least 6 over random initialization, we have
3/ polyyloogm((m//
|2 S
m m2/401/28 3/2 m
Claim 3.
|B I
110
72
poly(log(m/66)
Smin
min
The above three claims are the slightly modified version of Claim C,8-10 in Huang et al. (2021), So the proofs from there naturally extend to the proofs of these three claims.
Theorem 3. For T sufficiently large,
(Xpoly((og n. (og((16)//n))
Qn 16 poly(log n, log(1 1og(1/0),A-1))
the loss function 1 being 1-Lipschitz in its first argument, then with probability least 1 over the random initialization, the population loss LD(f) of the global model W = N Ee W. is upper bounded by
LD(f) S 20 A A(A)-1y/n O( ((VVoo((// smin /) /2n)
where A(A) -) () H 0 AmK
Proof. This is extension of the result in Huang et al. (2021), by substituting lemma into the proof of Theorem C.11 in Huang et al. (2021).
C.4 CLIENT LEVEL
For further inspection the algorithm, consider client level generalization. That is, sider the generalization bound on a client's parameter W.. Note that:
vre((.(t++)) = vec( (t) Am Σ Ja(t, (t), -
7
Σ (tt,))
N
k
We first define matrix that will be used later. Definition (Cross Gram matrix).
- of Ing 100
1
H (1,K)ES
m
where E S. and spans all clients.
This matrix describes the effect of the global update on the client's local model.
Theorem 4. For sufficiently large, o
OO)oly((og n, (n poly(log n, log(1/0), >1)) the loss function I being 1-Lipschitz in its first argument, then with probability at least 1 over the random initialization, the population loss Lp (f( of the client model W. is upper bounded by
m
Lp(f) (G.N))A + 0 Vlg(n///////// smin 5) 3)22m)
where G.( = (1- 2)2 H
K2
X
K + X(1 XnnY n, K2
This theorem is a natural extension of theorem B and the proof also naturally extends. Further discussion is left for future work.
24

---

Published as a conference paper at ICLR 2024
APPENDIX D EXPERIMENTS
In this section, we show more experimental results and implementation details. Sec. D.1 gives more details for implementation, including dataset details, model architectures, training, and testing details. Sec. Dec. more experiment results, including the performance regarding various batch sizes, the complete evaluation metrics on the binary classification on the Camelyon17 dataset.
D.1 EXPERIMENTAL DETAILS
We here introduce the complete details of datasets, data splitting, and implementation details.
Digits5. Digits-5 zhou2020leaming.
with digits images showing drastic differences in font style, color, and background. We take each data source/style as client. We train a 6-layer convo- lutional neural network for the digit classification, specifically, the model has 3 convolutional layers and 3 fully-connected layers, we further add batch normalization layers after the first five layers to fit the requirements of FedBN, We use the SGD optimizer with a learning rate of 0.01 and batch size of 128. The loss function is the Cross Entropy loss. The total number of training rounds is 100 with a local update epoch of 1. All input images are resized to 28 x 28.
OfficeCCattcc10.. Office-Caltech10 gong201 2geodesicc contains images acquired in different cam- eras or environments, with four different data sources in total. We take ResNet-18 18 as the backbone and use the SGD optimizer with learning rate of 0.01 and batch size of 32. The loss funetion is the Cross Entropy loss. The total number of training rounds is 200 with a local update epoch of 1. The input images are normalized using the mean and std of Imagenet in PyTorch, which is specifi- cally mean = [0.485, 0.456, 0,406] and std [0.229, 0.224, 0.225]. All input images are resized to 256 256
DomainNet peng201 9moment has images with different image styles (clipart, info- graph, painting, quickdraw, real, and sketch). Following FedBN, we choose the top-10 class based on data amount from DomainNet containing images over 345 categories for simplicity. The training settings are the same as the Office-Caltech dataset, and we change the training round from 200 to 100, since the model converges faster on the DomainNet dataset.
DomainNet.
Camelyon 7 bandi201 Sdetection shows histology images with different stains from 5 hospitals. All histopathology images are stained with the H.E. staining and show various appear- ances. We use the DenseNet121 as the backbone, SGD optimizer with a learning rate of 0.01, and batch size of 32. We train the model for 40 rounds in total, and the local update epoch is 1. The image input size is 96 x 96. The loss function is the Cross Entropy loss. Note that this dataset is a very large dataset which contains over 450,000 histology images.
Camelyon17.
Retinal. Retinal fundus dataset contains retinal fundus images acquired from 6 different institu- tions (Fumero et al.) 2011 Siiaawamyy et al., 2015 Almazroa al. 2018 Orlando et al.) 2020 We use the U-Net for segmentation, the optimizer is Adam with a learning rate of 1c and = (0.9, 0.99). We train the model for 100 communication rounds in total with local update epoch of 1. The batch size is 8. We use the dice loss and report both the Dice score and HD distance. All images are resized 256 256.
For all datasets, we take each data source as one client and split the data of each client into train, validation, and testing sets with a ratio of 0.6, 0.2, and 0.2. We choose the best model based on the validation data and report the test performance accordingly. Code will be released after acceptance.
D.2 MORE EXPERIMENTS
Here we present more experiment results, which mainly include two parts. The first part is the study on the effects of batch size on method's performance, and the second part is the complete evaluation results on the Camelyon17 dataset.
Effects of batch sizes. As our implementation accumulates the feature matrix iteratively during local steps (Line 7 in the algorithm box), the batch size may slightly change the values of the feature matrix. In this case, we further explore the performance changes by using different batch sizes (4,8,16,32,64, 128) on the Digit5 dataset. The results are shown in Table. From the results it can be observed that changing batch size has very mild effects on the final performance, the overall accuracy
25

---

Published as a conference paper at ICLR 2024
Table 6: Performance using different batch sizes the Digits dataset.
Batchsize
MNIST
SVHN
USPS
Synth
MNISTM
Average
99.40
93.53
98.92
99.70
97.19
97.75
16
99.49
93.85
98.87
99.68
97.41
97.86
32
99.40
93.37
99.03
99.62
96.86
97.66
64
99.37
92.87
98.87
99.50
96.14
97.35
128
99.25
92.17
98.71
99.42
95.71
97.05
changes are less than 1%. This further supports our implementation of iteratively accumulating the feature matrix during local SGD, which takes less computational cost than re-caleulate samples again after local epoch.
Complete evaluation on Camelyon17. As the classification task on the Camelyon17 dataset is binary classification, so further report the full evaluation metrics, including the Accuracy, AUC, sensitivity, specificity, and the F1-score. From the Table. 7 it can be observed that our proposed method consistently outperforms all compared methods regarding all metrics.
Table 7: Complete evaluation metrics the Camelyon 17 dataset.
Accuracy
AUC
Sensitivity
Specificity
F1-score
FedAvg
95.30
98.90
94.91
95.69
95.30
APFL.
96.42
99.35
94.42
98.42
96.42
L2SGD
95.93
99.22
95.12
96.73
95.92
FedAlt
97.91
99.57
96.55
97.90
97.22
PerFedAvg
96.08
99.24
95.37
96.79
96.08
FedBN
95.77
99.15
95.02
96.53
95.77
FedFOMO
95.62
99.15
95.35
95.90
95.62
FedRep
96.78
99.39
96.78
96.79
96.78
FedBABU
95.77
99.15
95.03
96.52
95.77
FedHKD
95.89
98.43
93.89
94.67
94.28
LG-Mix (Ours)
98.75
99.89
98.63
98.87
98.75
Loss and accuracy curves. have further analyzed the convergence speed our method by comparing the training loss, validation loss and validation accuracy the Digits5 dataset. We select the top 3 ranked methods on the Digits5 dataset, and the results are shown in Fig. 4 From the figure, can be observed that other methods show quicker convergence speed and higher validation performance than the baseline method FedAvg, and our method further promotes the convergence speed, especially the training loss. These curves further demonstrate the efficacy the convergence rate our proposed method.
JP1)
6.68
681-
186
Commurizzion rounds
Communicat rounds
Communsation rourds
Figure 4: Training loss, validation loss, and validation accuracy. The comparison includes the FedAvg, the top ranked SOTA methods, and our method the Digits5 dataset.
Training time costs. We further investigate the training time costs and report the wall-clock time by comparing method and others all tasks. The GPU used for training is GeForce RTX
26

---

Published as a conference paper at ICLR 2024
Table 8: Wall-clock time (hours) for different methods on different tasks.
Method
Digits5
OfficeCaltech10
DomainNet
Camelyon17
Retinal
FedAvg
3.70
1.28
17.37
55.01
25.95
APFL
4.38
1.70
19.59
79.12
26.36
L2SGD
3.83
1.40
18.42
57.17
26.79
FedAlt
4.09
1.47
28.77
76.80
34.16
PerFedAvg
3.79
1,45
18.66
56.97
26.44
FedBN
3.73
1.26
17.43
55.03
26.56
FedFOMO
6.02
2.80
19.04
76.80
36.90
FedRep
4.32
1,41
19.18
55.77
FedBABU
3.96
1.31
15.92
62.12
FedHKD
22.43
2.64
37.33
68.71
LG-Mix (Ours)
5.86
1.32
18.92
57.98
26.55
Table 9: GPU memory cost (MB) comparison on different tasks.
Method
Digits5
OfficeCaltech10
DomainNet
Camelyon17
Retinal
FedAvg
270.24
1042.32
1042.32
1096.89
1814.29
APFL
600.77
2993.48
2993.48
2955.24
4504.02
L2SGD
456.18
2057.62
1157.62
2072.09
2329.91
FedAlt
458.27
1042.33
1042.33
1096.89
2297.57
PerFedAvg
270.24
1042.32
1042.32
1096.89
1814.29
FedBN
458.27
1042.32
1042.32
1097.99
2295.21
FedFOMO
458.28
1948.98
1948.98
1985.06
2837.97
FedRep
458.27
1042.30
1042.30
1096.89
FedBABU
458.27
1042.30
1042.30
1096.89
FedHKD
864.28
3589.56
3589.56
3021.02
LG-Mix (Ours)
472.37
2101.01
2101.01
1905.38
3730.69
2080 Ti. The results are shown in Table g We report the training ours. From the table, we can see FedHKD takes significant time cost than other methods, which is because it requires performing an extra validation process using different client models, order generate the hyper-knowledge For method APFL., it needs to take an extra forward pass and also optimize the mixing ratio factor in their method by calculating the parameter differences. Please note that we do not report the time cost of FedRep, FedBABU, and FedHKD on the retinal dataset, because they are specifically designed for the classification task. Compared with all these methods, our proposed method shows reasonable computational time cost. The cost is slightly higher than some baseline methods, but it a trade-off between the computational cost and performance.
GPU memory cost. We have investigated the GPU memory cost by comparing method and others. Compared with FedAvg, our method additionally stores a copy of the global model during local training, and it calculates the trace of latent feature matrices. We have tracked the peak GPU memory cost of each method and list the costs on different tasks in Table 9 It can be observed that APFL and FedHKD show significantly higher GPU memory costs than others, which is because they require more memory to store the local copy of the global model. Specifically, APFL stores the local model. the local copy of the global model, and local personalized model. FedHKD requires storing local models from other clients generate the hyper knowledge for distillation. Our method lies in a reasonable range of GPU memory cost while presenting higher performance. Please note that we do not report FedRep, FedBABU, and FedHKD on the retinal dataset because they are specifically designed for the classification task.
Extended experiments on the BCI dataset. We extend our evaluation from the image domain to the Crain-Commuter Interface (BCI) data, which consists of classifying the mental imagery EEG use four datasets from the MOABB benchmark (Jayaram & Barachant, 2018). We choose four datasets (AlexMI, BNC12014 001, BNCI2015 004, Zhou2016) which con- tain common classes of right hand and feet. We estimated the covariance matrix representations
datasets. Specifically,
27

---

Published as a conference paper at ICLR 2024
Table 10: Comparison the BCI dataset.
Method
Accuracy
AUC
Sensitivity
Specificity
Fl-score
FedAvg
67.38
69.71
68.14
66.77
67.29
APFL
71.01
75.48
71.21
71.08
70.97
L2SGD
67.39
69.96
69.11
65.88
67.31
FedAlt
68.46
72.74
69.72
67.52
68.45
PerFedAvg
67.47
71.25
64.22
70.56
67.35
FedBN
68.06
70.62
68.28
68.03
68.03
FedFOMO
64.69
68.56
65.36
64.18
64.53
FedRep
68.10
69.82
69.75
66.68
68.01
FedBABU
68.03
70.45
69.99
66.25
67.95
FedHKD 68.69
70.34
64.99
72.32
68.44
I.GMMix (Ours)
73.74
78.11
75.19
72.58
73.72
of each EEG signal as a feature and performed the tangent space projection for the matrices. The final input of each dataset is 1-d vector, and we further add aero-paddings have the same input dimension. We compared our method with others and reported the performance under five metrics in Table [O] From the table, FedAvg on such heterogeneous features, while our method outperforms all compared methods on five metries. This further validates the efficacy of our method 1-d signal data.
observe that PFL methods show better performance than the
APPENDIX E DISCUSSIONS
In this paper, present novel PFL approach aimed addressing the challenges posed by hetero- geneous features during the training process. Our proposed method incorporates considerations of the convergence rate of both local and global models. To achieve the model personalization, take the trace the Gram matrix using local/global updates for gradient descent as an approximation of the convergence rate. By employing the ratio of the trace, perform línear combination of local and global updates. Additionally, due to the substantial increase in computational costs associated with the Gram matrix as the number of samples grows, our implementation further approximates the calculation by calculating the trace of latent feature matrix. The latent feature is extracted from the last hidden layer. However, our method only considers the feature matrix to mix local and global updates. While this approximation is reasonable and reflects the convergence rate in principle, it may not be an accurate estimation when using large foundation models as the model backbone. When using large foundation models, such pre-trained deep neural networks, the convergence behavior may differ due to the model's architecture and complexity. such cases, relying solely on the last-layer latent feature for trace approximation might not capture the true convergence rate accurately. Therefore, there is a risk of obtaining less precise estimations in these scenarios. To ad- dress this limitation and obtain updates, one potential approach to consider the prediction errors of local and global models. By incorporating the model errors into the mixing process, we can potentially gain insights into the convergence behavior of the models. This approach would provide a more holistic perspective on the convergence rate and guide the combination of local and global updates more effectively. How- ever, exploring the use of large foundation models in federated learning is still an open question. The unique challenges and considerations introduced by these models need be thoroughly inves- tigated. Future research should focus validating the effectiveness of our method when applied to federated learning scenarios involving large foundation models.
comprehensive measurements for combining local and global
Our method aims to improve the personalization of client models for heterogeneous features. This can have benefits in various applications, such as healthcare, where medical images collected from different hospitals are heterogeneous. By improving the model personalization, our method can potentially improve the accuracy and effectiveness for better healthcare outcomes. Our work also has several potential implications for future research. First, approach provides on addressing heterogeneous features PFL, and can be extended to segmentation asks, whereas many existing PFL methods are only validated classification tasks. Second, work proposes
perspective
28

---

Published as a conference paper at ICLR 2024
an NTK viewpoint for analyzing PFL method by combining local and global updates, it will be interesting to extend the NTK framework for analyzing the general PFL frameworks.
29


Reviews:
[{"review_id": "G0QA9JdJSX", "reviewer": "Reviewer_i8zc", "summary": "This paper proposes a novel approach to address the challenge of data heterogeneity in personalized federated learning. The proposed method leverages the convergence rate induced by Neural Tangent Kernel to quantify the importance of local and global updates, and subsequently mix these updates based on their importance. The authors have theoretically analyzed and experimentally evaluated their method on five datasets with heterogeneous data features in natural and medical images. The results show that the proposed method outperforms existing methods in terms of convergence rate and accuracy. Overall, this paper contributes to the development of more effective and efficient personalized federated learning methods.", "questions": "1. how is eq. 4 derived? please give more explanantion on it. y is label, w(t) is the local model weights, what's the mearning of y-w(t)?\n2. What's the cost (e.g., GPU memory cost, computational time) of the proposed method, compared to the baseline methods?\n3. The proposed method is intuitive and interesting. What are the possible limitations of the proposed method, and how can they be addressed in future work?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed Local-Global updates Mixing approach is original and innovative, leveraging the convergence rate induced by Neural Tangent Kernel to address the challenge of data heterogeneity in personalized federated learning. This approach is a creative combination of existing ideas and provides a new perspective on how to handle feature heterogeneity.\n\nThe clarity of the paper is excellent, with a clear and concise writing style, well-organized sections, and informative figures and tables.", "weaknesses": "the authors could discuss the scalability and robustness of their method to larger and more complex datasets, and explore potential applications in other domains beyond natural and medical images. The paper could benefit from a more detailed discussion of the limitations and potential extensions of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZU68S7Vik6", "reviewer": "Reviewer_efqn", "summary": "This paper proposes a mixing local initialization in pFL studies via the ratio of traces of the local feature matrix and global feature matrix. The proposed method experimentally and efficiently improves the final performance. Ablation studies are also introduced to learn the efficiency of the mixing rate.", "questions": "Thanks for the great submission but I still find there are some important issues to be resolved in the current version. My concerns are lined in the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Writing is clear and easy to follow.\n2. The motivation of local initialization is sound and efficient in the pFL community. I appreciate the author's comparative analysis of ablation studies to verify the details of the proposed algorithm in the training process.\n3. A brief convergence analysis is provided.", "weaknesses": "1. In the Algorithm Line.6, how to calculate the $h(x_i)$? This feature seems to adopt the global feature extractor to generate the approximation of global convergence. While in the local client, this seems to be untenable without the access to the global model. Does it require storing a global model locally? If so, I think a storage comparison among baselines is required to comprehensively compare the efficiency.\n2. The authors should provide the training wall-clock time required to compare the practical efficiency. \n3. The motivation is to select the faster convergence to match the local or global model. So I think it is necessary to introduce the loss curve or gradient norm to illustrate its improvements. This paper does not analyze the improvements in the test error and generalization performance. Under the specific convergence analysis, how does the proposed method match the convergence analysis? And does the training loss or gradient norm decrease faster?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZpvpDFzETC", "reviewer": "Reviewer_Ho8d", "summary": "The paper addressed the issue of improving convergence of personalized federated learning using a mix of local and global model updates. The optimal mix is found through a measure related to the neural tangent kernel / trace of the Gram matrix. An approximation of the trace is introduced based on taking only the last layer into account. The method is demonstrated on several image problems.", "questions": "Is this critique fair: \"However, despite advancements, these methods aim to learn a common\nglobal model, but it is challenging to ensure the consensus global model is best-for-all.\"? It is my understanding that the objective of these methods is to learn a global model, so they are designed for another purpose than PFL.\n\nAre the methods described only applicable in the context of stochastic gradient descent? It would be beneficial to have a clear definition of the problem setup in which the proposed methods apply.\n\nIn sec. 3.1 both covariate shift and concept drift are mentioned. Is this method applicable to both scenarios? This should preferably be clear from the introduction.\n\nAre there any previous papers that consider the update strategy in eq. 3, or is this an idea that has not yet been explored?\n\nIt is not always clear to me if w_c and u denote models or model weights?\n\nWhere does eq. 4 come from? There is no details regarding its derivation or any clear reference.\n\nIs proposition 1 a novel result, and if so in which sense?\n\nIt is uncler to me if the mixing ratio in eq. 5 is optimal in some sense? Intuitively, the global and local model might both have different error and different convergence properties, so should the model error not also be taken into account?\n\nDoes it make a big difference that you only consider the last layer in the approximation of the Gram matrix? Can the arguments be supported empirically or theoretically?\n\nWhat exactly is meant by this statement: \"since the data is distributed and not directly accessible, we propose employing features from local data in conjunction with the global model u\"? Do you employ the representation (first layers) of the local model?\n\nIs it correctly understood that you use a fixed learning rate for all experiments, which is not optimized?\n\nWhat are the practical implications of the convergence analysis in sec. 4? How is the theorem and proof different form exising convergence results?\n\nI do not understand the arguments in the paragraph titled \"Feature value distribution by our personalized model\". What results is the statement \"If a model learns confident representations, then the related neurons should be highly activated...\" based on?\n\nIf instead we used a fixed (optimized) ratio of local and global updates, does your method have a clear advantage?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of mixing local and global model updates in a manner that optimizes the convergence in PFL is simple and intuitive. Making use of the Gram trace in this context makes good sense.\n\nThe proposed method is compared with many other relevant methods.", "weaknesses": "The problem setup could be more clearly defined from the beginning. The paper is motivated by a discussion of the challenges of data heterogeneity; however, it would be more clear if the authors in the beginning of the paper would include a clear definition of the type of heterogeneity that is addressed, and which personalized settings the developed methods apply to. Statements such as \"To solve the data heterogeneity problem...\" seem to indicate that heterogenous data is a kind of external obstacle, rather than an inherent property of the problem PFL is aiming to solve. \n\nIn the related works section, there is the following statement: \"However, the existing approaches have primarily focused on addressing heterogeneity related to class distributions, system resources, and model architecture. The issue of feature distributional shifts in PFL remains under-explored.\" While it may be true that the issue is underexplored, it would be great with a clear delineage of which existing works address this issue, and which of the related methods could reasonably expected to work under covariate shift.\n\n\"In FL, each client will update the local model using the global update after each communication\nround.\" While this is not wrong per se, there are many ways local and global updates are carried out in different algorithms, depending on the problem setup and assumptions behind the particular algorithm.\n\nIt is not clear if the results are a fair comparison, i.e. whether or not each algorithm is independently optimized sufficiently well.\n\nThe section on the convergence analysis seems a bit disintegrated with the rest of the paper.\n\nSimple experiments that highlight the applicability of the method are not included. I would have appreciated a clear demonstration, e.g. in comparison with fixed local-global ratios.\n\nAn empirical demonstration of proposition 1 would have been a strong argument for the specific choices made for the mixing ratio.\n\nMinor issues\nparamters -> parameters\nfootnote -> subscript\ntr should be roman in eq. 7", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)