FedL2P: Federated Learning to Personalize
Authors: Royson Lee
Minyoung Kim
Da Li
Xinchi Qiu
Timothy Hospedales
Ferenc Huszár
Nicholas Donald Lane
Venue: NeurIPS 2023 poster (2023)
Keywords: federated learning; meta-learning; hyperparameter optimization
TL;DR: FedL2P aims to learn a function (dubbed "meta-net") in the federated learning way that automatically outputs the optimal personalized hyperparameters for finetuning without time-consuming hyperparameter search.

Abstract: Federated learning (FL) research has made progress in developing algorithms for distributed learning of global models, as well as algorithms for local personalization of those common models to the specifics of each client’s local data distribution. However, different FL problems may require different personalization strategies, and it may not even be possible to define an effective one-size-fits-all personalization strategy for all clients: Depending on how similar each client’s optimal predictor is to that of the global model, different personalization strategies may be preferred. In this paper, we consider the federated meta-learning problem of learning personalization strategies. Specifically, we consider meta-nets that induce the batch-norm and learning rate parameters for each client given local data statistics. By learning these meta-nets through FL, we allow the whole FL network to collaborate in learning a customized personalization strategy for each client. Empirical results show that this framework improves on a range of standard hand-crafted personalization baselines in both label and feature shift situations.

FedL2P: Federated Learning to Personalize
Royson Leel2; Minyoung Kim², Da Li Xinchi Qiu', Timothy Hospedales Ferenc Huszár", Nicholas D. Lane
University of Cambridge, UK University of Edinburgh, UK 4 Flower Labs
Samsung AI Center, Cambridge, UK
Abstract
Federated learning (FL) research made progress in developing algorithms for distributed learning of global models, as well as algorithms for local personalization those common models to the specifies of cach client's 's local data distribution. However, differrnt FL problems may require different personalization strategies, and may not even be possible to define an effective one-size- fite-all personal- ization strategy for all clients: depending how similar each client's 's optimal predictor is to that of the global model, different personalization strategies may be preferred. In this paper, consider the federated meta-learning problem learning personalization strategies. Specifically, we consider meta-nets that induce the batch-norm and learning rate parameters for each client given local data statistics. By learning these meta-nets through FL., we allow the whole FL network collaborate in learning customized personalization strategy for each client. Empirical results show that this framework improves on range of standard hand-crafted personalization baselines both label and feature shift situations
Introduction
Federated learning (FL) is an emerging approach to enable privacy-preserte collaborative learning among clients who hold their own data. A major challenge of FL is learn from differing degrees of statistical data heterogeneity among clients. This makes it hard to reliably learn global model and also that the global model may perform sub-optimally for each local client. These two issues often dealt with respectively by developing robust algorithms for learning the global model B3 32 339 and then offering each client the opportunity personalize the global model its own unique local statistics via fine-tuning. In this paper, we focus on improving the fine-tuning process by learning personalization strategy for each client.
variety of approaches have been proposed for client personalization. Some algorithms directly learn the personalized models 662 (33, but the majority obtain the personalized model after global model learning by fine-tuning techniques such as basic fine-tuning 46 48). regularised fine-tuning [3] [5], and selective parameter fine-tuning [17, 2, 37]. Recent benchmarks (45] g showed that different personalized FL methods suffer from lack comparable evaluation setups. In particular, dataset- and experiment-specift. personalization strategies art performance. Intuitively, different datasets and FL scenarios require different personalization strategies. For example, scenarios with greater or lesser heterogeneity among clients, would imply different strengths of personalization are optimal. Furthermore, exactly how that personalization should be conducted might depend whether the heterogeneity is primarily in marginal label shift, marginal feature shift, or conditional shift. None of these facets can be well addressed by one size fits all personalization algorithm. Furthermore, we identify previously understudied issue: even
often required to achieve state-of of the-
Conresponning Author: dsr120cam. uk "Code available at http::/ggghuuh.coo/oyos///////te
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

for a single federated learning scenario, heterogeneous clients may require different personalization strategies. For example, the optimal personalization strategy will have client-wise dependence on whether that client is more or less similar to the global model in either marginal or conditional data distributions. Existing works that learn personalized strategies through the use of personalized weights are not scalable to larger setups and models [55] 14] On the other hand, the few studies
that attempt to optimize hyperparameters for fine-tuning do not sufficiently address this issue as they
either learn a single set of personalization hyperparameters [65] [2] and/or learn hyperparameter
distribution without taking account of the client's 's data distribution
30
In this paper, we address the issues above by considering the challenge of federated meta- weta-learning of personalization strategies. Rather than manually defining a personalization strategy as is mainstream in FL [27] 9 we use appeergraddent meta-learning strategies to efficiently estimate personalized hyperparameters. However, apart from standard centralised meta-learning and hyperparameter opti- mization (HPO) studies which only need to learn a single set of hyperparameters, we learn meta-nets which inductively map from local client statisties to client-specific personalization hyperparameters. More specifically, our approach FedL2P introduces meta-nets to estimate the extent in which to utilize the client-wise BN statistics as opposed to the global model's BN statistics, as well as to infer layer-wise learning rates given each client's 's metadata.
Our FedL2P thus enables per deseetttstt/sccnariio, as well as per-client, personalization strategy learning. By conducting federated meta- mneaa-learnngg of the personalization hyperparameter networks, we simul- tancously allow each client to benefit from its own personalization strategy. (e.g.. learning rapidly, depending on similarity to the global model), and also enable all the clients to collaborate by learning the overall hyperparameter networks that map local meta-data to local personalization strategy. Our FP generalizes many existing frameworks as special cases, such as FedBN (35.. which makes a manual choice to normalize features using the client BN's statistics, and various selective fine-tuning which make manual choices which layers to personalize.
approaches 37 48
2
2 Related Work
Existing FL works aim to tackle the statistical heterogeneity of learning personalized models by either first learning a global model [46] 39 32
48 [9] and then fine-tuning it on local data or
directly learning the local models, which can often be further personalized using fine-tuning, Many
personalized Fl. approaches include the use of transfer learning between global (56) and local
models 44). model regularization 34] Moreau envelopes [59] and meta-learning [15] 10 Besides
algorithmic changes, many works also proposed model decoupling, in which layers are either shared
54 or client clustering, which assumes a local model for each 53 7 These methods often rely on or adopt a fixed personalization policy for local fine-tuning in order to adapt a global model or further improve personalized performance. Although there exists numerous FL approaches that propose adaptable personalization policies [55] 40 14] these works are memory intensive and do not scale to larger setups and models. On the other hand, our approach has a low memory footprint (Appendix () and is directly applicable and complementary to existing FL approaches as it aims to solely improve the fine-tuning process.
or personalized 2
11
64
48
37
17
cluster 42
Another line of work involves HPO for FL. (or FL for HPO). These methods either learn one set of hyperparameters for all clients 22 30 65 or random sample from learnt hyperparameter categorical distributions which does not take into account of the client's meta-data (30]. Moreover, some of these methods [22 65 search for set of hyperparameters based on the local validation loss given the initial set of weights prior to the federated learning of the model 222 65), which might be an inaccurate proxy to the final performance. Unlike previous works which directly learn hyperparameters, we deploy FL to learn meta-nets that take in, as inputs, the client meta-data to generate personalized hyperparameters for given pretrained model. A detailed positioning of our work in comparison with existing literature can be found in Appendix B
3 Proposed Method
3.1 Background & Preliminaries
Centralized FL. A typical centralized FL setup using FedAvg 46) involves training a global model 0, from C clients whose data are kept private. At round t, g is broadcast to a subset of clients selected, O using a fraction ratio r. Each selected client, i C, would then update the model using
2

---

Elient
Denaneliation Sindleyy2 muc)
Model % (Pize (re-trraned))
Steca-
Daite
Meeeegradiend
Cutrut
Pensmelication Strategy X (Meta-net)
Now Cliant Meta Data
Server
Global Model
Fine-Tune
Personalized Model of:
Global Mode 6,
New Personalized Model y
Fine-Tume
(1) FL of Personalization Strategy on known clients
Data
Client
Client
NEW
(2) Deploy
Persanalization
Dath
Strategy for new client
Figure 1: FcdL2P's workflow. For each client, the meta-net (paramcterized by A) takes the client meta-data data (e.g., local data profile) and outputs hyperparameters. We update A by optimizing the meta objective which is the validation loss of the model finetuned with the hyperparameters returned by the meta-net. The updated personalization strategies X" from clients are collectedlaggregateed (via FedAvg) the server for the next round.
a set of hyperparameters and its own data samples, which are drawn from its local distribution P; defined over X' y for compact space X' and label space y for e epochs. After which, the learned local models, 0--1, are sent back to the server for aggregation 0g " Σ.
N. 0 N,: where N; is
the total number of local data samples in client i and the resulting 0t is used for the next round. The aim of FL is either to minimize the global objective E(.,,))-<[(0 v.,) for the global data distribution P or local objective E(z.y))-; Ce(O;; x. y) for all E C where (,( x, y) is the loss given the model parameters at data point (x. y). As fine-tuning the dominant approach to either personalize from a high-performing 0g to optimize 0, further for each client, achieving competitive or state-of-the-ar results in recent benchmarks 45 21, we focus on the collaborative learning of meta-nets which generates personalized set of hyperparameters used during fine-tuning further improve personalized performance without compromising global performance.
Non-IID Problem Setup. Unlike many previous works, our method aims to handle both common label and feature distribution shift across clients. Specifically, given features and labels y, we can rewrite the joint probability (a y) as P: (xlv) P:(y) and P; (. (:r) following
(28). We focus on
three data heterogeneity settings found in many realistic settings: both label & distribution skew in which the marginal distributions (1) & P. (z) may vary across clients, respectively, and concept drift in which the conditional distribution (r(r)) may vary across clients.
3.2 FedL2PP FL of Personalization Strategies
We now present our proposed method, FedL,2P, to tackle collaborative learning of personalization strategies under data heterogeneity. Our main motivation is that the choice of the hyperparameters for the personalized fine-tuning, such as the learning rates and feature mean and standard deviations (SD) statisties of the batch normalization (26) (BN) layers, is very crucial. Although existing FL-HPO approaches aim to learn these hyperparameters directly 122 65| 30,, we aim to do it in meta learning or hypernetwork-linn profiles of client data (c.g., summary statisties of personalized data) input and outputs near-optimal hyperparameters. The meta-net is learned collaboratively in an FL manner without sharing local data. The returned hyperparameters from the meta-net then deployed in client's personalized fine-tuning. The main advantage of this meta-net approach over the direct HPO is that brand new client needs not do time-consuming HPO, but just gets their optimal hyperpameters by single feed-forward pass through the meta-net. Our idea is visualized in Fig. | For each FL round, the latest meta-net distributed to the participating clients; each client then performs learning to update the meta-net; the local updated meta-nets are sent to the server for aggregation. Details of our algorithm are described below.
fashion: learning a neural network (dubbed meta-net) that takes certain
Hyperparameter Selection. There is a wide range of local training hyperparameters that can be optimized, some of which were explored in previous FL works (22
65 30 As local training is often a costly process, we narrowed it down to two main sets of hyperparameters based on previous works that showed promising results in dealing with non-IID data: BN hyperparameters and selective update hyperparameters.
Batch Normalization Hyperparameters. The first set of hyperparameters involves BN and explic- itly deals with feature shift. Notably, FedBN (35 proposed keeping the BN layers local and the other layers global to better handle feature shift across clients; BN has found success at mitigating domain
TTo our knowledge, previous FL-HPO works leam BN statistics, crucial for dealing with feature shifts.
3

---

shifts in various domain adaptation tasks 36 8
and can be formulated as follows:
((x)
Vo2+c Y + 5
(I)
where y is a BN layer, is its input features, (ū,, 62) are the estimated running mean, variance, and used for numerical stability. During training, the batch statistics E(x) and V Y'rraa (æ) are used instead, keeping running estimates of them, f and 8². During inference, these estimates are used Both and are learnable parameters used to scale and shift the normalized features.
Although deploying local BN layers is useful to counteract the drawbacks of feature shift, using global BN layers is often beneficial for local fine-tuning in cases where the feature shift is minimal as it helps speed-up convergence and reduces overfitting. Hence, we propose learning a hyperparameter for each BN layer as follows:
f = (1 - 3) Apt + B * Pr.
8² = (1- * 6p + B x 6²
(2)
where Apt and oe are the estimated running mean and variance by the given pretrained model; p.; and of are the running mean and variance estimated by the client. When B - 0, the client solely 1, it uses its local statistics to normalize features. Thus 3 indicates the degree in which the client should utilize its own BN statistics against pretrained model's, to handle the feature shift in its local data distribution.
uses pretrained model's BN statistics and when B >,,,
Selective Update Hyperparameters. A va- riety of recent personalized FL works achieved promising results by manually selecting and fine-tuning a sub-module of 0g during person- alization 48 2 37 (e.g. the feature extrac- tor or the classifier layers), while leaving the remaining modules in fg frozen. It is bene- ficial because it allows the designer to man- age over- vs under-fitting in personalization. egg., if the per-elient dataset is small, then fine-tuning many parameters can easily lead to overfitting, and thus better freezing some layers during personalization. Alternatively, if the clients differ significantly from the global model and/or if the per-client dataset is larger,
Algorithm 1 FedI 2P: FL. of meta-nets for Personaliza- tion Hyperparameters
Input: Pretrained global model 0g with M layers and B BN layers, each has J input channels. Fraction ratio т. Total no. of clients C. No. update iterations K. Training loss Ly and validation loss Lv. Ç is the learning rate for X.
1: initialize À = [won, Wir: 7) 2: for round t = I,..., T do
3: Ö <- Random sample Cr clients
4: for elient E Ö do
5: send 0g. À client
6: Forward pass of local dataset to compute
1) E(rm) ). (D(m)) for 1 - m I M -1
2) pb.j. oh for b = 1,..., B and
1.
then more layers could be beneficially person-
7:
Compute Eu for b 1, I,.... B using Eq 3
alised without underfitting. Clearly the opti-
8:
for iteration == I,..., do
mal configuration for allowing model updates
9:
0, + Finetune 0g using Lr for e epochs
10:
X - x Hypergraddentt(V. fr, X, 0,)
depends on the specific scenario and the spe-
11:
end for
cific client. Furthermore, it may be beneficial
12: send À and num of data samples N server
to consider a more flexible range of hyper-
13: end for
parameters that control a continuous degree of fine-tuning strength, rather than binary
14: À + Σ ΣNA
15: end for
frozen/updated decision per module.
Output: À
To automate the search for good personaliza- tion strategies covering range of wider and more challenging non- IID setups, we consider layer-wise learning rates, 17 for all learnable weights and biases. This parameterization of personalization encompasses all the pre- vious manual frozen/updated split approaches as special cases. Furthermore, while these approaches have primarily considered hetero- geneity in the marginal label distribution, we also aim to cover feature distribution shift be- tween clients. Thus we also include the learn- ing rates for the BN parameters, y and 8, allowing us to further tackle feature shift by adjusting the means and SD of the normalized features.
Algorithm Hypergradient
non-IID
Input: Validation Loss Ly and training Loss Lr. Learning rate 1 and no. of iterations Q. Fixed point (X', 0 (X))).
1: v= BoLvlex (=(2')))
2: for iteration 1,... Q do
3: r 1-ーーシ t v 86 Lor
4: p+ppe
5: end for
Output: 0 OxC (X'))) p Pro (x())
When the running mean & variance not tracked, the batch statistics is used in both training and inference.

---

Hyperparameter Inference for Personalization. We aim to estimate a set of local hyperparameters that can best personalize the pretrained model for each client given group of clients whose data might not be used for pretraining. To accomplish this, we learn to estimate hyperparameters based on the degree of data heterogeneity of the client's local data with respect the data that the model is pretrained on. There are many ways to quantify data heterogeneity, such as utilizing the earth mover's distance between the client's s data distribution and the population distribution for label distribution skew [63] or taking the difference in local covariance among clients for feature shift [35]. In our case, we aim to distinguish both label and feature data heterogeneity across clients. To this end, we utilize the client's local input features to each layer with respect to the given pretrained model. Given pretrained model with M layers and B BN layers, we learn 77 = ./..... 72M and 3 = .I...., .,.....,.... SB. using two functions, each of which is parameterized as a multilayer perceptron (MLP), named meta-net, with one hidden layer due to its ability to theoretically approximate almost any continuous function [12 157- We named the meta-net that estimates (3 and the meta-net that estimates 7 as BNNet and LRNet respectively. Details about the architecture can be found in Appendix, E
To estimate 3, we first perform forward pass of the local dataset on the given pretrained model, computing the mean and SD of each channel of each input feature for each BN layer. We then measure the distance between the local feature distributions and the pretrained model's running estimated feature distributions of the b-th BN layer as follows:
E,b = -= 1 (DKL(P)1Q)) (Dx(((11(1110))) + (KL(()11P)))
where P3 = (N ), Q2 = (Npptt. on p. DEL is the KL divergence and J is the number of channels of the input feature. is then used as an input to BNNet, which learns to estimate as shown in Eq.
Similarly, we compute the mean and SD of each input feature per layer by performing a forward pass of the local dataset on the pretrained model and use it an input to LRNet. Following best practices from previous oL hyperparameter optimization works 31, we use a learnalee post-multiplier M, ................................ π/a to avoid limiting the range of the resulting learning rates (Eq
B BNNet(Wbni E1, . (B-1, 7 LRNet(wi,; F(zo), (z))): F(m)),
where is the Hadamard product, Im is the input feature to the m-th layer, and Whn and wh are the parameters of BNNet and LRNet respectively. ,3 is used to compute the running mean and variance in the forward pass for each BN layer as shown in Eq, and vy is used as the learning rate for each weight and bias in the backward pass. We do not restrict 77 to be positive as the optimal learning rate might be negative (5).
Federated Hy Hyperparameter Learning. We deploy FedAvg 446 to federatedly learn set of client- specific personalization strategies. Specifically, we learn the common meta-net {wb,.. Wir, n that generates client- wise personalization hyperparameters {B,, ), J. such that group of clients can better adapt a pre-trane trained model fgy by fine-tuning to their local data distribution. So we solve:
C
And min F(),0)) Σ
N
(O No
E.
s.t. (7(A) = arg min L (0,, 9:
(5)
where 0 is the set of optimal personalized model parameters after fine-tuning 0g for epochs on the local dataset, (,, A) = E (,,, x; y) and V; is the validation set (samples from P.) for the client i similarity for L,Tr
For each client the validation loss gradient with respect to X, known as the hypergradient, can be computed as follows:
dv (8" =
(O)) (,CC(((A)))))))CC(((((((),.,,,,
(6)
To compute 0z0" in Eq. 6 we use the implicit function theorem (IFT):
We assume all layers have weights and biases here.
5

---

1x (0) (x) 9x(x))
The full derivation is shown in Appendix A We use Neumann approximation and efficient vector- Jacobian product as proposed by Lorraine et al. (38] to approximate the Hessian inverse in Eq. 7 and compute the hypergradient, which is further summarized in Algorithm 2 In practice, 0" is approximated by fine-tuning Og on Lr using the client's dataset. Note that unlike in many previous works [38] 47 where 0xCv is often 0 as the hyperparameters often do not directly affect the validation loss, in our case Sws.. Lv * 0.
Algorithm| [ summarizzes FedL2P. Given a pretrained model and a new group of clients to personalize. we first initialize Ả (line 1). For every FL round, sample Cr clients and send both the parameters of the pretrained model and À to each client (lines 3-5). Each client then performs a forward pass of their local dataset to compute the mean (E) and standard deviation (SD) of the input features to each layer and the statistical distance between the local feature distributions and the pretrained model's 's running estimated feature distributions for each BN layer (lines 6-7). is then trained for K' iterations; each iteration optimizes the pretrained model on LT for e epochs, applying 3 and n computed using Eq. (lines 8-9) at every forward and backward pass respectively. Each client then computes the hypergradient of À as per Algorithm. 2 and update À at the end of every iteration (line 10). Finally, after K iterations, each client sends back the updated and its number of data samples, which is used for aggregation using FedAvg [46] (lines 12-14). The resulting À is then used for personalization: each client finetunes the model using its training set and evaluates using its test set.
3.3 Adapting the Losses for IFT
In the IFT, we solve the following problem:
min (v (A), s.t. (A()) arg min Lr A)-
For the current X, we first find 0 (A) in (8) by performing several SGD steps with the training loss fr- Once 0* (A) is obtained, we can compute the hypergradient d, Lv (A), A) by the IFT, which is used for updating A. As described in (6) and (7). this hypergradient requires OxCr(0, A), implying that the training loss has to be explicitly dependent on the hyperparameter X. As alluded in Lorraine al. (38), is usually not straightforward to optimize the learning rate hyperparameter via the IFT, mainly due to the difficulty of expressing the dependency of the training loss on learning rates. To address this issue, we define the training loss as follows:
Lr A) = B(z.)) CEWs (T)) where = - (a
(10)
Here fo,)(z) indicates the forward pass with network weights 0 and the batch norm statistics B. and CE() is the cross-entropy loss. Note that in (10). we can take several (not just one) gradient update steps to obtain 8'. Now, we can see that Lr X) defined as above has explicit dependency on the learning rates (A). Interestingly, the stationary point 0* (A) of Cr(O, A) coincides with 0', that is, (A) = 0', which allows for a single instance of inner-loop iterations as Line 9 in Alg. 1 Finally, the validation loss is defined as:
(v, A) = -x CE(fo,s()))))..
showing clear dependency on BNNet parameters through ((A) as discussed in the previous section.
4 Evaluation
Table 1: FedL.2P complements existing FL
methods by improving the finetuning pro-
4.1 Experimental Setup
cess. Experiments on CIFAR1O (c = 15).
Experiments are conducted on image classification tasks of different complexity. We use ResNet-18 (20 for all experiments and SGD for all optimizers. All
Approach
D
+F
1090 ( beleogeneitt))
FodAvg
65042002
65. 1381122
Perfed/vgiLF)
34.5820113
47.5840.01
FedBADU
65.0000.77
66 0940.3
details of the pretrained models can be found in
PedAvg
06.76A
PerfedAggiUF)
44.8520
50.211.26
Appendix. D Additionally, the batch size is set to
FedBADU
68 $8.920011
78.71
32 and the number of local epochs, e, is set to 15 unless stated otherwise. The learning rate (C) for {urm, Wir, 7) is set to (10-3,10 3,10-,1100---11 spectively. The hypergradient is clipped by value
PedAvg
3420114
08.45A1LS
H))
12.43-0.16
55.0540.53
FedRARU
2640.1
72.8740.22
0.1 batooggnwity)
KrlAAg
N 15
50. 20.2010.07
re-
77.51-0.05
77 6840.11
FedRABU
79.50e0.08
79.58441.04

---

Table 2: Experiments CIFAR-10 using pretrained model trained using FedAvg 46 Both initial &
personalized accuracies are learnt and personalized on the same set of clients.
Epochs (e)
Global Accuracy
+FT (BN C)
+FT (BN G)
+BT
+LPP
+FedL2P
1000
5
65. 13
64.3560.03
62.14.0.13
56.58+0.08
536140.11
64.53=0.06
( beterugencitt)
15
65.13
63.04+0.02
59.85+0.04
55,72+0.03
58.4120.38
65.1320.02
M0.19
39.7240.18
63.454004
50.8=0006
55.81+6003
66.08200.09
15
60.19
61.42+0.13
63,2340.15
54,9440.07
61.77+0.25
65.76=0.31
0.5
5
57.12
58. 58.22:00.02
67.1620.11
50,3320.01
39.7920.09
68.9620.09
15
57.12
62.34a0.12
67.420.06
58.1240.07
28,00.22
68.45.0.5
0.1 <t beterngencity)
5
44.86
68,0420.05
78.7320.04
61.1140066
74,2620.29
80.33=0.13
15
44.86
79.15s0.07
78.9740.07
75.94.0.0
78.620.21
80.28
[,,, Q= 3, and a = 0.1 in Alg. 2 The maximum number of communication rounds is set to 500, and over the rounds we save the À value that leads to the lowest validation loss, averaged over the participating clients, as the final learned X. The fraction ratio r r0 0.1 unless stated otherwise, sampling 10% of the total number of clients per FL round. We focus non-IlI) labels and feature shifts while assuming that each client has an equal number of samples. Finally, to generate heterogeneity in label distributions, we follow the latent Dirichlet allocation (LDA) partition method [24 61 52]: ~ Dír(a) for each client. Hence, the degree of heterogeneity in label distributions is controlled by a; a decreases, the label non-IIDness increases, and vice versa.
4.1.1 Datasets
CIFARIO (31] A widely-used image classification dataset, also popular an FL benchmark. The number of clients C' is set to 1000 and 20% of the training data is used for validation.
CIFAR-10-C (21). The test split of the CIFARIO dataset is corrupted with common corruptions. We used 10 corruption types with severity level 3 and split each corruption dataset into 40%%20% training/test sets. A subset (20%) of the train set is further held out to form validation set. Each corruption type is partitioned by Dir(a) among 25 clients, hence C = 250.
Office-Caltechh 10 19] & DomainNet (51). These datasets are specifically designed to contain several domains that exhibit different feature shifts. We the no. of samples of each domain be the smallest of the domains, random sampling the larger domains. For Caltech-10, we set r = 1.0 and C 4, one for each domain and thus partitioned labels are IID. For NomainNet, we set = 150, of which each of the 6 domain datasets are partitioned by Dir(a) among 25 clients, resulting in challenging setup with both feature & label shifts. Following FedBN [35] we take the top 10 most commonly used classes in DomainNet.
Speech Commands V2 [60]. We use the 12-class version that is naturally partitioned by speaker, with one client per speaker. It is naturally imbalanced in skew and frequency. with 2112, 256, 250 clients/speakers for train, validation, and test. We sampled 250 of 211 12 training clients with the most data for our seen pool of clients and sampled 50 out of 256+250 validation and test clients for our unseen client pool. Each client's data is then split 80%/20% for train and test sets, with a further 20% of the resulting train set held out to form a validation set.
4.1.2 Baselines
We run all experiments three times and report the mean and SD of the test accuracy. We consider three different setups of fine-tuning in our experiments as below. As basic fine-tuning (FT) used ubiquitously in FL. our experiments directly compare FT with FedL2P in different non-IID scenarios. We are agnostic to the FL method used to obtain the global model or to train the meta-nets.
FT (BN C). Client BN Statistics. Equivalent to setting 3 = 1 in Eq. 2 thus using client statistics to normalize features during fine-tuning. This is similar to FedBN [35] and is adopted by FL works such as Ditto 34] and PerFedAvg [15].
FT (BN G). Equivalent to setting 3 = O in Eq. 2 BN layers use the pretrained global model's BN
statistics to normalize its features during fine-tuning.
FT (BN D). BN layers use the incoming feature batch statistics to normalize its features during fine-tuning. This setting is adopted by FL works such FedBABU (48).
L2P. Our proposed method without FL; À is learnt locally given the same compute budget before being used for personalization. Hence, 1.2P does client-wise HPO independently.
4.2 Experiments on Marginal Label Shift
We evaluate our approach based on the conventional setup of personalized FL approaches [9 45). where global model, fg. is first learned federatedly using existing algorithms and then personalized
brightness frost. compression, contrast, snow, motion blur, pixelate, speckle_noiieeee noise. saturate
7

---

Table 3: Personalized test accuracies of CIFAR-IO--- -O-C, EfficeCCaltech-De 10, Domainnet (e 15).
Dataset
+FT (BN C)
+FT (BN G)
+FT (BN D
+1.2P
+FedI.2P
1000
CIFAR-TO-C
59,58+0.03
57.03+0.08
55.7740.12 12
8..8020.1
59.9720.22
(1 heterogeneity)
Caltech-10
36.0222..21
81,4342.16
75.52+3.83
88.85=0.89
DomainNer
52.17+1.55
30,55+1.07
50,4740.88
45.04+1.56
54.38=0.45
1.0
CIFAR-TD-C
67.3750.08
66.4520.03
63.36#0.07
66.6350009
68.83=0.13
DomainNet
62.27200.8
44.1540.11
59,420.7
54.7520.12
63.7720.44
0.5
CI-AR-10-C
74.9260.08
75.24#0.17
71.38+0.011
74,48+0.06
76.78=0.22
DomainNet
71.3910.97
49.81+1.98
68.9440.71
66.38+0.78
72.6440.3
0.1
CI-AR-10-C
87,25+0.06
88.5=0.02
83,9140.04
87,9340.31
89.23=0.15
(1 heterogeneity)
DomainNet
86.03+0.47
69,41=1.95
85.35+1.14
83.93=1.02
86.36=0045
the same set of clients via fine-tuning. To this end, given the CIFAR- 10 dataset partitioned among
a group of clients, we pretrained 0, following best practices from 23 using FedAvg and finetune on the same set of clients. Table 2 shows the personalized accuracy of the various fine-tuning baselines (Section 4.1.2) and FedL2P using e = 5& local epochs on groups of clients with varying label distribution, P.( a 1000 represents the IID case and a 1.0, 0.5, 0.1 represents more heterogeneous case. As observed in many previous works [48] 27 increasing label heterogeneity would result in better initial global model a expense of personalized performance. Our method instead retains the initial global performance and focuses on improving personalized performance.
We also show that in many cases, especially for clients with limited local compute budget e utilizing the pretrained model's BN statistics result (BN G) can be more beneficial as CIFAR-10 consists of images from the same natural image domain; in contrast, previous works mainly use either the client's $ BN statistics (BN C) or the incoming feature batch statistics (BN I) to normalize the features. This strategy is discovered by FedL2P, as illustrated in Fig. 2a where the learned / is O for all BN layers of all clients. For the IID case in particular, FedL2P learn sparsity fo of 1.0,
learning rate 77 = 0, for all layers in all clients, forgoing tine-tuning and using the initial global model as the personalized model. For a = 1.0 and 0.1, FedL2P learns highly sparse models similar to recent works that proposed fine-tuning only a subset of hand-picked layers 48 obtain performance gains. Lastly, 1.2P performs worse than some standard fine-tuning baselines as it meta-overfits on each client's validation set, highlighting the benefits of FL. over local HPO.
2 37 to
FedL2P's Complementability with previous FL works. As our proposed FedL2P learns to improve the FT process, it is complementary, not competing, with other FL methods that learn shared mo)el(s). Hence, besides FedAvg, we utilize FedL2P to better personalize 0g pretrained using PerFedAvg(HFF [1] and FedBABU 48 as shown Table. where we compare FedL2P against the most commonly used FT approach, BN C. Our results show that applying FedL2P to all three FL methods can lead to further gains, in most cases outperforming FT in each respective method. This performance improvement can also bridge the performance gap between different methods. For instance, while FedAvg+FT has worse performance than FedBABU+FT in all cases, FedAvg+Fedl.22 obtained comparable or better performance than FedBABU+FT for a 1000 & 0.1.
Sparsity
Sparsity
0.00
0.1100
3.6
0.5
5.0
0.0000
0.4
5.022
-0.00
5.000
1760
1.0
35
21
2300
1.0
7.5
24
3002
1.0
05
0.2
1100
3.5
5.5
0.2
1010
3.2
61
1000
1.0
2.4
(a) CIFARIO (e 15)
(b) CIFARIOC
Sparsity
Sparsity
1.0
5.125
0.0se
0.54
0.8
3.003
0:22
0.0125
0.0120
0.30
CA
0.0075
0.2
50125
0.2
0.0050
0.54
0.
(c) DomainNet
(d) Office-Caltec--11
Figure 2: Locality, spread, and skewness of FedL_2P's learned hyperparameters, f and n, of different layers across clients and the model's sparsity of all clients for each personalization scenario.
4.3 Personalizing Unseen Clients
Unseen during Pretraining. We evaluate the performance of FedL.2P on CIFAR-10-C starting from the pretrained model trained using FedAvg on CIFAR-10 in the IID setting a = 1000 (Section. 4.2) as
*Sparsity refers the percent of parameters whose learned learning rate for FT is

---

Table 4: Experiments on Speech Commands. Both the pretrained model and meta-nets are only learned on the seen pool of clients.
Client Food
Epochs (c)
Global Accurncy
4FT C)
OBN G)
4FT (BN 1)
+1.2P
4FedL.2P
Seen
82.1
70.070063
2,98:0.00
65.9740.43
64.79+1.44
87.77H00.4
15
82.11
74.15a0.70
2,9849.0
68.9820,19
65.8011.66
84.7440.388
Unser
5
81
62.76=188
2.5580.00
6488411.56
87.88=0018
15
81.62
08.7623.40
2.8500.00
07.28%11.66
84.60=0135
These new clients, whose data is partitioned from CIFAR-IO-C, did not participate in the pretraining of the global model and, instead, only participate in the training of the meta-nets. As added noise would impact the feature shift among clients, Fig. 2b shows FedL2P learns to use = as compared to sole use of 3 = in the CIFAR-IO case (Fig. 2a); some clients benefit more by using its own BN statistics. Hence, the performance gap for non- IID cases between BN C and BN
shown in Table. 3]
G is significantly smaller as compared to CIFAR-10 10 shown in Table.
Unseen during Learning of Meta-nets. We evaluate the case where some clients are not included in the learning of the meta-nets through our experiments on the Speech Commands dataset. We first pretrain the model using FedAvg and then train the meta-nets using our seen pool of clients. The learned meta-nets are then used to fine-tune and evaluate both our seen and unseen pool of clients as shown in Table. 4 Note that L2P is not evaluated on the unseen pool of clients the meta-nets learned on the seen pool of clients. We see that line-tuning the pretrained model on each client's 's train set resulted in significant drop in test set performance. Moreover, each client represents a different speaker, fine-tuning using the pretrained model's BN statistics (BN G) fails entirely. FedL2P, unseen client pools, e.g. by utilizing a mix of pretrained model's 's BN statistics and the client's 's BN statistics shown in Appendix F
only evaluate
the other hand, led to a significant performance boost both the seen and
Table 5: Ablation study for FedL2P with 15.
Dataset
(BN C)
+FT (BN G)
-FedL2F
(BNNet)
(LRNet) 3-1
-FedL2F
(LKNet) (-0
+Fed
+FedL.2P
1080
CIFAR-10
63.04-0.22
85+0.00
02.35-032
62 01
65.13=0.02
() trimopractt)
CIFAR-10.C
59.50+000
07.0000
59.57
60.09
59.97=0
Cattech-10
80,9720.33
36.02225.21
88. 88.12-1118
85 5025.
42.509222 H7
88.85=0.89
DomainNer
52,1741.55 1.55
30 55+1.07
53 0
28.59
44.43- 46
54.58-0.45
1.6
CITAR 10
61.4210.13
132310.15
63.35004
64 06
6161100 19
65.7620.31
CIFAR-10-C
67.3720008
45.00.0
120
82.0.
68.3
DomainNet
62.2720.58
44.15
52.72-051
63.0920.43
diverge
63.77:0.44
0.5
CIFAR-TO
62.3220.144
67.40006
6831=8005
60
63.4 65.452030
CI-CR-IOCC
74,92+00.08
75.24
76. 76.36.0.08
76.8629.06
76.11
76.82.0.19
DomainNet
71.3920.99
49.8121.96
1059
72.74:0.51
72,64.0 30
CIEAR-I 10
79.1550:07
78.9720:07
19447002
80.2410004
80,3920115
07
(1 (enoogeneltt)
CIFAR- I0.C
87,25.0006
85. 5...00
88 6.0
89 89.08.0.04 04
89 14.0
89.23.0.15
DomsinNer
86.03+0447
69.41+1.95
85.87-1.31
85.78 65.78-0.6
diverge
86.36-0.45
4.4 Personalizing to Different Domains
Table 6: ARI values between the clustering of clients by their domains and inputs/outputs of the BNNet and LRNet.
We evaluate our method on Office- Cafiie-Caltec-100 and DomainNet datasets commonly used in domain gen- cralization/adappttiion which exhibit both marginal and conditional feature shifts. Differing from the con- ventional FL setup, we adopted a pretrained model trained using ImageNet [13) and attached prediction head as our global model. Similar to CIFAR- 10-C experiments, we compare our method with the base- lines in Table. 3 Unlike CIFAR- 10/10-C, OfficeCClltech-00 and DomainNet consist of images from
Dulasel
BNNo
1.RNer
Input (E)
Ouiput (1)
Trput (at)
Ouiput (t))
Caltech-10
1000
1.0
1.0
1.0
1.0
0.65
0.64
0.76
0.77
DomainNer
0.5
0.70
0.52
0.59
0.53
0.77
0.73
0.63
0.74
0.1
0.27
0.52
0.65
0.59
distinct domains and adapting to these domain result in a model sparsity of 0 as shown in Fig. 2c and Fig. 2d Hence, a better personalization performance is achieved using the client's is own local BN statistics (BN C) or the incoming batch statistics (BN I) than the pretrained model's (BN G). Lastly, each client's ' personalized model uses an extent of the pretrained natural image statistics as well as its own domain statistics, for both datasets.
Further Analysis To further investigate how well the hyperparameters returned by our FedL2P- trained meta-nets capture the domain-specifie information, we analyze the clustering of local features, (Do),, (DD to the meta-nets), and the resulting respective hyperparameters, B and n7 (i.e., outputs of the meta- nets), among clients. Specifically, we compute the similarity matrix using the Euclidean distance between the features/hyperparametes of any two clients and perform spectral clustering") using the
namely & = (E1, En) and æ = (F(zo),
(i.e., the inputs F(xM-1), SD
We use the default parameters for spectral clustering from scikit-learn 50

---

(a) Office-Caltech-ID 10
(b) DomainNet (a- 0.5)
Figure 3: Cluster distance maps. Each block represents normalized distance, where the distance of the block (j. k) is measured as the average of the Euclidean distances between all pairs of clients' B that belong to domain j and domain k (similarly for v7 y's), normalized by the within-domant distance (see text). An off-diagonal block > 0 indicates that the corresponding clusters are better aligned with the true domains.
discretization approach proposed in [58] to assign labels on the normalized Laplacian embedding. We then compute the Adjusted Rand Index (ART) [25] between the estimated clusters and the clusters partitioned by the different data domains as shown in Table. 6
We also visualize the alignment between the estimated clusters and the true domains in Fig. where each block (i, k)) represents the normalized average Euclidean distance between all pairs of clients in domain j and k. Specifically, we divide the mean distance between domain j and k by the within-doman mean distances and d(j.k)
take the log scale for better visualization: log( VaGisy /(kkA) where d(j,, k) is the mean Euclidean
distance between j and k. Thus, random clustering has distance close to 0 and indicates better alignment between the clustering and the true domains.
As shown, for DomainNet, both BNNet and LRNet consistently preserve the cluster information found in their inputs, & & 2. respectively. However, perfect clustering not achieved due to the inherent difficulty. For instance, the real and painting domains share similar features, resulting in similar hyperparameters; the cross-domain distance between real and painting is in log-scale in Fig. 3 and hence indistinguishable from their true domains. In contrast, the clients' features and resulting hyperparameters of the Effice-Caltech 10 dataset are perfectly clustered (ARI=I) as visualized in Fig. 3a and Appendix. F
4.5 Ablation Study
To elucidate the individual impact of BNNet & LRNet, we an ablation study of all of the datasets
used in our experiments and present the results in Table. 5 where CIFAR10 10 adopts the pretrained model trained using FedAvg. As BNNet learns to weight between client's BN statistics (BN C) and pretrained model's i BN statistics (BN G), running FedL.2 with BNNet alone leads to either better or similar performance the better performing baseline. Running LRNet standalone, on the other hand, can result in further gains, surpassing the use of both BNNet and LRNet on some benchmarks. Nonetheless, requires prior knowledge the data feature distribution of the client in order set suitable B, of which B = uses BN C and / O uses BN G. Our approach assumes no knowledge of the client's data and learns an estimated /3 per-scenario and per-client using BNNet.
Conclusion
In this paper, we propose FedL2P, a framework for federated learning of personalization strategies specific individual FL scenarios and datasets as well as individual clients. We learned meta-nets that use clients' local data statistics relative to the pretrained model, to generate hyperparameters that explicitly target the normalization, scaling, and shifting of features as well as layer-wise parameter se- lection to mitigate the detrimental impacts of both marginal and conditional feature shift and marginal label shift, significantly boosting personalized performance. This framework is complementary to ex- isting FL works that learn shared model(s) and can discover many previous hand-designed heuristics for sparse layer updates and BN parameter selection as special cases, and learns to apply them where appropriate according to the specific scenario for each specific client. As a future work, our approach can be extended to include other hyperparameters and model other forms of heterogeneity, e.g. using the number of samples as expert input feature to a meta-net.
Acknowledgements
This work was supported by Samsung Al and the European Research Council via the REDIAL project.
10

---

References
[1] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375, 2018.
[2] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers, 2019.
[3] Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta- learning with adaptive hyperparameters. Advances in Neural Information Processing Systems, 33:20755-20766 2020.
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308. 3432, 2013.
[5] Alberto Bernacchia. Meta-learning with negative learning rates. In International Conference on Learning Representations, 2021.
[6] Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas D Lane. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007. /4390, 2020.
[7] Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical cluster- ing of local updates to improve training non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.
[8] Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain- specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 7354-7362, 2019.
[9] Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding. pfl-bench: A com- prehensive benchmark for personalized federated learning, Advances in Neural Information Processing Systems, 2022.
[10] Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.0788 2018.
[1I] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning. PMLR, 2021.
[12] Balázs Csanád Csáji et al. Approximation with artificial neural networks. Faculty of Sciences, Etvs Lornd University, Hungary, 24(48):7, 2001.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248 245. IEEE, 2009.
[14] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, arXiv preprint arXiv:2003. 13461, 2020.
[15] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 2020.
[16] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning. PMLR, 2017.
[17] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 2020.
[18] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
11

---

[19] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. pages 2066-2073 IEEE, 2012.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference Learning Representations, 2019.
[22] Stephanie Holly, Thomas Hiessl, Safoura Rezapour Lakani, Daniel Schall, Clemens Heitzinger, and Jana Kemnitz. Evaluation of hyperparameter-opiiiizz federated learning system. In Data Science-Analyie and Applications. Springer, 2022.
approaches in an industrial
[23] Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I Venieris, and Nicholas D Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. arXiv preprint arXiv:2102. 13451, 2021.
[24] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.00 2019.
[25] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2(1): 193-218, 1985.
[26] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[27] Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan, Improving federated learning personalization via model agnostic meta learning, arXiv preprint arXiv: /2488, 2019.
(28) Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2): 1-210, 2021.
[29] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
[30] Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florina FF Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight-sharing. Advances in Neural Information Processing Systems, 34, 2021.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers 2009.
of features from tiny images.
[32] Qinbin Li, Bingsheng Hc, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[33] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 2020.
[34] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning. PMLR, 2021.
[35] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2 2021.
12

---

[36] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv: 1603.0479 2016.
[37] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Phillppe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.015223 2020.
[38] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In Infernational Conference Artificial Intelligence and Statistics, pages 1550-1552. PMIR, 2020.
[39] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of hetero- geneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 2021.
[40] Xiaosong Ma, Jie Zhang. Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[41] TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. huttps://itthin cww wow/pytorch/vissi.tittoooooeeeee 2016
[42] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 1066/99902./6999, 2020.
[43] Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Lactitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 2021.
[44] Koji Matsuda, Yuya Sasaki, Chuan Xiao, and Makoto Onizuka. Fedme: Federated learning via model exchange. In Proceedings of the 2022 SIAM International Conference Data Mining (SDM). SIAM, 2022.
[45] Koji Matsuda, Yuya Sasaki, Chuan Xiao, and Makoto Onizuka. An empirical study of personal- ized federated learning, arXiv preprint arXiv:2206.1311 2022.
[46] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficii learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 2017.
[47] Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. In International Conference on Learning Representations, 2021.
[48] Jaehoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. arXiv preprint arXiv:2106. 2021.
[49] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste, Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31, 2018.
[50] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-leam: Machine learning in Python, Journal of Machine Learning Research, 12:2825-2883 2011.
[51] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1406-1415, 2019.
[52] Xinchi Qiu, Javier Fernandez- Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. In International Conference Learning Representations, 2021.
13

---

[53] Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[54] Mehdi Setayesh, Xiaoxiao Li, and Vincent WS Wong. Perfedmask: Personalized federated learning with optimized masking vectors. The Eleventh International Conference on Learning Representations.
[55] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, 2021.
[56] Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXiv:2006. 16765, 2020.
[57] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zonghen Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.
[58] Yu Stella and Jianbo Shi. Multiclass spectral clustering. IEEE International Conference on Computer Vision, volume 2, pages 313-313, IEEE Computer Society, 2003.
[59] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:11394-24405, 2020.
[60] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 03209, 2018
[61] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pages 7252-7261. PMLR, 2019.
[62] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. In International Conference Learning Representations, 2021.
[63] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint 1rXiv:/806.000 2018.
[64] Aoxiao Zhong, Hao He, Zhaolin Ren, Na Li, and Quanzheng Li. Feddar: Federated domain-
aware representation learning. In International Conference on Learning Representations, 2023.
[65] Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. Single-shot general hyper-parameter optimization for federated learning. The Eleventh International Conference on Learning Representations, 2023.
14

---

Appendix
A Derivation of Equation
From Eq (]. we know that:
dC,,T (0; d0
(11)
we Based on the implicit functional theorem (IFT), we get that if we have a function F(z, y) can derive that y/ (x) -Fa/F Fy. Therefore, plus the Eq (11) into the theorem, we can get:
0g (dC,, de
d0
(0)Cr(0, OxoLr(0,A)
(12)
dA
0₂ dCi.r (0)
B Positioning of FedL2P.
Table 7 shows the positioning of FedL2P against existing literature. Note that this list by no means exhaustive but representative to highlight the position of our work. Most existing approaches obtained personalized models using a personalized policy and local data, often through finetuning-basee approach. This personalized policy can either 1) be fixed, e.g. hand-crafting hyperparameters, layers freeze, selecting number of mixture components, number of clusters or learned, e.g. learning a hypernetwork to generate weights or meta-nets to generate hyperparameters. These approaches also grouped based on whether this personalized policy dependent on the local data during inference, e.g. meta-nets require local client meta-data to generate hyperparameters.
In order to adapt to per-dataset per-client scenarios, many works rely on storing per-client personalized layers, which are trained only on each client's local data. Unfortunately, the memory cost of storing these models scales with the number of clients, C, restricting previous works to small scale experiments. We show that these works are impractical in our CIFAR-10 setup of 1000 clients in Appendix. C Moreover, most existing methods rely fixed personalized policy, such deriving shared global hyperparameters for all clients in FLoRA, or they do not dependent on local data, such FedEx which randomly samples per-elient hyperparameters from learned categorical distributions. Hence, these methods do not adapt as well to per-dataset per-client scenarios and are ineffective at targeting both label and feature shifts. Lastly, although PFedHN and pFedLA target both label and feature shift cases, they do not scale well to our experiments as shown in Appendix. C
Table 7: Positioning of FedL2P with existing FL approaches. C is the total number of clients, M is the number layers the model, the number BN layers in the model, D is the number of data domains, H is the number of hidden layers in the hypernetork.
Approach
Shared
Learns
Personalzzed
Layers
Persomilzzdd
Pullcy by?
Persmalized
Policy Dependent?
Targets
Targets
Memory
Large
Scale
FeaProx
33
PerFed/ing
15
PFedMe
Ditte
34
39
Hised
No
Yes
No
FedBABU 48
Port Maakk
S4
Yes
No
Fixed
No
Yes
No
OLM
FedBNN
Hised
No
No
OCCB)
FedRop |TI
APFI.
Yes
Yes
Fixed
No
Yes
No
OXCM)
1.G-RedAve St
IFCA [7
FeaLAR
(XDM)
PLaKA 65
Vos
No
Fiscl
No
Yoe
No
COM
Fealxx
Yes
Supported
Leamed
No
Yes
No
OLMO
KallEM
1edFOMO 62
FedMe 40
No
Yes
Fixed
No
Yos
No
OXCM)
PlediNN 64
Leuned
Yes
Yes
OICMH)
Nell.PP (Oun))
Supported
Lammel
Yes
Yes
Yos
OMM
15

---

Most importantly, all existing FL approaches shown in Table 7 can use finetuning either to personalize shared global model(s) or as a complementary personalization strategy further adapt their personal- ized models. Since our proposed Fedl. .PP focus on better personalizing shared global model(s) by learning better a personalized policy which leverages the clients' local data statisties relative to the given pretrain model(s), our approach is complementary to all existing FL. solutions that learn shared model(s); we showed improvements over a few of these works in Table
Use Cases of FedL2P Mainstream FL focuses on training from scratch, but we focus on federated learning of strategy to adapt an existing pre-trained model (whether obtained by FL or not) on an unseen group of clients with heterogeneous data. There are several scenarios where this setup and our solution would be useful:
1. Scenarios where it's expensive to train from scratch for a new group of clients, e.g. adopting FedEx [30] from scratch for new group of clients would require thousands of rounds to retrain the model and HP while our method takes hundreds to learn two tiny meta-nets (Appendix. (
2. Scenarios where there is publicly available pre-trained foundation model that can be exploited. This is illustrated in Section (.4 where we adapt a publicly available pretrained model trained using ImageNet domain generalization datasets.
3. Scenarios where it's important to also maintain global model with high initial accuracy often neglected by previous personalized FL works.
Note that our approach also does not critically depend on the global model's performance. Even in the worst case where the input statisties derived from the global model are junk (e.g., they degenerate to constant vector, or are simply random noise vector), then it just means the hyperparameters learned are no longer input-dependent. In this case, FedL2P would effectively learn a constant vector of layer-wise learning rates + BN mixing ratio, as opposed to a function that predicts them. Thus, in this worst case we would lose the ability to customize the hyperparameters differently to different heterogeneous clients, but we would still be better off than the standard approach where these optimization hyperparameters are not learned. In the case where our global-mode derived input features are better than this degenerate worst case, FedL2P's meta-nets will improve on this already strong starting point.
C Cost of FedL2P
Computational Cost of Hessian Approximation. We compare with hessian-free approaches, namely first-order (FO) MAML and hessian-free (HF) MAML, both of which are used by PerFedAvg. and measure the time it took to compute the meta-gradient after fine-tuning- Specifically, we run 100 iterations of each algorithm and report the mean of the walltime. Our proposed method takes 0.24 seconds to compute the hypergradient, 0.12 seconds of which is used to approximate the Hessian. In comparison, FO-MAML took 0.08 seconds and HF-MAML took 0.16 seconds to compute the meta- gradient. Hence, our proposed method is not significant overhead relative to simpler non-Hessian methods. It is also worth noting that the number of fine-tuning epochs would not impact the cost of computing the hypergradient.
Memory Cost. In our CIFARIO experiments, the meta-update of FedL2P has a peak memory usage of 1.3GB. In contrast, existing FL methods that generate personalized policies require order(s) of magnitude more memory and hence only evaluated in relatively small setups with smaller networks. For instance, HFedHN (55) requires in a peak memory usage of 17.93GB in our CIFAR 10 setup as its user embeddings and hypernetwork scale up with the number of clients and model size. Moreover, they fail to generate reasonable client weights these techniques do not scale to larger ResNets used in our experiments. APFL [4,, on the other hand, requires each client to maintain three models: local, global, and mixed personalized. Adopting APFL in our CIFAR 10 setup of 1000 clients requires over 134GB of memory just to store the models per experiment, which is infeasible.
Communication Cost. For each FL round, we transmit the parameters of the meta-nets, which are lightweight MLP networks to from server to client and vice versa. Note that transmitting the global pretrained model to each new client is one-time cost. Offfic-Caltecc-11., DomainNet, and Speech Commands setups take a maximum of 100 communication rounds, 0.24% of the pretraining cost, to learn the meta-nets. CIFAR-10 10 and CIFAR-LO-C, on the other hand, can take hundreds of
16

---

rounds up to a maximum of 500 rounds, 0.38% of the pretraining cost. In contrast, joint model and hyperparameter optimization typically takes thousands of rounds [30], having to transmit both the model and the hyperparameter distribution across the network. In summary, Fedl.2P .2P incurs < % additional costs on top of pretraining and forgoes the cost of federatedly learning a model from scratch, which can be advantageous in certain scenarios as listed in Section. B
Inference Cost. During the fine-tuning stage, given the learned meta-nets, FedL2P requires 2 forward pass of the model per image and one forward pass of each meta-net to compute the personalized hyperparameters. This equates to 0.55GFLOPs per image and would incur a minor additional cost of 4.4% more than the regular finetuning process of 15 finetune epochs.
D Pretrained Model and Setup Details
We use the Flower federated learning framework 6 and NVIDIA GeForce RTX 2080 Ti GPUs for all experiments. R8sNet-18 200 is adopted with minor differences in the various setups:
CIFAR-10. We replaced the first convolution with a smaller convolution x 3 kernel with stride= 1 and padding- 1 instead of the regular 7 x 7 kernel. We also replaced the max pooling operation with the identity operation and set the number of output features of the last fully connected layer to 10. The model is pretrained in federated manner using FedAvg 46 or FedBABU [48] or PerFe- dAvg(HF) [15] with a starting learning rate of 0.1 for 500 communication rounds. For PerFedAvg, we adopted the recommended hyperparameters used by the authors to meta-train the model. The fraction ratio is set to r = 0.1; 100 clients, each of who perform a single epoch update on own local dataset before sending the updated model back to the server, participate per round. We dropped the learning rate by factor of 0.1 round 250 and 375. This process is repeated for each a = 1000, 1.0, 0.5, 0.1, resulting in a pretrained model for each group of clients. We experiment with various fine-tuning learning rates {1.0, 0.1, 0.01, le - 3, le 4, le - 5) and pick the cest-pefoormnnggg one, le 3 3 for all experiments; the initial value of r in FedL2P is also set at le- - 3.
CIFAR-10-C. We adopted the pretrained model trained in CIFAR-10 10 for a 1000 and used the same fine-tuning learning rate for all experiments.
& DomainNet. We adopted Resnet-18 model that was pretrained on Ima- geNet 13 and provided by torchvision [4]. We replaced the number of output features of the last fully connected layer 10. Similar to CIFAR-10 setup, we experiment with the same set of learning rates and pick the best-perforning one, le - 2 for our experiments.
Resnet-1 18 Speech Commands. We adopted the setup and hyperparameters from ZeroFL [52|
model is trained using FedAvg for 500 rounds using a starting learning rate of 0.1 and an exponential learning rate decay schedule with base learning rate of 0.01. We use the base learning rate for fine-tuning for all experiments.
E
Architecture & Initialization Details
We present the architecture of our proposed meta-nets, BNNet and LRNet. Both networks are 3-layer MLP models with 100 hidden layer neurons and ReLU [I activations in-between layers. BNNet and LRNet clamp the output to value of [0, and [0, 1000) respectively and use straight-through estimator 4 (STE) to propagate gradients. We also tried using sigmoid function for BNNet which converges the same solution but much slower pace. We initialize the weights of BNNet and LRNet with Xavier initialization
[88 using the normal distribution with a gain value of 0.1. To control the starting initial value of BNNet and LRNet, we initialize the biases of BNNet and LRNet with constants 0.5 and 1.0, resulting in initial values of ~ 0.5 and ~ 1.0 respectively. We also tried experimenting BNNet with different initializations by setting the biases to [0.2, 0.5, 0.8] and got similar results.
F Additional Results
Relative Clustered Distance Maps. We present an extension of Fig. for both the inputs, & & a. and outputs, (3 & 17, of the meta-nets in Fig.
17

---

Table 8: Comparison between using +OMAML+ and IFT in FedL2P for Sifice-Caltech- 10 and
Domainnet (e 15).
Dataset
+BN C)
WFOL2P(FOMMAML
+FedL (IFT)
1000
Caltech-10 10
80.9200335
83.20=1
DomainNet
52.17+1.55
52.70=0.17
54.38200455
1.0
DomainNer
62,27=0.58
63.14+0.38 58
63.77=00.4
DomainNet
71.39+0.97
71.37=0.79
0.1
DomainNet
86.03a0.47
86,2240.16
86.3620.45
B
Sparsity
0.4
0.015
0.8
0.3
0.010
0.6
0.4
0.005
0.1
0.2
0.0
0.000
1000
1.0
0.5
0.1
1000
1.0
0.5
0.1
1000
1.0
0.5
0.1
α
α
α
(a) CIFARIO (e
5)
n
Sparsity
1.0
0.0150
0.710
0.0125
0.705
0.0100
0.6
0.0075
0.4
0.700
0.0050
0.2
0.0025
0.695
0.0
0.0000
(b) Speech Commands
Figure 4: Locality, spread, and skewness of FedL2P's learned hyperparameters, f and n, of different layers across clients and the model's sparsity all clients for each personalization scenario.
Learned Personalized Hyperparameters. We present the learned hyperparameters for the other client groups not shown in Fig. 2 in Fig.
Comparison with FOMAML. We remark that off-the-shelf FOMAML [66] (as initial condition learner) is not meaningful point of comparison because our problem is to fine-tune/personaliie pre-trained models. Therefore, compare with FOMAML, we focus learning our FedL2P meta-nets with FOMAML style meta-gradient instead of IFT meta-gradients. In order to apply FOMAML to learning rate optimization, this also required extending FOMAML with the same trick as we did for our IFT approach as shown Eq. 9 & TO However, FOMAML ignores the learning trajectory except the last step, which may result in performance degradation over longer horizons. We term this baseline FOMAMI+ Table, reports results on the multi-domain datasets described in Section. 4..4. keeping the same initial condition and meta representation (LRNet and BNNet meta-nets), and varying only the optimization algorithm. Our approach of using IFT compute the best response Jacobian, outperforms FOMAML+ for 1000, 1.0, 0.5 and has comparable performance for a = 0.1.
18

---

(a) Office-Calteee Caltech- (a 1000)
(b) DomainNet (a 1000)
(c) DomainNet (a 1.0)
(d) DomainNet (a 0.5)
SK
44
10
64
(c) DomainNet (a 0.1)
Figure 5: Cluster distance maps. Each block represents normalized distance between two domains (e.g., Caltech vs. DSLR), where the distance of the block (j, k) is measured as the average of the Euclidean distances between all pairs of clients' 3 that belong domain j and domain & (similarly for v))). We normalize distance by the within-domain distances (see text), and take log for better visualization. Hence, an off-diagonal block greater than indicates that the corresponding clusters are better aligned with the true domains.
19


Reviews:
[{"review_id": "mUpTGI2d0S", "reviewer": "Reviewer_P9q9", "summary": "The paper proposes FedL2P, a federated learning framework for learning personalized strategies in the federated meta-learning problem. The authors introduce meta-nets to estimate hyper-parameters for personalized fine-tuning, such as batch normalization parameters and learning rates, based on client-specific data statistics. The empirical results show that FedL2P outperforms standard hand-crafted personalization baselines in both label and feature shift situations.", "questions": "- What is the computational cost associated with the proposed method?\n- How many parameters will be personalized in this study?\n- Could you provide insights into the learning rate distribution for different blocks/layers in the paper and elaborate on what the model has learned? This would greatly enhance the understanding of the results.\n", "limitations": "I'm not aware of any potential negative societal impact of the work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method demonstrates soundness in its approach. Utilizing hyperparameter inference for federated learning shows promise and effectiveness in achieving personalization, as it tries to customize only a few crucial hyper-parameters.\n- The experiments are meticulously evaluated across diverse distribution shifts, encompassing feature shift, label shift, and domain shift.", "weaknesses": "- Calculating meta-gradients involves time-consuming computations of second-order derivatives.\n- The comparison is limited to variants of finetuning, neglecting the majority of personalized federated learning baselines such as [1,2]. Additionally, recent works [3] addressing distribution shifts by optimizing different blocks of the network should be thoroughly discussed in the paper.\n- The sufficiency and effectiveness of communicating the selected hyper-parameters for personalized federated learning lack rigorous verification.\n\n[1] Towards personalized federated learning. TNNLS\n\n[2] Layer-wised model aggregation for personalized federated learning. CVPR 2022\n\n[3] Surgical fine-tuning improves adaptation to distribution shifts. ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and feedback. We hope our response below will address the reviewer’s concerns and questions. We will revise the paper acoordingly to include these discussions.\n\n> Q5.1) Calculating meta-gradients involves time-consuming computations of second-order derivatives. What is the computational cost associated with the proposed method?\n\nThe computation cost of approximating the Hessian and computing the hypergradient is detailed in Appendix C. To reiterate, our proposed method takes 0.24 seconds to compute the hypergradient as compared to FOMAML, which takes 0.08 seconds, and HF-MAML, which takes 0.16 seconds, to compute the meta-gradient. Hence, our approach does not incur a significant overhead compared to these simpler non-Hessian methods. During fine-tuning, our proposed method incurs a minor additional cost of 4.4% more than regular fine-tuning.\n\n> Q5.2) The comparison is limited to variants of finetuning, neglecting the majority of personalized federated learning baselines such as [1,2]. Additionally, recent works [3] addressing distribution shifts by optimizing different blocks of the network should be thoroughly discussed in the paper.\n\n[1] surveys a wide range of methods used for personalized FL. The majority of the works listed in this paper can be mostly improved via further local fine-tuning as shown in recent large-scale personalized FL benchmark papers [9,43 in the paper]. As our proposed method learns to improve the fine-tuning process, it is complementary to many of these works. In Table 1, we showed three representative cases, namely FedAvg, PerFedAvg, and FedBABU. FedAvg optimizes on the initial global accuracy, PerFedAvg optimizes directly on the personalized accuracy after fine-tuning, and FedBABU improves both initial and personalized accuracy through model splitting (global encoder + local prediction heads). Our FedL2P is shown to improve on all the above. \n\nIn Appendix B, we discussed the scalability of a wide range of existing personalized FL works and Appendix C details the cost of our approach in comparison with existing works that require storing of local models and hypernetworks to generate per-client models. To reiterate, in relation to the reviewer’s suggested baseline [2], hypernetwork-based approaches scale poorly with the number of clients and the model size. In our experiments with pFedHN, the hypernetwork fails to generate reasonable client weights as it does not scale to larger ResNets used in our experiments. Note that pFedHN and [2] used a toy model of 5 layers with a maximum of 100 clients whereas we used a full-blown ResNet with a maximum of 1000 clients.\n\n[3], along with other papers that fine-tunes a different subset of the model [46,11,14,36,16], manually handcraft a selection of layers to freeze and personalize. Specifically, [3] showed that, depending on the dataset and the pretrained model, freezing and updating different parts of the model showed performance gains. In contrast, our approach, automatically learns not only which layers to freeze/update but also the learning rate of each layer according to the specific scenario for each specific client, namely the client’s data distribution and dataset in relation to the pretrained model. Hence, our approach encompasses, generalizes, and automates many previous manually designed heuristics for layer-wise updates.\n\n> Q5.3) The sufficiency and effectiveness of communicating the selected hyper-parameters for personalized federated learning lack rigorous verification. Could you provide insights into the learning rate distribution for different blocks/layers in the paper and elaborate on what the model has learned? This would greatly enhance the understanding of the results. How many parameters will be personalized in this study?\n\nIn our results, the learning rates of different layers is heavily dependent on the scenario and client. While previous handcrafted solutions mentioned in Q5.2) surgically freeze specific parts (e.g. first block/middle block/last layer) of the model for **all** clients, we observed that, in most cases, our approach often freezes and updates layers of various different parts of the model with different layer-wise learning rates for **each** client. For instance, in our CIFAR-10 ($\\alpha=0.1$) setup, our approach learns highly sparse models (~0.9 as shown in Figure 2a), updating the first convolution layer of the 1st, 2nd, and 3rd block, along with the last layer weights and 2nd convolution layer of the 2nd block depending on the client. An exception to this is found in our CIFAR-10 IID ($\\alpha=1000$) setup, where our approach learns to freeze the global model (0 parameters personalized) as seen in Figure 2a ($\\eta=0$ and model sparsity = 1). Similarly, in our Office-Caltech-10 and DomainNet experiments, our approach learns to update all layers of the model to adapt to the feature and concept distribution drift as seen in Figure 2c & d (model sparsity = 0). All in all, the number of parameters, which parameters, and the extent in which we update the parameters depends on the scenario and the client.\n\n\n\n\n\n\n\n\n"}, {"review_id": "Xb1ZOw2yBo", "reviewer": "Reviewer_nqFj", "summary": "The paper introduces a method for fine-tuning a client's model based on a pretrained model using its local data. The proposed personalization approach involves fine-tuning the learnable weights and biases (assume, there are $M$ such layers) as well as the means and standard deviations (SDs) of batch normalization (BN) layers (assume, there are $B$ BN layers).\n\nThis fine-tuning process is facilitated by two meta neural networks. The first meta neural network predicts $2M$ learning rates, which correspond to each weight and bias layer, enabling the local model to perform several gradient steps from the pretrained model. The input for this first meta neural network consists of the mean and standard deviation values for each of the $M$ layers.\n\nOn the other hand, the second meta neural network directly predicts the means and SDs. Its input comprises the distance between the pretrained means and the running means of the model with the pretrained weights on local data, as well as a similar distance measure between the SDs.\n\nBoth meta neural networks are trained using FedAvg on hyper gradients.\n\n--------\nPost-rebuttal update: I did not have significant concerns about this paper. I acknowledge the authors' rebuttal and extend my gratitude to them, particularly for adding the new experiment. I have opted not to modify my favorable rating of 6.", "questions": "1. Line 225: If I understand correctly, the text implies that $\\partial_{{\\omega}_{lr}} {\\cal L}_V = 0$. If my interpretation is correct, could you please explain why this is the case?\n\n2. Equation 3 & Line 199: What is the range of the index $i$?\n\nMinor comments & suggestions:\n1. Line 6: Depending -> depending\n2. Line 35: Even -> even\n4. Inaccuracy on lines 87-91: Although it is evident what the authors intended to convey, the wording is imprecise. I would recommend specifying the subset of chosen clients as $\\tilde{C}$ on line 87, as done in Algorithm 1, and then using $i \\in \\tilde{C}$ instead of $i \\in C$ on line 88. The statement $P_i = \\cal{X} \\times \\cal{Y}$ on line 90 is also misleading: $\\cal{X} \\times \\cal{Y}$ represents the domain of the distribution and does not define the distribution itself. Accordingly, the summation on line 91 should be taken over $i \\in \\tilde{C}$.\n5. Line 93: The space before the word \"for\" is missing.\n6. The first part of the statement on lines 93-95, in my opinion, is incorrect. From a statistical perspective, the aim of Federated Learning is to minimize the generalized local loss function $\\mathbb{E}_{(x, y) \\in P_i} \\cal{L}_i (\\theta, x, y)$. However, many Federated Learning frameworks minimize the finite sum because they make an implicit assumption that the distributions are similar. Otherwise, one could construct two distributions $P_1$ and $P_2$ such that the global solution for $1/2(P_1 + P_2)$ is ineffective for both clients.\n7. The abbreviation SD is used on line 109 but only defined later on line 180.\n8. Line 121: \"are\" -> \"is\"\n10. Line 191: For clarity, I would suggest spelling out the abbreviation MLP.\n11. Line 190: Please provide explicit definitions and domains for $\\eta_i$ and $\\beta_j$ as they are not scalars.\n12. Equation 5 is misleading. The loss function on both lines is taken over the data distribution $P_i$, but there are also indices $T$ and $V$, which denote train and test datasets. $theta_i(\\lambda)$ is not a minimizer of the generalized loss function but rather a fine-tuning of the global solution $\\theta_g$. It takes some time to understand the authors' intended meaning here.\n13. Line 226: There is a redundant full stop after the first word \"Algorithm.\"\n14. Algorithm 1, the input section: The phrases \"Total no. of clients C. Num of update iterations K\" use different notation to express \"number,\" leading to inconsistency within the text.\n15. Lines 9-10 in Algorithm 1 do not clearly depict the intended FedAvg iterates.\n16. The document concludes on page 8 with Section \"4.4 Personalizing to Different Domains.\"", "limitations": "Authors adequately address the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper demonstrates sufficient originality by introducing a novel approach using meta neural networks for computing personalized hyperparameters, as outlined in the summary above. Notably, the concept of trainable meta nets holds promise and appears to have practical applicability.\n\n2. The authors effectively elaborate on the algorithm's intricacies, providing comprehensive descriptions of various components of the main method. Additionally, Figure 1 illustrates a schematic depiction of the main approach, enhancing the clarity of their explanations.\n\n3. Extensive computational experiments are incorporated within the paper to substantiate the efficacy of the proposed method. \n\n4. The paper is a valuable contribution to the fields of personalized Federated Learning and Meta Learning.", "weaknesses": "1. Despite the authors' efforts in providing extensive explanations for their ideas, regrettably, the text still contains several notable inconsistencies, as detailed in the \"Questions\" section.\n\n2. A comparative analysis with the FOMAML method [1] is notably absent in the work. It is worth mentioning that FOMAML shares significant similarities with the authors' proposed method, as both utilize the MAML approach for fine-tuning the model using the training dataset and making optimization steps for the meta parameter based on the validation dataset, as observed in the current study. Additionally, in another related work [2], which builds upon MAML, the hyper gradient is explicitly addressed (see lines 14 and 16 of Algorithm 1 in [2]).\n\n[1] \"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks\", Finn et al.\n\n[2] \"Unsupervised Meta-Learning for Few-Shot Image Classification\", Khodadadeh et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and for providing constructive feedback to improve our work. We appreciate the questions and all the detailed minor comments and suggestions and will adjust them accordingly in the revised paper, fixing algorithmic, notation, and text ambiguities and errors.\n\n> Q1.1) Line 225: If I understand correctly, the text implies that $\\partial\\_{\\boldsymbol{w}\\_{lr}}\\mathcal{L}\\_{V} = 0$. If my interpretation is correct, could you please explain why this is the case?\n\nYes, your understanding is correct. This is because validation loss does not directly depend on $\\boldsymbol{w}\\_{lr}$; there is no direct gradient as the learning rate used does not contribute to the validation loss. $\\boldsymbol{w}\\_{bn}$, on the other hand, is used in the forward pass in order to compute the validation loss, hence $\\partial\\_{\\boldsymbol{w}\\_{bn}} \\mathcal{L}\\_{V} \\neq 0$.\n\n> Q1.2) Equation 3 & Line 199: What is the range of the index $i$?\n\n$i$ represents client $i \\in C$. Hence, the range is the number of clients, $|C|$.\n\n> Q1.3) A comparative analysis with the FOMAML method [1] is notably absent in the work. It is worth mentioning that FOMAML shares significant similarities with the authors' proposed method, as both utilize the MAML approach for fine-tuning the model using the training dataset and making optimization steps for the meta parameter based on the validation dataset, as observed in the current study. Additionally, in another related work [2], which builds upon MAML, the hyper gradient is explicitly addressed (see lines 14 and 16 of Algorithm 1 in [2]).\n\nThank you for the suggestion. The primary difference in MAML-based approaches and our approach is in the choice and design of the quantity to meta-optimize in the bilevel optimization – i.e., our meta-nets. MAML learns a **constant** initial condition of the inner loop of Eq. 5. We learn a **function** that predicts the hyperparameters of the Eq5. inner loop optimization from local client metadata (Eq. 4), e.g. the BN and learning rate hyperparameters. A secondary difference is our use of IFT to obtain meta-gradients, in contrast to FOMAML’s first order approximation to meta-gradient computation. We remark that the intersection of these points – our use of learning a learning rate prediction function with IFT – is noteworthy, because it is not trivial to learn learning rates directly with IFT meta-gradient as discussed in L243-246. \n\nWe remark that off-the-shelf FOMAML (as an initial condition learner) is not a meaningful point of comparison because our problem is to fine-tune/personalize pre-trained models. Therefore, to compare with FOMAML, we focus on learning our FedL2P meta-nets with FOMAML style meta-gradients instead of IFT meta-gradients. In order to apply FOMAML to learning rate optimization, this also required extending FOMAML with the same trick as we did for our IFT approach as shown in Eq 9 & 10. However, FOMAML ignores the learning trajectory except the last step, which may result in performance degradation over longer horizons. We term this baseline FOMAML+. \n\nThe table below, reports results on the multi-domain datasets as described in Section 4.4, keeping the same initial condition and meta representation (LRNet and BNNet meta-nets), and varying only the optimization algorithm.  Our approach of using IFT to compute the best response Jacobian, outperforms FOMAML+ for $\\alpha=1000,1.0,0.5$ and has comparable performance for $\\alpha=0.1$. \n\n| Dataset    | α    | +FT (BN C) | +FedL2P (FOMAML+) | +FedL2P (IFT) |\n|------------|------|------------|-------------------|---------------|\n| Caltech-10 | 1000 | 80.97±0.33 | 83.2±1.92         | 88.85±0.89    |\n| DomainNet  | 1000 | 52.17±1.55 | 52.7±0.17         | 54.38±0.45    |\n| DomainNet  | 1.0  | 62.27±0.58 | 63.14±0.38        | 63.77±0.44    |\n| DomainNet  | 0.5  | 71.39±0.97 | 71.37±0.79        | 72.64±0.3     |\n| DomainNet  | 0.1  | 86.03±0.47 | 86.22±0.16        | 86.36±0.45    |\n\n\n\n"}, {"review_id": "6bgaTgdhmS", "reviewer": "Reviewer_aauE", "summary": "The paper introduces a novel approach that utilizes meta-learning to learn hypernets that aim to determine the optimal fine-tuning hyperparameters for individual clients. The proposed idea is intriguing and holds promise. The results presented in the paper are compelling. However, the paper would benefit from improvements in terms of writing and clarity. Overall, it is a borderline paper. \n\n--- post rebuttal --\n\nIn the rebuttal, the author address my questions about clarity and positioning their work. So I increase my score from 4 to 5. ", "questions": "- (line 184) Could you explain what is \"the degree of data heterogeneity of the client’s local data with respect to the pretrained model\"? The data heterogeneity comes from local data and the pretrained model?\n- (line 205) Should $BNNet(w_{bn}; ξ_1, ξ_2, . . . , ξ_{B−1}, ξ_B)$ be $BNNet(ξ_1, ξ_2, . . . , ξ_{B−1}, ξ_B;w_{bn})$? It seems that  $ξ_1, ξ_2, . . . , ξ_{B−1}, ξ_B$ are BNNet's input.\n- (line 206) Could you explain why there is a Hadamard product for the LRNet's output?\n- title of section 3.3 is a bit weird. why you are Define Training/Validation Losses for IFT (implicit function theorem). Isn't it that training/validation losses should defined by your tasks?", "limitations": "Not discussed. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes a novel idea of using meta-learning to learn hypernet that can estimate the best fine-tuning hyperparameters for personalizing local models. \n- The results show the proposed FedL2P can effectively improve performance on multiple standard FL methods and remain robust when there is a data distribution shift across clients. ", "weaknesses": "- The **clarity** needs to be improved. Especially, some details of the proposed method are missing. Please see the question section below.\n- The proposed method is **limited** to one certain type of personalized federated learning, i.e., fine-tuning local models. There are other strategies of personalization that are not applicable to this work. For example, one strategy is decoupled model (global shared encoder + local predictive head) [1,2]. Another strategy could be using Hypernet to directly optimize different local models [3].\n\n[1] Zhong, A., He, H., Ren, Z., Li, N. and Li, Q., 2022, September. FedDAR: Federated Domain-Aware Representation Learning. In The Eleventh International Conference on Learning Representations.\n\n[2] Collins, L., Hassani, H., Mokhtari, A. and Shakkottai, S., 2021, July. Exploiting shared representations for personalized federated learning. In International conference on machine learning (pp. 2089-2099). PMLR.\n\n[3] Shamsian, A., Navon, A., Fetaya, E. and Chechik, G., 2021, July. Personalized federated learning using hypernetworks. In International Conference on Machine Learning (pp. 9489-9502). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback and valuable time spent on our paper. We will revise the paper accordingly to address the points you have made to improve clarity.\n\n> Q2.1) The proposed method is limited to one certain type of personalized federated learning, i.e., fine-tuning local models. There are other strategies of personalization that are not applicable to this work. For example, one strategy is decoupled model (global shared encoder + local predictive head) [1,2]. Another strategy could be using Hypernet to directly optimize different local models [3].\n\nAs summarized in recent FL benchmark studies ([9,43] in the paper), the majority of FL works can benefit from local fine-tuning, often leading to state-of-the-art personalized FL results in different datasets and metrics. As our approach aims to improve the fine-tune process, it is widely applicable and complementary to most non-fine-tuning approaches to personalization. This includes the works you mentioned [1,2] as demonstrated by further gains on FedBABU, which likewise learns a global shared encoder + local predictive head, as shown in Table 1. For a more comprehensive positioning of our work as compared to existing personalized FL literature, please refer to Appendix B.\n\nUnfortunately, pFedHN [3], which you mentioned, does not work in practice for larger setups. Note that the authors in [3] used a small model of 5 layers with a maximum of 100 clients as opposed to e.g. our CIFAR10 experiments, full-blown ResNet with 1000 clients. We tried comparing with pFedHN using their official codebase in our setup but the hypernet failed to scale to larger ResNets. This, along with its memory cost, is detailed in Appendix C. \n\n> Q2.2) (line 184) Could you explain what is \"the degree of data heterogeneity of the client’s local data with respect to the pretrained model\"? The data heterogeneity comes from local data and the pretrained model?\n\nSorry for the confusing wording. We learn functions (meta-nets) to generate hyperparameters according to how different the client’s local data to the data that the model is pretrained on. As we do not store the pretraining data, we summarize it in terms of the mean and variance stored in the batch normalization layers of the pretrained model. Hence, there is a notion of data heterogeneity between the client’s data and the BN statistics stored in the pretrained model  (which summarize the pre-trained data). \n\n> Q2.3) (line 205) Should $\\text{BNNet}(\\boldsymbol{w}\\_{bn};\\xi\\_1,\\xi\\_2,...,\\xi\\_{B-1},\\xi\\_{B})$ be $\\text{BNNet}(\\xi\\_1,\\xi\\_2,...,\\xi\\_{B-1},\\xi\\_{B};\\boldsymbol{w}\\_{bn})$? It seems that $\\xi\\_1,\\xi\\_2,...,\\xi\\_{B-1},\\xi\\_{B}$ are BNNet's input.\n\nYes. We adopted the notation of putting the parameters before the input consistently throughout the paper. If you recommend putting the input first, we are happy to change the notation in the revised paper.\n\n> Q2.4) (line 206) Could you explain why there is a Hadamard product for the LRNet's output?\n\nThe learning rate used for fine-tuning is a product of the output of the LRNet and the learnable post multiplier parameter (Eq. 4). This parameter is the starting learning rate, initialized with the best learning-rate from our grid search prior to meta-training (Appendix E). Having this scaling parameter fixed would limit the range of the resulting learning rate as the LRNet has to learn the optimal learning-rate given a fixed number of optimization steps. Making this parameter learnable would better control the potential effect this parameter could have on the learning rate.\n\n> Q2.5) title of section 3.3 is a bit weird. why you are Define Training/Validation Losses for IFT (implicit function theorem). Isn't it that training/validation losses should defined by your tasks?\n\nYes, we agree that the training and validation losses are defined by the task (e.g. cross entropy loss on a classification dataset). However, we had to explicitly modify this loss to make the loss dependent on the learning rate as shown in Eq. 9 & 10, in order to be able to optimize it with IFT. We will revise the title of Section 3.3 to “Adapting the Losses for IFT”.\n\n\n"}, {"review_id": "l6NTwk9gaQ", "reviewer": "Reviewer_wFN9", "summary": "This paper proposes a new personalization strategy for the Federated Learning setting. Their key idea is to train a meta-network that takes as input the client's data and outputs optimal hyperparameters for that data. These optimal hyperparameters can then be used to personalize the pretrained model. The hyperparameters they choose to output control the batch-norm mean, std-dev and per layer learning rate. They evaluate their technique on 4 datasets, and show that their technique outperforms other personalization strategies on these datasets.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The core idea of using a meta-net proposed by the authors is quite novel and I believe it is a very promising direction.\n- The hypergradient technique they propose to train this metanet is also quite novel and interesting.\n- For the most part their algorithm and technique is clearly presented.", "weaknesses": "- The experiment setup is not clearly defined and confusing. Examples\n   - In their CIFAR10 experiments, they say 20% of the data is used for test, it's not clear if that means 20% of the clients are not used in the FL step or 20% of the data in each client is not used in the FL step.\n  - It is not clear what fraction of clients are used for training the FL model in each expt\n  - Figures 2 and 3 are poorly labeled and not clearly explained. Figure 3 is completely unreadable.\n\n- Experimental results are not very strong\n  - FedL2P seems very computationally expensive (total compute used is not compared anywhere) and the results are only marginally better than other techniques.\n  - No evaluation on actual non-IID datasets. There are several non-IID datasets available publicly for FL - EMNIST, Shakespeare, StackOverflow etc. Evaluating their method on these datasets would make the results much more compelling.\n - No evaluation on non-image datasets.\n \n- Hyperparameters limited to continuous values - many hyperparameters like # epochs, choice of optimizer etc. are not continuous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We provide clarifications and evaluate on a speech dataset, including the case where some clients are not used in learning the meta-nets.\n\n> Q3.1) Clarify client/data split.\n\nIn our CIFAR10 experiments, 20% of the train data is used to construct the val set. The resulting train set is used to pretrain the model in a federated setting and both the train and val sets are used to train the meta-nets given the pretrained model.\n\n> Q3.2) Fraction of clients used for training the FL model in each exp?\n\nCIFAR10 (Sec 4.2): all clients are used to pretrain the FL model and the same set of clients is subsequently re-used to learn the meta-nets - this is based on the conventional FL setup of learning global model(s) and then personalizing to each client via finetuning.\n\nCIFAR10-C (Sec 4.3): we used the pretrained model trained using CIFAR10 clients in Sec 4.2 and learned the meta-nets using all CIFAR10-C clients. Hence, the meta-nets are learnt on data which is not seen during the pretraining stage.\n\nOfficeCaltech/DomainNet (Sec 4.4): we used a publicly available ImageNet pretrained model and learned the meta-nets using all clients, showcasing FedL2P’s ability to adapt to different image domains using existing models which may not be pretrained in a FL setting.\n\nUpon reading the reviewer’s comments, we are inspired and included the case where some clients are not included in the learning of the meta-nets in Q3.4.\n\n> Q3.3) Fig. 2 & 3 are unclear.\n\nSorry for the lack of clarity. The figures should be interpreted with the text in L317-353. We will revise the figures to make them clearer and self-contained. \n\nTo summarize the findings:\n\n**Fig. 2** shows the outputs of the BNNet ($\\beta$) LRNet ($\\eta$), and the sparsity (% of params with learning rate 0), across layers and clients for $\\alpha=1000,1.0,0.5,0.1$. \n\n- In CIFAR10 (Fig 2a), BNNet learns to use the pretrained model’s batch norm (BN) statistics ($\\beta=0$) as images come from the same natural image domain as the images used to pretrain the model. When we adapt a natural image pretrained model to other domains, BNNet learns to utilize a mix of pretrained model’s BN statistics and the client’s BN statistics (Fig 2b,c,d). In some cases, it uses solely the client’s BN statistics ($\\beta=1$). Conclusion: BNNet explicitly learns to handle the feature shift in both marginal and conditional distribution skews.\n\n- LRNet learns the layer-wise learning rate, freezing a proportion of the model indicated by the sparsity value. Notably, as standard fine-tuning led to performance degradation in the IID case for CIFAR10 (Fig 2a, $\\alpha=1000$), LRNet learned to freeze the entire model ($\\eta=0$) and hence skip fine-tuning entirely. In contrast, LRNet correctly finetunes the entire model (sparsity=0) when the image domain largely differs from the one used to train the pretrained model (Fig 2c,d).\n\n**Fig. 3** visualizes how close the estimated clusters, clustered by the generated $\\beta$ and $\\eta$, are to the true domain clusters. Each block $(j,k)$ shows the mean Euclidean distance of $\\beta$ and $\\eta$ between all pairs of clients in domain $j$ and $k$, normalized by the within-domain Euclidean distance, hence 0 in log-scale for diagonal blocks. If $\\beta$ and $\\eta$ are similar in two different domains (off-diagonal blocks), then the distance would be close to 0 and the domains are indistinguishable. Likewise, the bigger the distance, the more distinguishable the domains are to their true domains.\n\n> Q3.4) No evaluation on actual non-IID and non-image datasets\n\nWe follow standard evaluation practice [23,58,50] of non-IID splitting existing datasets with controlled class proportion skew p(y), to analyze how performance varies with class heterogeneity (Dirichlet $\\alpha=1000,1.0,0.5,0.1$; Tab 1-4). We already go beyond most existing studies and study two other types of heterogeneity: marginal feature p(x) and concept p(x|y) shift – as described in L102-105 using Office-Caltech and DomainNet. These challenging benchmarks are very much “real” non-IID – see their original papers for details of the substantial distribution shifts involved. Nevertheless, given the reviewer 's suggestion on non-image data, we have added a new speech dataset in our evaluation. Please kindly refer to Q4.4 under our response to Reviewer Jjav.\n\n> Q3.5) FedL2P computationally expensive + marginal performance improvements\n\nIn CIFAR10/SpeechCommands, we trained our meta-nets for 500/100 rounds respectively (vs 500 rounds for pre-training in both cases). Nonetheless, the comms cost required to learn the meta-nets is only 0.38% and 0.24% of the pre-training cost respectively. Thus, there is a modest, but not prohibitive, additional compute cost required to learn the tiny meta-nets; and the comms cost, which is usually the main concern in FL, is marginal at < 1%. \n\nFurthermore, we emphasize that once the meta-nets are trained, the cost is **amortized**; we can use them for as many new unseen clients – as we demonstrated in Q3.4. In the limit of a large no. of new clients the per-client cost of meta-net training tends toward 0. \nOverall FedL2P is also memory-efficient and scalable as compared to existing works (Appendix B & C), has a non-significant overhead compared to hessian-free approaches, and has a marginal additional cost of 4.4% during fine-tuning compared to standard fine-tuning (Appendix C).\n\nWe are not sure why the reviewer thinks the improvement is marginal. For instance, we extend Table 1 by listing the improvement in accuracy over the baselines (Rebuttal PDF Table R2), where the only marginal improvement is the case for FedBABU $\\alpha=0.1$ \n\n> Q3.6) Hyperparameters limited to continuous values\n\nWe chose batch norm and layer-wise selective update hyperparameters as they have been shown [2,34,36,46] to be effective at dealing with feature and label shift respectively. Optimizing discrete hyperparameters is left for future work.\n"}, {"review_id": "xqhImKF1AL", "reviewer": "Reviewer_Jjav", "summary": "\nThis study primarily concentrates on refining hyperparameters in federated learning, especially concerning non-IID data. The researchers highlight two key hyperparameter groups—Batch Normalization (BN) and selective update hyperparameters—that yield impressive outcomes.\n\nBN hyperparameters help mitigate feature shifts, a prevalent problem in federated learning. FedBN, a cited method, aids in managing feature shift more effectively by localizing the BN layers while keeping other layers global. Details of the BN layer formation and its constituents are also given.\n\nMoreover, the study explores a global model's application, initially trained through existing federated algorithms, which is then fine-tuned to the client set. The CIFAR-10 dataset split among various clients is utilized for this process, demonstrating that their approach maintains global performance while enhancing personalized performance.\n\nThe researchers also incorporate the Jacobian product to approximate the Hessian inverse and calculate the hyper gradient, forming a part of their suggested algorithm.\n\nFor experimental purposes, ResNet-18 and SGD are consistently employed for all tests. Batch size, local epoch number, learning rate, hyper gradient clipping, maximum communication rounds, and other parameters are also predefined.\n\nThis paper contributes to federated learning by introducing a method to optimize hyperparameters, especially in non-IID data scenarios, and backs its effectiveness through empirical results.", "questions": "1. Could you elaborate on how generalizable your method is to different federated learning scenarios? For instance, how would your method perform with different data distributions, privacy requirements, and computational resources?\n2. Could you provide a more comprehensive comparison with other state-of-the-art methods? It would be helpful to understand how your method performs relative to these other approaches in terms of accuracy and computational efficiency. I am sill unclear about whether your baselines cover enough range of comparable works. \n3. What particular `pre-trained` models did you use in your experiments? \n4. Additional Datasets and Models: Have you considered testing your method on various datasets with different characteristics (e.g., image, text, tabular data) and different model architectures? \n5. Could you discuss the limitations of your approach and potential ways to overcome these limitations? This would provide a more balanced view of your work and could help guide future research in this area.", "limitations": "In terms of broader societal impacts, it's not clear from the provided excerpts whether the authors have considered this aspect.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:** The paper stands out due to its unique approach to handling non-IID data within federated learning. Implementing Batch Normalization (BN) hyperparameters to tackle feature shift is an inventive fusion of existing deep learning knowledge and federated learning's distinct hurdles. Moreover, the employment of the Jacobian product to estimate the Hessian inverse and compute the hyper gradient is a distinct application within the federated learning sphere, marking an inventive merge of established concepts to address a recurring federated learning issue.\n\n**Quality:** The paper's quality appears robust. It reflects the authors' thoughtful research methodology and the theoretical integrity of their approach to hyperparameter optimization. Leveraging the CIFAR-10 dataset for validation is a norm in this field, and the authors thoroughly outline their experimental setup. Nevertheless, a broader evaluation across diverse datasets and juxtaposition with other methodologies could further enrich the paper's quality.\n\n**Clarity:** The paper is well-crafted and clearly depicts the research. The authors present exhaustive descriptions of their methodology and the theoretical underpinnings of their strategy. They also lucidly detail their experimental setup and findings. Such explicitness in communication renders the paper comprehensible to a broad audience, from novices in federated learning to seasoned researchers.\n\n**Significance:** The paper holds substantial significance. It addresses pivotal challenges, such as dealing with non-IID data and optimizing hyperparameters, vital in federated learning. The authors' solutions to these challenges could potentially lead to the more efficient and practical training of federated learning models, making a notable contribution to the field. Additionally, this paper's insights could stimulate new research pathways and methodologies within the broader realm of machine learning.", "weaknesses": "In summary, while the paper is fundamentally and conceptually solid, there are a few weaknesses that weaken its contribution and soundness. \n\n1. **Dataset and Model Constraints:** The study principally uses the CIFAR-10, Office-Caltech-10, and DomainNet datasets in its experiments. Although these are recognized standards, the findings may not extend to different datasets and models. Testing the proposed method across diverse datasets (such as images, text, and tabular data) and various model architectures could better showcase the method's adaptability and resilience.\n2. **Comparison to Other Techniques:** A broader comparison with cutting-edge methods could enhance the paper. While some algorithms are referenced, the relative performance of the authors' method remains unclear. A side-by-side comparison could clarify the comparative advantages and disadvantages of their approach.\n3. **Computational Overhead:** The authors suggest the Jacobian product for approximating the Hessian inverse and computing the hypergradient, a fresh perspective. Appendix C mentions that the computational cost compared to FO / HF-MAML is still relatively large. \n4. **Applicability:** The paper's significant contribution to federated learning cannot be overlooked, yet the universal applicability of their method warrants examination. Federated learning is applied across various sectors, each with unique data distributions, privacy stipulations, and computational resources. Discussing the limitations and potential solutions of their method would fortify the paper.\n5. **Reproduction**: The codes are not available yet. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed constructive feedback. We further evaluate our approach on a speech dataset, clarify our setups, discuss the position, costs and limitations of our approach.\n\n> Q4.1) Generalizability of our method to different FL scenarios\n\nOur method can handle both label p(y) and feature distribution p(x|y) shifts as defined in L100-105 (eg: distribution shift between amazon product & home webcam images in the Caltech-10 benchmark). Through our experiments, we show that our approach adapts well to various degrees of data heterogeneity by learning which layers to freeze or personalize and determine the extent in which to utilize the client’s or pretrained model’s data statistics, both of which were hand-designed in previous works. Hence, our approach enables per-scenario & per-client personalization learning. Please refer to Appendix B for a summary of our work in comparison with existing FL works. Regarding privacy, our method fulfills the same privacy requirements as FedAvg as we used FedAvg to train our meta-nets. Lastly, we evaluate FedL2P using both 5 & 15 finetune epochs, reflecting different computational budgets, as shown in Table 2 and Table R1 (Rebuttal PDF).\n \n> Q4.2) Could you provide a more comprehensive comparison with other state-of-the-art methods? \n\nIn terms of accuracy, recent comprehensive personalized FL benchmark papers [9,43] have shown that no existing method consistently outperforms other methods in all datasets & metrics. Notably, FedAvg+FT is a high-performing baseline, even achieving state-of-the-art performance in some scenarios. Moreover, these benchmark papers also show that local fine-tuning further improves personalized accuracy for most cases, when applied across a wide range of personalized FL works. As our work aims to improve the fine-tuning process, it is directly applicable to most existing works that aim to learn shared model(s) and we showed a few examples in Table 1. \n\nIn terms of efficiency, in Appendix B, we highlight that our approach only scales linearly with the no. of layers of the pretrained model. Previous works that require the storing of local layers [2,11,14] for each client do not scale as well, e.g. limiting the number of clients in their CIFAR10 experiments to 100 while we evaluate our approach on 1000 clients. In Appendix C, we quantify the memory and compute cost by comparing with a few of these existing works. Notably, we mentioned that the compute cost to compute our hypergradient is **not** a significant overhead compared to FOMAML and HF-MAML. Additionally, the cost of our approach during fine-tuning is only 4.4% more than the standard fine-tuning. Lastly, the communication cost needed to train the meta-nets is 0.38% and 0.24% of the cost needed to pretrain the model in our CIFAR10 and Speech Commands setup respectively.\n\n> Q4.3) Pretrained models used in experiments\n\nSection 4.2 (CIFAR10) follows a conventional FL setup where the model is pretrained using an existing FL approach - we showed three approaches: FedAvg, PerFedAvg, and FedBABU. \n\nSection 4.3 (CIFAR10-C) uses the model pretrained using FedAvg on $\\alpha=1000$ CIFAR10 clients. \n\nSection 4.4 (Caltech-10 and DomainNet) uses a publicly available model pretrained on ImageNet, illustrating the scenario where FedL2P can also be utilized to federatedly learn how to fine-tune a publicly available model.\n\n> Q4.4) Additional Datasets and Models\n\nWe have added a new evaluation on the non-IID Speech Commands V2 dataset[a]. We use the 12-class version that is naturally partitioned by speaker, with one client per speaker. It is naturally imbalanced in skew and frequency, with 2112, 256, 250 clients/speakers for train, val, and test. \n\nWe randomly sampled 250 of 2112 training clients for our seen pool of clients and sampled 50 out of 256+250 validation & test clients for our unseen client pool. Each client’s data is then split 80%/20% for train & test sets, with a further 20% of the resulting train set held out to form a validation set. The pool of 250 seen clients then follows the conventional FL setup described in Sec 4.2: they both pretrain the FL model and subsequently learn the meta-nets. For pretraining, we adopt the setup & hyperparameters from ZeroFL[b], which is trained using FedAvg for 500 rounds. We then use the seen pool of clients to learn the meta-nets for an additional 100 rounds. The learned meta-nets are then used to finetune and evaluate both our seen and unseen pool of clients. \n\nIn Table R1 (rebuttal PDF), we compare with standard finetuning baselines and our method without FL (L2P) for 5 or 15 epochs as described in Sec 4.1.2. Note that L2P is not evaluated on the unseen pool of clients as we only evaluate the meta-nets learned on the seen pool of clients. We see that finetuning the pretrained model on each client’s train set resulted in a significant drop in test set performance. Moreover, as each client represents a different speaker, fine-tuning using the pretrained model’s BN statistics (FT (BN G)) fails entirely. Our FedL2P, on the other hand, led to a significant performance boost on both the seen and unseen client pools, e.g. by utilizing a mix of pretrained model’s & the client’s BN statistics as seen in Fig. R1.\n\n[a] Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition, arXiv 2018\n\n[b] ZeroFL: Efficient On-Device Training for Federated Learning With Local Sparsity, ICLR 2022\n\n\n> Q4.5) Limitations and potential solutions\n\nWe select hyperparameters that are previously shown to be critical to performance - learning rate and batch norm statistics. As mentioned by the reviewer, discrete hyperparameters, such as batch size & local epoch are manually defined. Optimizing discrete hyperparameters are outside of our scope. A possible extension of our work could include these hyperparameters by defining them as categorical distributions. Another limitation is our approach requires additional rounds to learn the meta-nets. \n"}]

Author rebuttal:
We would like to express our appreciation and gratitude to the reviewers for the detailed comments and helpful feedback to further clarify and improve our paper. We will revise the paper accordingly, adding the new evaluations, clarifying ambiguities, and fixing notation & spelling errors. We have attached a Rebuttal PDF, which consists of results on our newly added experiments on the Speech Commands dataset (in response to Reviewer wFN9 Q3.4 and Reviewer Jjav Q4.4.) and an extended Table 1 to highlight the improvement gains of FedL2P compared to standard fine-tuning in response to Reviewer wFN9 Q3.5.

Meta-review: All reviewers found the work interesting and relevant. After the author’s response, the reviewers did not see the need for further clarifications. All reviewers recommend acceptance.
Decision: Accept (poster)