Title: FedL2P: Federated Learning to Personalize
Venue: NeurIPS 2023
Accepted: yes
Keywords: federated learning; meta-learning; hyperparameter optimization
Abstract: Federated learning (FL) research has made progress in developing algorithms for distributed learning of global models, as well as algorithms for local personalization of those common models to the specifics of each client’s local data distribution. However, different FL problems may require different personalization strategies, and it may not even be possible to define an effective one-size-fits-all personalization strategy for all clients: Depending on how similar each client’s optimal predictor is to that of the global model, different personalization strategies may be preferred. In this paper, we consider the federated meta-learning problem of learning personalization strategies. Specifically, we consider meta-nets that induce the batch-norm and learning rate parameters for each client given local data statistics. By learning these meta-nets through FL, we allow the whole FL network to collaborate in learning a customized personalization strategy for each client. Empirical results show that this framework improves on a range of standard hand-crafted personalization baselines in both label and feature shift situations.

FedL2P: Federated Learning to Personalize

Abstract

Federated learning (FL) research has made progress in developing algorithms for distributed learning of global models, as well as algorithms for local personalization of those common models to the specifics of each client's local data distribution. However, different FL problems may require different personalization strategies, and it may not even be possible to define an effective one-size-fits-all personalization strategy for all clients: depending on how similar each client's optimal predictor is to that of the global model, different personalization strategies may be preferred. In this paper, we consider the federated meta-learning problem of learning personalization strategies. Specifically, we consider meta-nets that induce the batch-norm and learning rate parameters for each client given local data statistics. By learning these meta-nets through FL, we allow the whole FL network to collaborate in learning a customized personalization strategy for each client. Empirical results show that this framework improves on a range of standard hand-crafted personalization baselines in both label and feature shift situations. 0

1 Introduction

Federated learning (FL) is an emerging approach to enable privacy-preserving collaborative learning among clients who hold their own data. A major challenge of FL is to learn from differing degrees of statistical data heterogeneity among clients. This makes it hard to reliably learn a global model and also that the global model may perform sub-optimally for each local client. These two issues are often dealt with respectively by developing robust algorithms for learning the global model [33, 32, 39] and then offering each client the opportunity to personalize the global model to its own unique local statistics via fine-tuning. In this paper, we focus on improving the fine-tuning process by learning a personalization strategy for each client.

Avariety of approaches have been proposed for client personalization. Some algorithms directly learn the personalized models [62, 43], but the majority obtain the personalized model after global model learning by fine-tuning techniques such as basic fine-tuning [46, 48], regularised fine-tuning [34, 59], and selective parameter fine-tuning [17, 11, 2, 37]. Recent benchmarks [45, 9] showed that different personalized FL methods suffer from lack of comparable evaluation setups. In particular, dataset- and experiment-specific personalization strategies are often required to achieve state-of-theart performance. Intuitively, different datasets and FL scenarios require different personalization strategies. For example, scenarios with greater or lesser heterogeneity among clients, would imply different strengths of personalization are optimal. Furthermore, exactly how that personalization should be conducted might depend on whether the heterogeneity is primarily in marginal label shift, marginal feature shift, or conditional shift. None of these facets can be well addressed by a one size fits all personalization algorithm. Furthermore, we identify a previously understudied issue: even

for a single federated learning scenario, heterogeneous clients may require different personalization strategies. For example, the optimal personalization strategy will have client-wise dependence on whether that client is more or less similar to the global model in either marginal or conditional data distributions. Existing works that learn personalized strategies through the use of personalized weights are not scalable to larger setups and models [55, 14]. On the other hand, the few studies that attempt to optimize hyperparameters for fine-tuning do not sufficiently address this issue as they either learn a single set of personalization hyperparameters [65, 22] and/or learn a hyperparameter distribution without taking account of the client's data distribution [30].

In this paper, we address the issues above by considering the challenge of federated meta-learning of personalization strategies. Rather than manually defining a personalization strategy as is mainstream in FL [27, 9], we use hyper-gradient meta-learning strategies to efficiently estimate personalized hyperparameters. However, apart from standard centralised meta-learning and hyperparameter optimization (HPO) studies which only need to learn a single set of hyperparameters, we learn meta-nets which inductively map from local client statistics to client-specific personalization hyperparameters. More specifically, our approach FedL2P introduces meta-nets to estimate the extent in which to utilize the client-wise BN statistics as opposed to the global model's BN statistics, as well as to infer layer-wise learning rates given each client's metadata.

Our FedL2P thus enables per-dataset/scenario, as well as per-client, personalization strategy learning. By conducting federated meta-learning of the personalization hyperparameter networks, we simultaneously allow each client to benefit from its own personalization strategy, (e.g., learning rapidly, depending on similarity to the global model), and also enable all the clients to collaborate by learning the overall hyperparameter networks that map local meta-data to local personalization strategy. Our FedL2P generalizes many existing frameworks as special cases, such as FedBN [35], which makes a manual choice to normalize features using the client BN's statistics, and various selective fine-tuning approaches [37, 48, 11, 2], which make manual choices on which layers to personalize.

2 Related Work

Existing FL works aim to tackle the statistical heterogeneity of learning personalized models by either first learning a global model [46, 39, 32, 48, 29] and then fine-tuning it on local data or directly learning the local models, which can often be further personalized using fine-tuning. Many personalized FL approaches include the use of transfer learning between global [56] and local models [44], model regularization [34], Moreau envelopes [59], and meta-learning [15, 10]. Besides algorithmic changes, many works also proposed model decoupling, in which layers are either shared or personalized [2, 11, 64, 48, 37, 17, 54] or client clustering, which assumes a local model for each cluster [42, 44, 53, 7]. These methods often rely on or adopt a fixed personalization policy for local fine-tuning in order to adapt a global model or further improve personalized performance. Although there exists numerous FL approaches that propose adaptable personalization policies [55, 40, 14], these works are memory intensive and do not scale to larger setups and models. On the other hand, our approach has a low memory footprint (Appendix C) and is directly applicable and complementary to existing FL approaches as it aims to solely improve the fine-tuning process.

Another line of work involves HPO for FL (or FL for HPO). These methods either learn one set of hyperparameters for all clients [22, 30, 65] or random sample from learnt hyperparameter categorical distributions which does not take into account of the client's meta-data [30]. Moreover, some of these methods [22, 65] search for a set of hyperparameters based on the local validation loss given the initial set of weights prior to the federated learning of the model [22, 65], which might be an inaccurate proxy to the final performance. Unlike previous works which directly learn hyperparameters, we deploy FL to learn meta-nets that take in, as inputs, the client meta-data to generate personalized hyperparameters for a given pretrained model. A detailed positioning of our work in comparison with existing literature can be found in Appendix B.

3 Proposed Method

3.1 Background & Preliminaries

Centralized FL. A typical centralized FL setup using FedAvg [46] involves training a global model θ g from C clients whose data are kept private. At round t , θ t -1 g is broadcast to a subset of clients selected, ˜ C , using a fraction ratio r . Each selected client, i ∈ ˜ C , would then update the model using

Global

Model

𝜃

!

(Pre-trained)

Server

(FedAvg on

𝜆

)

Global

Model

𝜃

!

Personalized

Model

𝜃

"

∗

Personalization

Strategy

𝜆

(Meta-net)

Client

Meta-

Data

Fine-Tune

Client

𝑖

Input

Meta-gradient

Personalization

Strategy

𝜆

∗

(Meta-net)

Client

Data

Output

New Client

Meta-Data

New

Client

Data

Global

Model

𝜃

!

New

Personalized

Model

𝜃

"

∗

Fine-Tune

(1) FL of Personalization

Strategy on known clients

(2) Deploy

Personalization

Strategy for new client

a set of hyperparameters and its own data samples, which are drawn from its local distribution P i defined over X ×Y for a compact space X and a label space Y for e epochs. After which, the learned local models, θ t -1 i , are sent back to the server for aggregation θ t g = ∑ ˜ C i N i ∑ i ′ N i ′ θ t -1 i where N i is the total number of local data samples in client i and the resulting θ t g is used for the next round. The aim of FL is either to minimize the global objective E ( x,y ) ∼ P L ( θ g ; x, y ) for the global data distribution P or local objective E ( x,y ) ∼ P i L i ( θ i ; x, y ) for all i ∈ C where L i ( θ ; x, y ) is the loss given the model parameters θ at data point ( x, y ) . As fine-tuning is the dominant approach to either personalize from a high-performing θ g or to optimize θ i further for each client, achieving competitive or state-of-the-art results in recent benchmarks [45, 9], we focus on the collaborative learning of meta-nets which generates a personalized set of hyperparameters used during fine-tuning to further improve personalized performance without compromising global performance.

Non-IID Problem Setup. Unlike many previous works, our method aims to handle both common label and feature distribution shift across clients. Specifically, given features x and labels y , we can rewrite the joint probability P i ( x, y ) as P i ( x | y ) P i ( y ) and P i ( y | x ) P i ( x ) following [28]. We focus on three data heterogeneity settings found in many realistic settings: both label & distribution skew in which the marginal distributions P i ( y ) & P i ( x ) may vary across clients, respectively, and concept drift in which the conditional distribution P i ( x | y ) may vary across clients.

3.2 FedL2P: FL of Personalization Strategies

We now present our proposed method, FedL2P, to tackle collaborative learning of personalization strategies under data heterogeneity. Our main motivation is that the choice of the hyperparameters for the personalized fine-tuning, such as the learning rates and feature mean and standard deviations (SD) statistics of the batch normalization [26] (BN) layers, is very crucial. Although existing FL-HPO approaches aim to learn these hyperparameters directly 2 [22, 65, 30], we aim to do it in a meta learning or hypernetwork-like fashion: learning a neural network (dubbed meta-net ) that takes certain profiles of client data (e.g., summary statistics of personalized data) as input and outputs near-optimal hyperparameters. The meta-net is learned collaboratively in an FL manner without sharing local data. The returned hyperparameters from the meta-net are then deployed in client's personalized fine-tuning. The main advantage of this meta-net approach over the direct HPO is that a brand new client needs not do time-consuming HPO, but just gets their optimal hyperpameters by a single feed-forward pass through the meta-net. Our idea is visualized in Fig. 1. For each FL round, the latest meta-net is distributed to the participating clients; each client then performs meta learning to update the meta-net; the local updated meta-nets are sent to the server for aggregation. Details of our algorithm are described below.

Hyperparameter Selection. There is a wide range of local training hyperparameters that can be optimized, some of which were explored in previous FL works [22, 65, 30]. As local training is often a costly process, we narrowed it down to two main sets of hyperparameters based on previous works that showed promising results in dealing with non-IID data: BN hyperparameters and selective update hyperparameters .

Batch Normalization Hyperparameters. The first set of hyperparameters involves BN and explicitly deals with feature shift. Notably, FedBN [35] proposed keeping the BN layers local and the other layers global to better handle feature shift across clients; BN has found success at mitigating domain

shifts in various domain adaptation tasks [36, 8] and can be formulated as follows:

where g is a BN layer, x is its input features, ( ˆ µ , ˆ σ 2 ) are the estimated running mean, variance, and ϵ is used for numerical stability. During training, the batch statistics E ( x ) and V ar ( x ) are used instead, keeping running estimates of them, ˆ µ and ˆ σ 2 . During inference, these estimates are used 3 . Both γ and δ are learnable parameters used to scale and shift the normalized features.

Although deploying local BN layers is useful to counteract the drawbacks of feature shift, using global BN layers is often beneficial for local fine-tuning in cases where the feature shift is minimal as it helps speed-up convergence and reduces overfitting. Hence, we propose learning a hyperparameter β for each BN layer as follows:

where ˆ µ pt and ˆ σ 2 pt are the estimated running mean and variance by the given pretrained model; ˆ µ i and ˆ σ 2 i are the running mean and variance estimated by the client. When β → 0 , the client solely uses pretrained model's BN statistics and when β → 1 , it uses its local statistics to normalize features. Thus β indicates the degree in which the client should utilize its own BN statistics against pretrained model's, to handle the feature shift in its local data distribution.

Selective Update Hyperparameters. A variety of recent personalized FL works achieved promising results by manually selecting and fine-tuning a sub-module of θ g during personalization [48, 2, 37], ( e.g. , the feature extractor or the classifier layers), while leaving the remaining modules in θ g frozen. It is beneficial because it allows the designer to manage over- vs under-fitting in personalization. e.g. , if the per-client dataset is small, then fine-tuning many parameters can easily lead to overfitting, and thus better freezing some layers during personalization. Alternatively, if the clients differ significantly from the global model and/or if the per-client dataset is larger, then more layers could be beneficially personalised without underfitting. Clearly the optimal configuration for allowing model updates depends on the specific scenario and the specific client. Furthermore, it may be beneficial to consider a more flexible range of hyperparameters that control a continuous degree of fine-tuning strength, rather than a binary frozen/updated decision per module.

To automate the search for good personalization strategies covering a range of wider and more challenging non-IID setups, we consider layer-wise learning rates, η for all learnable weights and biases. This parameterization of personalization encompasses all the previous manual frozen/updated split approaches as special cases. Furthermore, while these approaches have primarily considered heterogeneity in the marginal label distribution, we also aim to cover feature distribution shift between clients. Thus we also include the learn-

Algorithm 1 FedL2P: FL of meta-nets for Personalization Hyperparameters

Input: Pretrained global model θ g with M layers and B BN layers, each has J input channels. Fraction ratio r . Total no. of clients C . No. of update iterations K . Training loss L T and validation loss L V . ζ is the learning rate for λ .

1: initialize λ = { w bn , w lr , ˜ η }

2: for round t = 1 , . . . , T do

3: ˜ C ← Random sample Cr clients

4: for client i ∈ ˜ C do

5: send θ g , λ to client

6: Forward pass of local dataset to compute 1) E ( x m ) , SD ( x m ) for 1 ≤ m ≤ M -1 2) µ b,j , σ 2 b,j for b = 1 , . . . , B and j = 1 , . . . , J .

7:

Compute

ξ

b

for

b

= 1

, . . . , B

using Eq. 3

8: for iteration k = 1 , . . . , K do

9: θ i ← Finetune θ g using L T for e epochs

10: λ ← λ -ζ Hypergradient ( L V , L T , λ , θ i )

11: end for

12: send λ and num of data samples N to server

13: end for

14: λ ← ∑ ˜ C i N i ∑ N i λ i

15: end for

Output: λ

Algorithm 2 Hypergradient

Input: Validation Loss L V and training Loss L T . Learning rate ψ and no. of iterations Q . Fixed point ( λ ′ , θ ∗ ( λ ′ )) .

1: p = v = ∂ θ L V | ( λ ′ ,θ ∗ ( λ ′ ))

2: for iteration 1 , . . . , Q do

4: p ← p + v

5: end for

Output: ∂ λ L V | ( λ ′ ,θ ∗ ( λ ′ )) -p ∂ λθ L T | ( λ ′ ,θ ∗ ( λ ′ ))

ing rates for the BN parameters, γ and δ , allowing us to further tackle feature shift by adjusting the means and SD of the normalized features.

Hyperparameter Inference for Personalization. Weaim to estimate a set of local hyperparameters that can best personalize the pretrained model for each client given a group of clients whose data might not be used for pretraining. To accomplish this, we learn to estimate hyperparameters based on the degree of data heterogeneity of the client's local data with respect to the data that the model is pretrained on. There are many ways to quantify data heterogeneity, such as utilizing the earth mover's distance between the client's data distribution and the population distribution for label distribution skew [63] or taking the difference in local covariance among clients for feature shift [35]. In our case, we aim to distinguish both label and feature data heterogeneity across clients. To this end, we utilize the client's local input features to each layer with respect to the given pretrained model. Given a pretrained model with M layers and B BN layers, we learn η = η 1 , . . . , η 2 M 4 and β = β 1 , . . . , β B , using two functions, each of which is parameterized as a multilayer perceptron (MLP), named meta-net, with one hidden layer due to its ability to theoretically approximate almost any continuous function [12, 57]. We named the meta-net that estimates β and the meta-net that estimates η as BNNet and LRNet respectively. Details about the architecture can be found in Appendix. E.

To estimate β , we first perform a forward pass of the local dataset on the given pretrained model, computing the mean and SD of each channel of each input feature for each BN layer. We then measure the distance between the local feature distributions and the pretrained model's running estimated feature distributions of the b -th BN layer as follows:

where P j = N ( µ i,b,j , σ 2 i,b,j ) , Q j = N (ˆ µ pt,b,j , ˆ σ 2 pt,b,j ) , D KL is the KL divergence and J is the number of channels of the input feature. ξ is then used as an input to BNNet, which learns to estimate β as shown in Eq. 4.

Similarly, we compute the mean and SD of each input feature per layer by performing a forward pass of the local dataset on the pretrained model and use it as an input to LRNet. Following best practices from previous non-FL hyperparameter optimization works [49, 3], we use a learnable post-multiplier ˜ η = ˜ η 1 , . . . , ˜ η 2 M to avoid limiting the range of the resulting learning rates (Eq 4).

where ⊙ is the Hadamard product, x m -1 is the input feature to the m -th layer, and w bn and w lr are the parameters of BNNet and LRNet respectively. β is used to compute the running mean and variance in the forward pass for each BN layer as shown in Eq. 2 and η is used as the learning rate for each weight and bias in the backward pass. We do not restrict ˜ η to be positive as the optimal learning rate might be negative [5].

Federated Hyperparameter Learning. We deploy FedAvg [46] to federatedly learn a set of clientspecific personalization strategies. Specifically, we learn the common meta-net λ = { w bn , w lr , ˜ η } that generates client-wise personalization hyperparameters { β i , η i } , such that a group of clients can better adapt a pre-trained model θ g by fine-tuning to their local data distribution. So we solve:

where θ ∗ i is the set of optimal personalized model parameters after fine-tuning θ g for e epochs on the local dataset, L i,V ( θ, λ ) = E ( x,y ) ∼ V i L i ( θ, λ ; x, y ) and V i is the validation set (samples from P i ) for the client i - similarity for L i,T .

For each client i , the validation loss gradient with respect to λ , known as the hypergradient, can be computed as follows:

To compute ∂ λ θ ∗ in Eq. 6, we use the implicit function theorem (IFT):

̸

The full derivation is shown in Appendix A. We use Neumann approximation and efficient vectorJacobian product as proposed by Lorraine et al. [38] to approximate the Hessian inverse in Eq. 7 and compute the hypergradient, which is further summarized in Algorithm 2. In practice, θ ∗ is approximated by fine-tuning θ g on L T using the client's dataset. Note that unlike in many previous works [38, 47] where ∂ λ L V is often 0 as the hyperparameters often do not directly affect the validation loss, in our case ∂ w bn L V = 0 .

Algorithm 1 summarizes FedL2P. Given a pretrained model and a new group of clients to personalize, we first initialize λ (line 1). For every FL round, we sample Cr clients and send both the parameters of the pretrained model and λ to each client (lines 3-5). Each client then performs a forward pass of their local dataset to compute the mean (E) and standard deviation (SD) of the input features to each layer and the statistical distance between the local feature distributions and the pretrained model's running estimated feature distributions for each BN layer (lines 6-7). λ is then trained for K iterations; each iteration optimizes the pretrained model on L T for e epochs, applying β and η computed using Eq. 4 (lines 8-9) at every forward and backward pass respectively. Each client then computes the hypergradient of λ as per Algorithm. 2 and update λ at the end of every iteration (line 10). Finally, after K iterations, each client sends back the updated λ and its number of data samples, which is used for aggregation using FedAvg [46] (lines 12-14). The resulting λ is then used for personalization: each client finetunes the model using its training set and evaluates it using its test set.

3.3 Adapting the Losses for IFT

In the IFT, we solve the following problem:

For the current λ , we first find θ ∗ ( λ ) in (8) by performing several SGD steps with the training loss L T . Once θ ∗ ( λ ) is obtained, we can compute the hypergradient d λ L V ( θ ∗ ( λ ) , λ ) by the IFT, which is used for updating λ . As described in (6) and (7), this hypergradient requires ∂ λ L T ( θ, λ ) , implying that the training loss has to be explicitly dependent on the hyperparameter λ . As alluded in Lorraine et al. [38], it is usually not straightforward to optimize the learning rate hyperparameter via the IFT, mainly due to the difficulty of expressing the dependency of the training loss on learning rates. To address this issue, we define the training loss as follows:

Here f θ, β ( x ) indicates the forward pass with network weights θ and the batch norm statistics β , and CE () is the cross-entropy loss. Note that in (10), we can take several (not just one) gradient update steps to obtain θ ′ . Now, we can see that L T ( θ, λ ) defined as above has explicit dependency on the learning rates η ( λ ) . Interestingly, the stationary point θ ∗ ( λ ) of L T ( θ, λ ) coincides with θ ′ , that is, θ ∗ ( λ ) = θ ′ , which allows for a single instance of inner-loop iterations as Line 9 in Alg. 1. Finally, the validation loss is defined as:

showing clear dependency on BNNet parameters through β ( λ ) as discussed in the previous section.

4 Evaluation

4.1 Experimental Setup

Experiments are conducted on image classification tasks of different complexity. We use ResNet-18 [20] for all experiments and SGD for all optimizers. All details of the pretrained models can be found in Appendix. D. Additionally, the batch size is set to 32 and the number of local epochs, e , is set to 15 unless stated otherwise. The learning rate ( ζ ) for λ = { w bn , w lr , ˜ η } is set to { 10 -3 , 10 -3 , 10 -4 } , respectively. The hypergradient is clipped by value

[ -1 , 1] , Q = 3 , and ψ = 0 . 1 in Alg. 2. The maximum number of communication rounds is set to 500 , and over the rounds we save the λ value that leads to the lowest validation loss, averaged over the participating clients, as the final learned λ . The fraction ratio r =0 . 1 unless stated otherwise, sampling 10% of the total number of clients per FL round. We focus on non-IID labels and feature shifts while assuming that each client has an equal number of samples. Finally, to generate heterogeneity in label distributions, we follow the latent Dirichlet allocation (LDA) partition method [24, 61, 52]: y ∼ Dir ( α ) for each client. Hence, the degree of heterogeneity in label distributions is controlled by α ; as α decreases, the label non-IIDness increases, and vice versa.

4.1.1 Datasets

CIFAR10 [31]. A widely-used image classification dataset, also popular as an FL benchmark. The number of clients C is set to 1000 and 20% of the training data is used for validation.

CIFAR-10-C [21]. The test split of the CIFAR10 dataset is corrupted with common corruptions. We used 10 corruption types 5 with severity level 3 and split each corruption dataset into 80% / 20% training/test sets. A subset ( 20% ) of the train set is further held out to form a validation set. Each corruption type is partitioned by Dir ( α ) among 25 clients, hence C = 250 .

Office-Caltech-10 [19] & DomainNet [51]. These datasets are specifically designed to contain several domains that exhibit different feature shifts. We set the no. of samples of each domain to be the smallest of the domains, random sampling the larger domains. For Caltech-10, we set r = 1 . 0 and C = 4 , one for each domain and thus partitioned labels are IID. For DomainNet, we set C = 150 , of which each of the 6 domain datasets are partitioned by Dir ( α ) among 25 clients, resulting in a challenging setup with both feature & label shifts. Following FedBN [35], we take the top 10 most commonly used classes in DomainNet.

Speech Commands V2 [60]. We use the 12-class version that is naturally partitioned by speaker, with one client per speaker. It is naturally imbalanced in skew and frequency, with 2112, 256, 250 clients/speakers for train, validation, and test. We sampled 250 of 2112 training clients with the most data for our seen pool of clients and sampled 50 out of 256+250 validation and test clients for our unseen client pool. Each client's data is then split 80%/20% for train and test sets, with a further 20% of the resulting train set held out to form a validation set.

4.1.2 Baselines

We run all experiments three times and report the mean and SD of the test accuracy. We consider three different setups of fine-tuning in our experiments as below. As basic fine-tuning (FT) is used ubiquitously in FL, our experiments directly compare FT with FedL2P in different non-IID scenarios. We are agnostic to the FL method used to obtain the global model or to train the meta-nets.

FT (BN C). Client BN Statistics. Equivalent to setting β = 1 in Eq. 2, thus using client statistics to normalize features during fine-tuning. This is similar to FedBN [35], and is adopted by FL works such as Ditto [34] and PerFedAvg [15].

FT (BN G). Equivalent to setting β = 0 in Eq. 2. BN layers use the pretrained global model's BN statistics to normalize its features during fine-tuning.

FT (BN I). BN layers use the incoming feature batch statistics to normalize its features during fine-tuning. This setting is adopted by FL works such as FedBABU [48].

L2P. Our proposed method without FL; λ is learnt locally given the same compute budget before being used for personalization. Hence, L2P does client-wise HPO independently.

4.2 Experiments on Marginal Label Shift

We evaluate our approach based on the conventional setup of personalized FL approaches [9, 45], where a global model, θ g , is first learned federatedly using existing algorithms and then personalized

to the same set of clients via fine-tuning. To this end, given the CIFAR-10 dataset partitioned among a group of clients, we pretrained θ g following best practices from [23] using FedAvg and finetune it on the same set of clients. Table 2 shows the personalized accuracy of the various fine-tuning baselines (Section 4.1.2) and FedL2P using e = 5&15 local epochs on groups of clients with varying label distribution, P i ( y ) ; α = 1000 represents the IID case and α = 1 . 0 , 0 . 5 , 0 . 1 represents more heterogeneous case. As observed in many previous works [48, 27], increasing label heterogeneity would result in a better initial global model at a expense of personalized performance. Our method instead retains the initial global performance and focuses on improving personalized performance.

We also show that in many cases, especially for clients with limited local compute budget e = 5 , utilizing the pretrained model's BN statistics result ( BN G ) can be more beneficial as CIFAR-10 consists of images from the same natural image domain; in contrast, previous works mainly use either the client's BN statistics ( BN C ) or the incoming feature batch statistics ( BN I ) to normalize the features. This strategy is discovered by FedL2P, as illustrated in Fig. 2a where the learned β is 0 for all BN layers of all clients. For the IID case in particular, FedL2P learns a sparsity 6 of 1 . 0 , learning rate η = 0 , for all layers in all clients, forgoing fine-tuning and using the initial global model as the personalized model. For α = 1 . 0 and 0 . 1 , FedL2P learns highly sparse models similar to recent works that proposed fine-tuning only a subset of hand-picked layers [48, 17, 11, 2, 37] to obtain performance gains. Lastly, L2P performs worse than some standard fine-tuning baselines as it meta-overfits on each client's validation set, highlighting the benefits of FL over local HPO.

FedL2P's Complementability with previous FL works. As our proposed FedL2P learns to improve the FT process, it is complementary, not competing, with other FL methods that learn shared model(s). Hence, besides FedAvg, we utilize FedL2P to better personalize θ g pretrained using PerFedAvg(HF) [15] and FedBABU [48] as shown in Table. 1, where we compare FedL2P against the most commonly used FT approach, BN C . Our results show that applying FedL2P to all three FL methods can lead to further gains, in most cases outperforming FT in each respective method. This performance improvement can also bridge the performance gap between different methods. For instance, while FedAvg+FT has worse performance than FedBABU+FT in all cases, FedAvg+FedL2P obtained comparable or better performance than FedBABU+FT for α = 1000 & 0 . 1 .

1000

1.0

0.5

0.1

0.04

0.02

0.00

0.02

0.04

1000

1.0

0.5

0.1

0.000

0.002

0.004

0.006

0.008

1000

1.0

0.5

0.1

0.4

0.6

0.8

1.0

Sparsity

(a) CIFAR10 (

e

= 15

)

1000

1.0

0.5

0.1

0.0

0.2

0.4

0.6

0.8

1.0

1000

1.0

0.5

0.1

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

1000

1.0

0.5

0.1

0.2

0.4

0.6

0.8

Sparsity

1000

1.0

0.5

0.1

0.0

0.2

0.4

0.6

0.8

1.0

1000

1.0

0.5

0.1

0.0025

0.0050

0.0075

0.0100

0.0125

1000

1.0

0.5

0.1

0.04

0.02

0.00

0.02

0.04

Sparsity

(c) DomainNet

1000

0.0

0.2

0.4

0.6

0.8

1.0

1000

0.0050

0.0075

0.0100

0.0125

0.0150

1000

0.04

0.02

0.00

0.02

0.04

Sparsity

(d) Office-Caltech-10

4.3 Personalizing to Unseen Clients

Unseen during Pretraining. We evaluate the performance of FedL2P on CIFAR-10-C starting from the pretrained model trained using FedAvg on CIFAR-10 in the IID setting α = 1000 (Section. 4.2) as

shown in Table. 3. These new clients, whose data is partitioned from CIFAR-10-C, did not participate in the pretraining of the global model and, instead, only participate in the training of the meta-nets. As added noise would impact the feature shift among clients, Fig. 2b shows FedL2P learns to use β = 1 as compared to sole use of β = 0 in the CIFAR-10 case (Fig. 2a); some clients benefit more by using its own BN statistics. Hence, the performance gap for non-IID cases between BN C and BN G is significantly smaller as compared to CIFAR-10 shown in Table. 2.

Unseen during Learning of Meta-nets. We evaluate the case where some clients are not included in the learning of the meta-nets through our experiments on the Speech Commands dataset. We first pretrain the model using FedAvg and then train the meta-nets using our seen pool of clients. The learned meta-nets are then used to fine-tune and evaluate both our seen and unseen pool of clients as shown in Table. 4. Note that L2P is not evaluated on the unseen pool of clients as we only evaluate the meta-nets learned on the seen pool of clients. We see that fine-tuning the pretrained model on each client's train set resulted in a significant drop in test set performance. Moreover, as each client represents a different speaker, fine-tuning using the pretrained model's BN statistics ( BN G ) fails entirely. FedL2P, on the other hand, led to a significant performance boost on both the seen and unseen client pools, e.g. by utilizing a mix of pretrained model's BN statistics and the client's BN statistics shown in Appendix F.

4.4 Personalizing to Different Domains

We evaluate our method on Office-Caltech-10 and DomainNet datasets commonly used in domain generalization/adaptation, which exhibit both marginal and conditional feature shifts. Differing from the conventional FL setup, we adopted a pretrained model trained using ImageNet [13] and attached a prediction head as our global model. Similar to CIFAR-10-C experiments, we compare our method with the base-

lines in Table. 3. Unlike CIFAR-10/10-C, Office-Caltech-10 and DomainNet consist of images from distinct domains and adapting to these domain result in a model sparsity of 0 as shown in Fig. 2c and Fig. 2d. Hence, a better personalization performance is achieved using the client's own local BN statistics ( BN C ) or the incoming batch statistics ( BN I ) than the pretrained model's ( BN G ). Lastly, each client's personalized model uses an extent of the pretrained natural image statistics as well as its own domain statistics, for both datasets.

Further Analysis To further investigate how well the hyperparameters returned by our FedL2Ptrained meta-nets capture the domain-specific information, we analyze the clustering of local features, namely ξ = ( ξ 1 , . . . , ξ B ) and x = ( E ( x 0 ) , SD ( x 0 ) , . . . , E ( x M -1 ) , SD ( x M -1 )) (i.e., the inputs to the meta-nets), and the resulting respective hyperparameters, β and η (i.e., outputs of the metanets), among clients. Specifically, we compute the similarity matrix using the Euclidean distance between the features/hyperparameters of any two clients and perform spectral clustering 7 , using the

amazon

caltech

dslr

webcam

amazon

caltech

dslr

webcam

0.0

9.39

10.04

9.51

9.39

0.0

10.46

10.15

10.04

10.46

0.0

9.13

9.51

10.15

9.13

0.0

amazon

caltech

dslr

webcam

amazon

caltech

dslr

webcam

0.0

0.69

1.79

1.39

0.69

0.0

1.61

1.39

1.79

1.61

0.0

1.1

1.39

1.39

1.1

0.0

0

2

4

6

8

10

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

(a) Office-Caltech-10

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.66

1.69

3.54

1.59

0.39

0.66

0.0

1.48

3.86

1.36

1.26

1.69

1.48

0.0

4.05

0.03

2.01

3.54

3.86

4.05

-0.0

3.96

3.55

1.59

1.36

0.03

3.96

0.0

1.91

0.39

1.26

2.01

3.55

1.91

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.59

1.27

1.25

1.22

0.59

0.59

-0.0

1.06

1.88

1.04

0.69

1.27

1.06

0.0

2.18

0.11

1.32

1.25

1.88

2.18

0.0

2.11

1.67

1.22

1.04

0.11

2.11

0.0

1.29

0.59

0.69

1.32

1.67

1.29

-0.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

(b) DomainNet (

α

= 0

.

5

)

B

n

discretization approach proposed in [58] to assign labels on the normalized Laplacian embedding. We then compute the Adjusted Rand Index (ARI) [25] between the estimated clusters and the clusters partitioned by the different data domains as shown in Table. 6. We also visualize the alignment between the estimated clusters and the true domains in Fig. 3, where each block ( j, k ) represents the normalized average Euclidean distance between all pairs of clients in domain j and k . Specifically, we divide the mean distance between domain j and k by the within-domain mean distances and take the log scale for better visualization: log( d ( j,k ) √ d ( j,j ) √ d ( k,k ) ) where d ( j, k ) is the mean Euclidean distance between j and k . Thus, a random clustering has distance close to 0 and > 0 indicates better alignment between the clustering and the true domains.

As shown, for DomainNet, both BNNet and LRNet consistently preserve the cluster information found in their inputs, ξ & x , respectively. However, perfect clustering is not achieved due to the inherent difficulty. For instance, the real and painting domains share similar features, resulting in similar hyperparameters; the cross-domain distance between real and painting is ∼ 0 in log-scale in Fig. 3 and hence indistinguishable from their true domains. In contrast, the clients' features and resulting hyperparameters of the Office-Caltech-10 dataset are perfectly clustered (ARI=1) as visualized in Fig. 3a and Appendix. F.

4.5 Ablation Study

To elucidate the individual impact of BNNet & LRNet, we run an ablation study of all of the datasets used in our experiments and present the results in Table. 5, where CIFAR10 adopts the pretrained model trained using FedAvg. As BNNet learns to weight between client's BN statistics ( BN C ) and pretrained model's BN statistics ( BN G) , running FedL2P with BNNet alone leads to either better or similar performance to the better performing baseline. Running LRNet as a standalone, on the other hand, can result in further gains, surpassing the use of both BNNet and LRNet on some benchmarks. Nonetheless, it requires prior knowledge of the data feature distribution of the client in order to set a suitable β , of which β = 1 uses BN C and β = 0 uses BN G . Our approach assumes no knowledge of the client's data and learns an estimated β per-scenario and per-client using BNNet.

5 Conclusion

In this paper, we propose FedL2P, a framework for federated learning of personalization strategies specific to individual FL scenarios and datasets as well as individual clients. We learned meta-nets that use clients' local data statistics relative to the pretrained model, to generate hyperparameters that explicitly target the normalization, scaling, and shifting of features as well as layer-wise parameter selection to mitigate the detrimental impacts of both marginal and conditional feature shift and marginal label shift, significantly boosting personalized performance. This framework is complementary to existing FL works that learn shared model(s) and can discover many previous hand-designed heuristics for sparse layer updates and BN parameter selection as special cases, and learns to apply them where appropriate according to the specific scenario for each specific client. As a future work, our approach can be extended to include other hyperparameters and model other forms of heterogeneity, e.g. using the number of samples as an expert input feature to a meta-net.

References

Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers, 2019.

Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Metalearning with adaptive hyperparameters. Advances in Neural Information Processing Systems , 33:20755-20765, 2020.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Alberto Bernacchia. Meta-learning with negative learning rates. In International Conference on Learning Representations , 2021.

Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas D Lane. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390 , 2020.

Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2020.

Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domainspecific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR) , pages 7354-7362, 2019.

Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding. pfl-bench: A comprehensive benchmark for personalized federated learning. Advances in Neural Information Processing Systems , 2022.

Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.07876 , 2018.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning . PMLR, 2021.

Balázs Csanád Csáji et al. Approximation with artificial neural networks. Faculty of Sciences, Etvs Lornd University, Hungary , 24(48):7, 2001.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255. IEEE, 2009.

Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461 , 2020.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems , 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning . PMLR, 2017.

Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems , 2020.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 2066-2073. IEEE, 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations , 2019.

Stephanie Holly, Thomas Hiessl, Safoura Rezapour Lakani, Daniel Schall, Clemens Heitzinger, and Jana Kemnitz. Evaluation of hyperparameter-optimization approaches in an industrial federated learning system. In Data Science-Analytics and Applications . Springer, 2022.

Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos I Venieris, and Nicholas D Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. arXiv preprint arXiv:2102.13451 , 2021.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification , 2(1): 193-218, 1985.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. PMLR, 2015.

Yihan Jiang, Jakub Koneˇ cn` y, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488 , 2019.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning , 14(1-2):1-210, 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning . PMLR, 2020.

Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florina F Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight-sharing. Advances in Neural Information Processing Systems , 34, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429-450, 2020.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning . PMLR, 2021.

Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 , 2021.

Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 , 2016.

Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 , 2020.

Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics , pages 1540-1552. PMLR, 2020.

Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems , 2021.

Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https://github.com/pytorch/vision , 2016.

Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619 , 2020.

Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems , 2021.

Koji Matsuda, Yuya Sasaki, Chuan Xiao, and Makoto Onizuka. Fedme: Federated learning via model exchange. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM) . SIAM, 2022.

Koji Matsuda, Yuya Sasaki, Chuan Xiao, and Makoto Onizuka. An empirical study of personalized federated learning. arXiv preprint arXiv:2206.13190 , 2022.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics . PMLR, 2017.

Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. In International Conference on Learning Representations , 2021.

Jaehoon Oh, Sangmook Kim, and Se-Young Yun. Fedbabu: Towards enhanced representation for federated image classification. arXiv preprint arXiv:2106.06042 , 2021.

Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems , 31, 2018.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1406-1415, 2019.

Xinchi Qiu, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. In International Conference on Learning Representations , 2021.

Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems , 2020.

Mehdi Setayesh, Xiaoxiao Li, and Vincent WS Wong. Perfedmask: Personalized federated learning with optimized masking vectors. In The Eleventh International Conference on Learning Representations .

Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning , 2021.

Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXiv:2006.16765 , 2020.

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Metaweight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems , 32, 2019.

X Yu Stella and Jianbo Shi. Multiclass spectral clustering. In IEEE International Conference on Computer Vision , volume 2, pages 313-313. IEEE Computer Society, 2003.

Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems , 33:21394-21405, 2020.

Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209 , 2018.

Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning , pages 7252-7261. PMLR, 2019.

Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations , 2021.

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 , 2018.

Aoxiao Zhong, Hao He, Zhaolin Ren, Na Li, and Quanzheng Li. Feddar: Federated domainaware representation learning. In International Conference on Learning Representations , 2023.

Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. Single-shot general hyper-parameter optimization for federated learning. In The Eleventh International Conference on Learning Representations , 2023.

A Derivation of Equation (7)

From Eq (5), we know that:

Appendix

Based on the implicit functional theorem (IFT), we get that if we have a function F ( x, y ) = c , we can derive that y ′ ( x ) = -F x /F y . Therefore, plus the Eq (11) into the theorem, we can get:

B Positioning of FedL2P.

Table 7 shows the positioning of FedL2P against existing literature. Note that this list is by no means exhaustive but representative to highlight the position of our work. Most existing approaches obtained personalized models using a personalized policy and local data, often through a finetuning-based approach. This personalized policy can either 1) be fixed, e.g. hand-crafting hyperparameters, layers to freeze, selecting number of mixture components, number of clusters or 2) learned, e.g. learning a hypernetwork to generate weights or meta-nets to generate hyperparameters. These approaches are also grouped based on whether this personalized policy is dependent on the local data during inference, e.g. meta-nets require local client meta-data to generate hyperparameters.

In order to adapt to per-dataset per-client scenarios, many works rely on storing per-client personalized layers, which are trained only on each client's local data. Unfortunately, the memory cost of storing these models scales with the number of clients, C, restricting previous works to small scale experiments. We show that these works are impractical in our CIFAR-10 setup of 1000 clients in Appendix. C. Moreover, most existing methods rely on a fixed personalized policy, such as deriving shared global hyperparameters for all clients in FLoRA, or they do not dependent on local data, such as FedEx which randomly samples per-client hyperparameters from learned categorical distributions. Hence, these methods do not adapt as well to per-dataset per-client scenarios and are ineffective at targeting both label and feature shifts. Lastly, although pFedHN and pFedLA target both label and feature shift cases, they do not scale well to our experiments as shown in Appendix. C.

Most importantly, all existing FL approaches shown in Table 7 can use finetuning either to personalize shared global model(s) or as a complementary personalization strategy to further adapt their personalized models. Since our proposed FedL2P focus on better personalizing shared global model(s) by learning better a personalized policy which leverages the clients' local data statistics relative to the given pretrain model(s), our approach is complementary to all existing FL solutions that learn shared model(s); we showed improvements over a few of these works in Table 1.

Use Cases of FedL2P Mainstream FL focuses on training from scratch, but we focus on federated learning of a strategy to adapt an existing pre-trained model (whether obtained by FL or not) on an unseen group of clients with heterogeneous data. There are several scenarios where this setup and our solution would be useful:

Scenarios where it's expensive to train from scratch for a new group of clients, e.g. adopting FedEx [30] from scratch for a new group of clients would require thousands of rounds to retrain the model and HP while our method takes hundreds to learn two tiny meta-nets (Appendix. C).

Scenarios where there is a publicly available pre-trained foundation model that can be exploited. This is illustrated in Section 4.4 where we adapt a publicly available pretrained model trained using ImageNet on domain generalization datasets.

Scenarios where it's important to also maintain a global model with high initial accuracy often neglected by previous personalized FL works.

Note that our approach also does not critically depend on the global model's performance. Even in the worst case where the input statistics derived from the global model are junk (e.g., they degenerate to a constant vector, or are simply a random noise vector), then it just means the hyperparameters learned are no longer input-dependent. In this case, FedL2P would effectively learn a constant vector of layer-wise learning rates + BN mixing ratio, as opposed to a function that predicts them. Thus, in this worst case we would lose the ability to customize the hyperparameters differently to different heterogeneous clients, but we would still be better off than the standard approach where these optimization hyperparameters are not learned. In the case where our global-model derived input features are better than this degenerate worst case, FedL2P's meta-nets will improve on this already strong starting point.

C Cost of FedL2P

Computational Cost of Hessian Approximation. Wecompare with hessian-free approaches, namely first-order (FO) MAML and hessian-free (HF) MAML, both of which are used by PerFedAvg, and measure the time it took to compute the meta-gradient after fine-tuning. Specifically, we run 100 iterations of each algorithm and report the mean of the walltime. Our proposed method takes 0.24 seconds to compute the hypergradient, 0.12 seconds of which is used to approximate the Hessian. In comparison, FO-MAML took 0.08 seconds and HF-MAML took 0.16 seconds to compute the metagradient. Hence, our proposed method is not a significant overhead relative to simpler non-Hessian methods. It is also worth noting that the number of fine-tuning epochs would not impact the cost of computing the hypergradient.

Memory Cost. In our CIFAR10 experiments, the meta-update of FedL2P has a peak memory usage of 1.3GB. In contrast, existing FL methods that generate personalized policies require an order(s) of magnitude more memory and hence only evaluated in relatively small setups with smaller networks. For instance, pFedHN [55] requires in a peak memory usage of 17.93GB in our CIFAR10 setup as its user embeddings and hypernetwork scale up with the number of clients and model size. Moreover, they fail to generate reasonable client weights as these techniques do not scale to larger ResNets used in our experiments. APFL [14], on the other hand, requires each client to maintain three models: local, global, and mixed personalized. Adopting APFL in our CIFAR10 setup of 1000 clients requires over 134GB of memory just to store the models per experiment, which is infeasible.

Communication Cost. For each FL round, we transmit the parameters of the meta-nets, which are lightweight MLP networks to from server to client and vice versa. Note that transmitting the global pretrained model to each new client is a one-time cost. Office-Caltech-10, DomainNet, and Speech Commands setups take a maximum of 100 communication rounds, 0.24% of the pretraining cost, to learn the meta-nets. CIFAR-10 and CIFAR-10-C, on the other hand, can take hundreds of

rounds up to a maximum of 500 rounds, 0.38% of the pretraining cost. In contrast, joint model and hyperparameter optimization typically takes thousands of rounds [30], having to transmit both the model and the hyperparameter distribution across the network. In summary, FedL2P incurs <1% additional costs on top of pretraining and forgoes the cost of federatedly learning a model from scratch, which can be advantageous in certain scenarios as listed in Section. B.

Inference Cost. During the fine-tuning stage, given the learned meta-nets, FedL2P requires 2 forward pass of the model per image and one forward pass of each meta-net to compute the personalized hyperparameters. This equates to 0.55GFLOPs per image and would incur a minor additional cost of 4.4% more than the regular finetuning process of 15 finetune epochs.

D Pretrained Model and Setup Details

We use the Flower federated learning framework [6] and 8 NVIDIA GeForce RTX 2080 Ti GPUs for all experiments. ResNet-18 [20] is adopted with minor differences in the various setups:

CIFAR-10. We replaced the first convolution with a smaller convolution 3 × 3 kernel with stride = 1 and padding = 1 instead of the regular 7 × 7 kernel. We also replaced the max pooling operation with the identity operation and set the number of output features of the last fully connected layer to 10. The model is pretrained in a federated manner using FedAvg [46] or FedBABU [48] or PerFedAvg(HF) [15] with a starting learning rate of 0 . 1 for 500 communication rounds. For PerFedAvg, we adopted the recommended hyperparameters used by the authors to meta-train the model. The fraction ratio is set to r = 0 . 1 ; 100 clients, each of who perform a single epoch update on its own local dataset before sending the updated model back to the server, participate per round. We dropped the learning rate by a factor of 0 . 1 at round 250 and 375 . This process is repeated for each α = 1000 , 1 . 0 , 0 . 5 , 0 . 1 , resulting in a pretrained model for each group of clients. We experiment with various fine-tuning learning rates { 1 . 0 , 0 . 1 , 0 . 01 , 1 e -3 , 1 e -4 , 1 e -5 } and pick the best-performing one, 1 e -3 for all experiments; the initial value of ˜ η in FedL2P is also set at 1 e -3 .

CIFAR-10-C. We adopted the pretrained model trained in CIFAR-10 for α = 1000 and used the same fine-tuning learning rate for all experiments.

Office-Caltech-10 & DomainNet. We adopted a Resnet-18 model that was pretrained on ImageNet [13] and provided by torchvision [41]. We replaced the number of output features of the last fully connected layer to 10. Similar to CIFAR-10 setup, we experiment with the same set of learning rates and pick the best-performing one, 1 e -2 for our experiments.

Speech Commands. We adopted the setup and hyperparameters from ZeroFL [52]; a Resnet-18 model is trained using FedAvg for 500 rounds using a starting learning rate of 0 . 1 and an exponential learning rate decay schedule with a base learning rate of 0 . 01 . We use the base learning rate for fine-tuning for all experiments.

E Architecture & Initialization Details

We present the architecture of our proposed meta-nets, BNNet and LRNet. Both networks are 3-layer MLP models with 100 hidden layer neurons and ReLU [1] activations in-between layers. BNNet and LRNet clamp the output to a value of [0 , 1] and [0 , 1000] respectively and use a straight-through estimator [4] (STE) to propagate gradients. We also tried using a sigmoid function for BNNet which converges to the same solution but at a much slower pace. We initialize the weights of BNNet and LRNet with Xavier initialization [18] using the normal distribution with a gain value of 0 . 1 . To control the starting initial value of BNNet and LRNet, we initialize the biases of BNNet and LRNet with constants 0 . 5 and 1 . 0 , resulting in initial values of ∼ 0 . 5 and ∼ 1 . 0 respectively. We also tried experimenting BNNet with different initializations by setting the biases to [0 . 2 , 0 . 5 , 0 . 8] and got similar results.

F Additional Results

Relative Clustered Distance Maps . We present an extension of Fig. 3 for both the inputs, ξ & x , and outputs, β & η , of the meta-nets in Fig. 5.

1000

1.0

0.5

0.1

0.0

0.1

0.2

0.3

0.4

0.5

1000

1.0

0.5

0.1

0.000

0.005

0.010

0.015

1000

1.0

0.5

0.1

0.2

0.4

0.6

0.8

Sparsity

(a) CIFAR10 (

e

= 5

)

0.0

0.2

0.4

0.6

0.8

1.0

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

0.0150

0.695

0.700

0.705

0.710

Sparsity

(b) Speech Commands

Learned Personalized Hyperparameters . We present the learned hyperparameters for the other client groups not shown in Fig. 2 in Fig. 4.

Comparison with FOMAML . We remark that off-the-shelf FOMAML [16] (as an initial condition learner) is not a meaningful point of comparison because our problem is to fine-tune/personalize pre-trained models. Therefore, to compare with FOMAML, we focus on learning our FedL2P meta-nets with FOMAML style meta-gradients instead of IFT meta-gradients. In order to apply FOMAML to learning rate optimization, this also required extending FOMAML with the same trick as we did for our IFT approach as shown in Eq. 9 & 10. However, FOMAML ignores the learning trajectory except the last step, which may result in performance degradation over longer horizons. We term this baseline FOMAML+. Table. 8 reports results on the multi-domain datasets as described in Section. 4.4, keeping the same initial condition and meta representation (LRNet and BNNet meta-nets), and varying only the optimization algorithm. Our approach of using IFT to compute the best response Jacobian, outperforms FOMAML+ for α = 1000 , 1 . 0 , 0 . 5 and has comparable performance for α = 0 . 1 .

amazon

caltech

dslr

webcam

amazon

caltech

dslr

webcam

0.0

14.19

14.82

14.35

14.19

0.0

15.17

14.83

14.82

15.17

0.0

13.95

14.35

14.83

13.95

0.0

amazon

caltech

dslr

webcam

amazon

caltech

dslr

webcam

0.0

9.39

10.04

9.51

9.39

0.0

10.46

10.15

10.04

10.46

0.0

9.13

9.51

10.15

9.13

0.0

amazon

caltech

dslr

webcam

amazon

caltech

dslr

webcam

0.0

12.0

13.09

12.5

12.0

0.0

12.77

12.08

13.09

12.77

0.0

12.38

12.5

12.08

12.38

0.0

x

amazon

caltech

dslr

webcam

amazon

caltech

dslr

webcam

0.0

0.69

1.79

1.39

0.69

0.0

1.61

1.39

1.79

1.61

0.0

1.1

1.39

1.39

1.1

0.0

0

2

4

6

8

10

12

14

0

2

4

6

8

10

0

2

4

6

8

10

12

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

(a) Office-Caltech-10 (

α

= 1000

)

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

1.35

2.12

4.09

2.3

1.16

1.35

0.0

1.49

4.13

1.69

1.55

2.12

1.49

0.0

4.31

0.21

2.04

4.09

4.13

4.31

0.0

4.47

3.58

2.3

1.69

0.21

4.47

0.0

2.21

1.16

1.55

2.04

3.58

2.21

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

1.15

2.42

4.16

2.67

0.73

1.15

-0.0

1.94

4.1

2.19

1.46

2.42

1.94

-0.0

4.38

0.07

2.39

4.16

4.1

4.38

-0.0

4.61

3.88

2.67

2.19

0.07

4.61

0.0

2.63

0.73

1.46

2.39

3.88

2.63

-0.0

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

1.1

1.73

2.21

1.69

1.21

1.1

0.0

1.21

2.54

1.21

1.4

1.73

1.21

-0.0

2.86

0.22

1.92

2.21

2.54

2.86

0.0

2.81

2.07

1.69

1.21

0.22

2.81

0.0

1.86

1.21

1.4

1.92

2.07

1.86

0.0

x

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

1.05

1.67

1.65

1.94

1.41

1.05

0.0

1.41

2.22

1.7

1.25

1.67

1.41

-0.0

2.5

0.2

1.5

1.65

2.22

2.5

0.0

2.74

2.05

1.94

1.7

0.2

2.74

0.0

1.79

1.41

1.25

1.5

2.05

1.79

0.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0

1

2

3

4

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.5

1.0

1.5

2.0

2.5

(b) DomainNet (

α

= 1000

)

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

1.08

1.81

3.69

1.73

0.87

1.08

0.0

1.37

3.92

1.29

1.4

1.81

1.37

0.0

4.09

0.1

1.86

3.69

3.92

4.09

0.0

4.0

3.28

1.73

1.29

0.1

4.0

0.0

1.77

0.87

1.4

1.86

3.28

1.77

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.89

2.1

3.76

2.08

0.44

0.89

0.0

1.75

3.94

1.72

1.27

2.1

1.75

0.0

4.28

0.02

2.19

3.76

3.94

4.28

0.0

4.25

3.55

2.08

1.72

0.02

4.25

0.0

2.16

0.44

1.27

2.19

3.55

2.16

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.9

1.54

1.87

1.41

1.0

0.9

0.0

1.05

2.21

0.97

1.19

1.54

1.05

0.0

2.57

0.15

1.74

1.87

2.21

2.57

-0.0

2.43

1.73

1.41

0.97

0.15

2.43

0.0

1.6

1.0

1.19

1.74

1.73

1.6

-0.0

x

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

0.85

1.67

1.59

1.73

0.85

0.85

-0.0

1.2

1.94

1.3

0.85

1.67

1.2

-0.0

2.36

-0.0

1.54

1.59

1.94

2.36

0.0

2.36

1.59

1.73

1.3

-0.0

2.36

-0.0

1.61

0.85

0.85

1.54

1.59

1.61

-0.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.5

1.0

1.5

2.0

(c) DomainNet (

α

= 1

.

0

)

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

0.85

1.42

3.36

1.37

0.75

0.85

0.0

1.1

3.71

1.04

1.38

1.42

1.1

0.0

3.72

0.06

1.7

3.36

3.71

3.72

0.0

3.68

3.22

1.37

1.04

0.06

3.68

0.0

1.65

0.75

1.38

1.7

3.22

1.65

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.66

1.69

3.54

1.59

0.39

0.66

0.0

1.48

3.86

1.36

1.26

1.69

1.48

0.0

4.05

0.03

2.01

3.54

3.86

4.05

-0.0

3.96

3.55

1.59

1.36

0.03

3.96

0.0

1.91

0.39

1.26

2.01

3.55

1.91

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

0.86

1.26

1.63

1.15

0.84

0.86

0.0

0.95

2.14

0.9

1.21

1.26

0.95

-0.0

2.25

0.1

1.5

1.63

2.14

2.25

0.0

2.14

1.56

1.15

0.9

0.1

2.14

0.0

1.38

0.84

1.21

1.5

1.56

1.38

0.0

x

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.59

1.27

1.25

1.22

0.59

0.59

-0.0

1.06

1.88

1.04

0.69

1.27

1.06

0.0

2.18

0.11

1.32

1.25

1.88

2.18

0.0

2.11

1.67

1.22

1.04

0.11

2.11

0.0

1.29

0.59

0.69

1.32

1.67

1.29

-0.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

(d) DomainNet (

α

= 0

.

5

)

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

0.85

1.28

3.12

1.14

0.69

0.85

-0.0

1.01

3.51

0.86

1.33

1.28

1.01

0.0

3.4

0.05

1.49

3.12

3.51

3.4

0.0

3.28

2.86

1.14

0.86

0.05

3.28

0.0

1.35

0.69

1.33

1.49

2.86

1.35

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.83

1.33

3.14

1.19

0.61

0.83

0.0

1.08

3.53

0.92

1.33

1.33

1.08

0.0

3.45

0.05

1.56

3.14

3.53

3.45

0.0

3.34

2.91

1.19

0.92

0.05

3.34

0.0

1.44

0.61

1.33

1.56

2.91

1.44

0.0

real

sketch

infograph

painting

quickdraw

real

sketch

0.0

0.72

1.09

1.49

0.98

0.75

0.72

0.0

0.72

1.89

0.69

1.01

1.09

0.72

0.0

1.98

0.07

1.27

1.49

1.89

1.98

0.0

1.86

1.35

0.98

0.69

0.07

1.86

0.0

1.16

0.75

1.01

1.27

1.35

1.16

-0.0

x

real

sketch

infograph

painting

quickdraw

real

sketch

-0.0

0.9

1.16

1.3

1.04

0.69

0.9

0.0

0.75

1.79

0.64

1.05

1.16

0.75

0.0

1.73

0.11

1.06

1.3

1.79

1.73

0.0

1.62

1.18

1.04

0.64

0.11

1.62

0.0

0.95

0.69

1.05

1.06

1.18

0.95

-0.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

E

E

β

n

β

n

R

E

β

n

(e) DomainNet (

α

= 0

.

1

)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study primarily concentrates on refining hyperparameters in federated learning, especially concerning non-IID data. The researchers highlight two key hyperparameter groups—Batch Normalization (BN) and selective update hyperparameters—that yield impressive outcomes.\n\nBN hyperparameters help mitigate feature shifts, a prevalent problem in federated learning. FedBN, a cited method, aids in managing feature shift more effectively by localizing the BN layers while keeping other layers global. Details of the BN layer formation and its constituents are also given.\n\nMoreover, the study explores a global model's application, initially trained through existing federated algorithms, which is then fine-tuned to the client set. The CIFAR-10 dataset split among various clients is utilized for this process, demonstrating that their approach maintains global performance while enhancing personalized performance.\n\nThe researchers also incorporate the Jacobian product to approximate the Hessian inverse and calculate the hyper gradient, forming a part of their suggested algorithm.\n\nFor experimental purposes, ResNet-18 and SGD are consistently employed for all tests. Batch size, local epoch number, learning rate, hyper gradient clipping, maximum communication rounds, and other parameters are also predefined.\n\nThis paper contributes to federated learning by introducing a method to optimize hyperparameters, especially in non-IID data scenarios, and backs its effectiveness through empirical results.\n\nSTRENGTHS:\n**Originality:** The paper stands out due to its unique approach to handling non-IID data within federated learning. Implementing Batch Normalization (BN) hyperparameters to tackle feature shift is an inventive fusion of existing deep learning knowledge and federated learning's distinct hurdles. Moreover, the employment of the Jacobian product to estimate the Hessian inverse and compute the hyper gradient is a distinct application within the federated learning sphere, marking an inventive merge of established concepts to address a recurring federated learning issue.\n\n**Quality:** The paper's quality appears robust. It reflects the authors' thoughtful research methodology and the theoretical integrity of their approach to hyperparameter optimization. Leveraging the CIFAR-10 dataset for validation is a norm in this field, and the authors thoroughly outline their experimental setup. Nevertheless, a broader evaluation across diverse datasets and juxtaposition with other methodologies could further enrich the paper's quality.\n\n**Clarity:** The paper is well-crafted and clearly depicts the research. The authors present exhaustive descriptions of their methodology and the theoretical underpinnings of their strategy. They also lucidly detail their experimental setup and findings. Such explicitness in communication renders the paper comprehensible to a broad audience, from novices in federated learning to seasoned researchers.\n\n**Significance:** The paper holds substantial significance. It addresses pivotal challenges, such as dealing with non-IID data and optimizing hyperparameters, vital in federated learning. The authors' solutions to these challenges could potentially lead to the more efficient and practical training of federated learning models, making a notable contribution to the field. Additionally, this paper's insights could stimulate new research pathways and methodologies within the broader realm of machine learning.\n\nWEAKNESSES:\nIn summary, while the paper is fundamentally and conceptually solid, there are a few weaknesses that weaken its contribution and soundness. \n\n1. **Dataset and Model Constraints:** The study principally uses the CIFAR-10, Office-Caltech-10, and DomainNet datasets in its experiments. Although these are recognized standards, the findings may not extend to different datasets and models. Testing the proposed method across diverse datasets (such as images, text, and tabular data) and various model architectures could better showcase the method's adaptability and resilience.\n2. **Comparison to Other Techniques:** A broader comparison with cutting-edge methods could enhance the paper. While some algorithms are referenced, the relative performance of the authors' method remains unclear. A side-by-side comparison could clarify the comparative advantages and disadvantages of their approach.\n3. **Computational Overhead:** The authors suggest the Jacobian product for approximating the Hessian inverse and computing the hypergradient, a fresh perspective. Appendix C mentions that the computational cost compared to FO / HF-MAML is still relatively large. \n4. **Applicability:** The paper's significant contribution to federated learning cannot be overlooked, yet the universal applicability of their method warrants examination. Federated learning is applied across various sectors, each with unique data distributions, privacy stipulations, and computational resources. Discussing the limitations and potential solutions of their method would fortify the paper.\n5. **Reproduction**: The codes are not available yet.\n\nQUESTIONS:\n1. Could you elaborate on how generalizable your method is to different federated learning scenarios? For instance, how would your method perform with different data distributions, privacy requirements, and computational resources?\n2. Could you provide a more comprehensive comparison with other state-of-the-art methods? It would be helpful to understand how your method performs relative to these other approaches in terms of accuracy and computational efficiency. I am sill unclear about whether your baselines cover enough range of comparable works. \n3. What particular `pre-trained` models did you use in your experiments? \n4. Additional Datasets and Models: Have you considered testing your method on various datasets with different characteristics (e.g., image, text, tabular data) and different model architectures? \n5. Could you discuss the limitations of your approach and potential ways to overcome these limitations? This would provide a more balanced view of your work and could help guide future research in this area.\n\nLIMITATIONS:\nIn terms of broader societal impacts, it's not clear from the provided excerpts whether the authors have considered this aspect.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new personalization strategy for the Federated Learning setting. Their key idea is to train a meta-network that takes as input the client's data and outputs optimal hyperparameters for that data. These optimal hyperparameters can then be used to personalize the pretrained model. The hyperparameters they choose to output control the batch-norm mean, std-dev and per layer learning rate. They evaluate their technique on 4 datasets, and show that their technique outperforms other personalization strategies on these datasets.\n\nSTRENGTHS:\n- The core idea of using a meta-net proposed by the authors is quite novel and I believe it is a very promising direction.\n- The hypergradient technique they propose to train this metanet is also quite novel and interesting.\n- For the most part their algorithm and technique is clearly presented.\n\nWEAKNESSES:\n- The experiment setup is not clearly defined and confusing. Examples\n   - In their CIFAR10 experiments, they say 20% of the data is used for test, it's not clear if that means 20% of the clients are not used in the FL step or 20% of the data in each client is not used in the FL step.\n  - It is not clear what fraction of clients are used for training the FL model in each expt\n  - Figures 2 and 3 are poorly labeled and not clearly explained. Figure 3 is completely unreadable.\n\n- Experimental results are not very strong\n  - FedL2P seems very computationally expensive (total compute used is not compared anywhere) and the results are only marginally better than other techniques.\n  - No evaluation on actual non-IID datasets. There are several non-IID datasets available publicly for FL - EMNIST, Shakespeare, StackOverflow etc. Evaluating their method on these datasets would make the results much more compelling.\n - No evaluation on non-image datasets.\n \n- Hyperparameters limited to continuous values - many hyperparameters like # epochs, choice of optimizer etc. are not continuous.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach that utilizes meta-learning to learn hypernets that aim to determine the optimal fine-tuning hyperparameters for individual clients. The proposed idea is intriguing and holds promise. The results presented in the paper are compelling. However, the paper would benefit from improvements in terms of writing and clarity. Overall, it is a borderline paper. \n\n--- post rebuttal --\n\nIn the rebuttal, the author address my questions about clarity and positioning their work. So I increase my score from 4 to 5.\n\nSTRENGTHS:\n- The paper proposes a novel idea of using meta-learning to learn hypernet that can estimate the best fine-tuning hyperparameters for personalizing local models. \n- The results show the proposed FedL2P can effectively improve performance on multiple standard FL methods and remain robust when there is a data distribution shift across clients.\n\nWEAKNESSES:\n- The **clarity** needs to be improved. Especially, some details of the proposed method are missing. Please see the question section below.\n- The proposed method is **limited** to one certain type of personalized federated learning, i.e., fine-tuning local models. There are other strategies of personalization that are not applicable to this work. For example, one strategy is decoupled model (global shared encoder + local predictive head) [1,2]. Another strategy could be using Hypernet to directly optimize different local models [3].\n\n[1] Zhong, A., He, H., Ren, Z., Li, N. and Li, Q., 2022, September. FedDAR: Federated Domain-Aware Representation Learning. In The Eleventh International Conference on Learning Representations.\n\n[2] Collins, L., Hassani, H., Mokhtari, A. and Shakkottai, S., 2021, July. Exploiting shared representations for personalized federated learning. In International conference on machine learning (pp. 2089-2099). PMLR.\n\n[3] Shamsian, A., Navon, A., Fetaya, E. and Chechik, G., 2021, July. Personalized federated learning using hypernetworks. In International Conference on Machine Learning (pp. 9489-9502). PMLR.\n\nQUESTIONS:\n- (line 184) Could you explain what is \"the degree of data heterogeneity of the client’s local data with respect to the pretrained model\"? The data heterogeneity comes from local data and the pretrained model?\n- (line 205) Should $BNNet(w_{bn}; ξ_1, ξ_2, . . . , ξ_{B−1}, ξ_B)$ be $BNNet(ξ_1, ξ_2, . . . , ξ_{B−1}, ξ_B;w_{bn})$? It seems that  $ξ_1, ξ_2, . . . , ξ_{B−1}, ξ_B$ are BNNet's input.\n- (line 206) Could you explain why there is a Hadamard product for the LRNet's output?\n- title of section 3.3 is a bit weird. why you are Define Training/Validation Losses for IFT (implicit function theorem). Isn't it that training/validation losses should defined by your tasks?\n\nLIMITATIONS:\nNot discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for fine-tuning a client's model based on a pretrained model using its local data. The proposed personalization approach involves fine-tuning the learnable weights and biases (assume, there are $M$ such layers) as well as the means and standard deviations (SDs) of batch normalization (BN) layers (assume, there are $B$ BN layers).\n\nThis fine-tuning process is facilitated by two meta neural networks. The first meta neural network predicts $2M$ learning rates, which correspond to each weight and bias layer, enabling the local model to perform several gradient steps from the pretrained model. The input for this first meta neural network consists of the mean and standard deviation values for each of the $M$ layers.\n\nOn the other hand, the second meta neural network directly predicts the means and SDs. Its input comprises the distance between the pretrained means and the running means of the model with the pretrained weights on local data, as well as a similar distance measure between the SDs.\n\nBoth meta neural networks are trained using FedAvg on hyper gradients.\n\n--------\nPost-rebuttal update: I did not have significant concerns about this paper. I acknowledge the authors' rebuttal and extend my gratitude to them, particularly for adding the new experiment. I have opted not to modify my favorable rating of 6.\n\nSTRENGTHS:\n1. The paper demonstrates sufficient originality by introducing a novel approach using meta neural networks for computing personalized hyperparameters, as outlined in the summary above. Notably, the concept of trainable meta nets holds promise and appears to have practical applicability.\n\n2. The authors effectively elaborate on the algorithm's intricacies, providing comprehensive descriptions of various components of the main method. Additionally, Figure 1 illustrates a schematic depiction of the main approach, enhancing the clarity of their explanations.\n\n3. Extensive computational experiments are incorporated within the paper to substantiate the efficacy of the proposed method. \n\n4. The paper is a valuable contribution to the fields of personalized Federated Learning and Meta Learning.\n\nWEAKNESSES:\n1. Despite the authors' efforts in providing extensive explanations for their ideas, regrettably, the text still contains several notable inconsistencies, as detailed in the \"Questions\" section.\n\n2. A comparative analysis with the FOMAML method [1] is notably absent in the work. It is worth mentioning that FOMAML shares significant similarities with the authors' proposed method, as both utilize the MAML approach for fine-tuning the model using the training dataset and making optimization steps for the meta parameter based on the validation dataset, as observed in the current study. Additionally, in another related work [2], which builds upon MAML, the hyper gradient is explicitly addressed (see lines 14 and 16 of Algorithm 1 in [2]).\n\n[1] \"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks\", Finn et al.\n\n[2] \"Unsupervised Meta-Learning for Few-Shot Image Classification\", Khodadadeh et al.\n\nQUESTIONS:\n1. Line 225: If I understand correctly, the text implies that $\\partial_{{\\omega}_{lr}} {\\cal L}_V = 0$. If my interpretation is correct, could you please explain why this is the case?\n\n2. Equation 3 & Line 199: What is the range of the index $i$?\n\nMinor comments & suggestions:\n1. Line 6: Depending -> depending\n2. Line 35: Even -> even\n4. Inaccuracy on lines 87-91: Although it is evident what the authors intended to convey, the wording is imprecise. I would recommend specifying the subset of chosen clients as $\\tilde{C}$ on line 87, as done in Algorithm 1, and then using $i \\in \\tilde{C}$ instead of $i \\in C$ on line 88. The statement $P_i = \\cal{X} \\times \\cal{Y}$ on line 90 is also misleading: $\\cal{X} \\times \\cal{Y}$ represents the domain of the distribution and does not define the distribution itself. Accordingly, the summation on line 91 should be taken over $i \\in \\tilde{C}$.\n5. Line 93: The space before the word \"for\" is missing.\n6. The first part of the statement on lines 93-95, in my opinion, is incorrect. From a statistical perspective, the aim of Federated Learning is to minimize the generalized local loss function $\\mathbb{E}_{(x, y) \\in P_i} \\cal{L}_i (\\theta, x, y)$. However, many Federated Learning frameworks minimize the finite sum because they make an implicit assumption that the distributions are similar. Otherwise, one could construct two distributions $P_1$ and $P_2$ such that the global solution for $1/2(P_1 + P_2)$ is ineffective for both clients.\n7. The abbreviation SD is used on line 109 but only defined later on line 180.\n8. Line 121: \"are\" -> \"is\"\n10. Line 191: For clarity, I would suggest spelling out the abbreviation MLP.\n11. Line 190: Please provide explicit definitions and domains for $\\eta_i$ and $\\beta_j$ as they are not scalars.\n12. Equation 5 is misleading. The loss function on both lines is taken over the data distribution $P_i$, but there are also indices $T$ and $V$, which denote train and test datasets. $theta_i(\\lambda)$ is not a minimizer of the generalized loss function but rather a fine-tuning of the global solution $\\theta_g$. It takes some time to understand the authors' intended meaning here.\n13. Line 226: There is a redundant full stop after the first word \"Algorithm.\"\n14. Algorithm 1, the input section: The phrases \"Total no. of clients C. Num of update iterations K\" use different notation to express \"number,\" leading to inconsistency within the text.\n15. Lines 9-10 in Algorithm 1 do not clearly depict the intended FedAvg iterates.\n16. The document concludes on page 8 with Section \"4.4 Personalizing to Different Domains.\"\n\nLIMITATIONS:\nAuthors adequately address the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes FedL2P, a federated learning framework for learning personalized strategies in the federated meta-learning problem. The authors introduce meta-nets to estimate hyper-parameters for personalized fine-tuning, such as batch normalization parameters and learning rates, based on client-specific data statistics. The empirical results show that FedL2P outperforms standard hand-crafted personalization baselines in both label and feature shift situations.\n\nSTRENGTHS:\n- The proposed method demonstrates soundness in its approach. Utilizing hyperparameter inference for federated learning shows promise and effectiveness in achieving personalization, as it tries to customize only a few crucial hyper-parameters.\n- The experiments are meticulously evaluated across diverse distribution shifts, encompassing feature shift, label shift, and domain shift.\n\nWEAKNESSES:\n- Calculating meta-gradients involves time-consuming computations of second-order derivatives.\n- The comparison is limited to variants of finetuning, neglecting the majority of personalized federated learning baselines such as [1,2]. Additionally, recent works [3] addressing distribution shifts by optimizing different blocks of the network should be thoroughly discussed in the paper.\n- The sufficiency and effectiveness of communicating the selected hyper-parameters for personalized federated learning lack rigorous verification.\n\n[1] Towards personalized federated learning. TNNLS\n\n[2] Layer-wised model aggregation for personalized federated learning. CVPR 2022\n\n[3] Surgical fine-tuning improves adaptation to distribution shifts. ICLR 2023\n\nQUESTIONS:\n- What is the computational cost associated with the proposed method?\n- How many parameters will be personalized in this study?\n- Could you provide insights into the learning rate distribution for different blocks/layers in the paper and elaborate on what the model has learned? This would greatly enhance the understanding of the results.\n\nLIMITATIONS:\nI'm not aware of any potential negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-07"}