SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left. \partial_ {\lambda} \theta^ {*} \right| _ {\lambda^ {\prime}} = - \left(\partial_ {\theta} ^ {2} \mathcal {L} _ {T} (\theta , \lambda)\right) ^ {- 1} \partial_ {\lambda \theta} \mathcal {L} _ {T} (\theta , \lambda) | _ {\lambda^ {\prime}, \theta^ {*} \left(\lambda^ {\prime}\right)} \tag {7}
$$
The full derivation is shown in Appendix A. We use Neumann approximation and efficient vector-Jacobian product as proposed by Lorraine et al. [38] to approximate the Hessian inverse in Eq. 7 and compute the hypergradient, which is further summarized in Algorithm 2. In practice,  $\theta^{*}$  is approximated by fine-tuning  $\theta_{g}$  on  $\mathcal{L}_T$  using the client's dataset. Note that unlike in many previous works [38, 47] where  $\partial_{\lambda}\mathcal{L}_{V}$  is often 0 as the hyperparameters often do not directly affect the validation loss, in our case  $\partial_{\boldsymbol{w}_{bn}}\mathcal{L}_V\neq 0$ .
Algorithm 1 summarizes FedL2P. Given a pretrained model and a new group of clients to personalize, we first initialize  $\lambda$  (line 1). For every FL round, we sample  $Cr$  clients and send both the parameters of the pretrained model and  $\lambda$  to each client (lines 3-5). Each client then performs a forward pass of their local dataset to compute the mean (E) and standard deviation (SD) of the input features to each layer and the statistical distance between the local feature distributions and the pretrained model's running estimated feature distributions for each BN layer (lines 6-7).  $\lambda$  is then trained for  $K$  iterations; each iteration optimizes the pretrained model on  $\mathcal{L}_T$  for  $e$  epochs, applying  $\beta$  and  $\eta$  computed using Eq. 4 (lines 8-9) at every forward and backward pass respectively. Each client then computes the hypergradient of  $\lambda$  as per Algorithm. 2 and update  $\lambda$  at the end of every iteration (line 10). Finally, after  $K$  iterations, each client sends back the updated  $\lambda$  and its number of data samples, which is used for aggregation using FedAvg [46] (lines 12-14). The resulting  $\lambda$  is then used for personalization: each client finetunes the model using its training set and evaluates it using its test set.
3.3 Adapting the Losses for IFT
In the IFT, we solve the following problem:
$$
\min  _ {\lambda} \mathcal {L} _ {V} \left(\theta^ {*} (\lambda), \lambda\right) \text {s . t .} \theta^ {*} (\lambda) = \arg \min  _ {\theta} \mathcal {L} _ {T} (\theta , \lambda). \tag {8}
$$
For the current  $\lambda$ , we first find  $\theta^{*}(\lambda)$  in (8) by performing several SGD steps with the training loss  $\mathcal{L}_T$ . Once  $\theta^{*}(\lambda)$  is obtained, we can compute the hypergradient  $d_{\lambda}\mathcal{L}_V(\theta^* (\lambda),\lambda)$  by the IFT, which is used for updating  $\lambda$ . As described in (6) and (7), this hypergradient requires  $\partial_{\lambda}\mathcal{L}_T(\theta ,\lambda)$ , implying that the training loss has to be explicitly dependent on the hyperparameter  $\lambda$ . As alluded in Lorraine et al. [38], it is usually not straightforward to optimize the learning rate hyperparameter via the IFT, mainly due to the difficulty of expressing the dependency of the training loss on learning rates. To address this issue, we define the training loss as follows:
$$
\mathcal {L} _ {T} (\theta , \lambda) = \mathbb {E} _ {(x, y) \sim P _ {T}} C E \left(f _ {\theta^ {\prime}, \beta (\lambda)} (x), y\right) \text {w h e r e} \tag {9}
$$
$$
\theta^ {\prime} = \theta - \boldsymbol {\eta} (\lambda) \nabla_ {\theta} \mathbb {E} _ {(x, y) \sim P _ {T}} C E \left(f _ {\theta , \boldsymbol {\beta} (\lambda)} (x), y\right). \tag {10}
$$
Here  $f_{\theta, \beta}(x)$  indicates the forward pass with network weights  $\theta$  and the batch norm statistics  $\beta$ , and  $CE()$  is the cross-entropy loss. Note that in (10), we can take several (not just one) gradient update steps to obtain  $\theta'$ . Now, we can see that  $\mathcal{L}_T(\theta, \lambda)$  defined as above has explicit dependency on the learning rates  $\pmb{\eta}(\lambda)$ . Interestingly, the stationary point  $\theta^*(\lambda)$  of  $\mathcal{L}_T(\theta, \lambda)$  coincides with  $\theta'$ , that is,  $\theta^*(\lambda) = \theta'$ , which allows for a single instance of inner-loop iterations as Line 9 in Alg. 1. Finally, the validation loss is defined as:
$$
\mathcal {L} _ {V} (\theta , \lambda) = \mathbb {E} _ {(x, y) \sim P _ {V}} C E (f _ {\theta , \boldsymbol {\beta} (\lambda)} (x), y),
$$
showing clear dependency on BNNet parameters through  $\beta (\lambda)$  as discussed in the previous section.
Experiments are conducted on image classification tasks of different complexity. We use ResNet-18 [20] for all experiments and SGD for all optimizers. All details of the pretrained models can be found in Appendix. D. Additionally, the batch size is set to 32 and the number of local epochs,  $e$ , is set to 15 unless stated otherwise. The learning rate  $(\zeta)$  for  $\lambda = \{\pmb{w}_{bn}, \pmb{w}_{lr}, \tilde{\pmb{\eta}}\}$  is set to  $\{10^{-3}, 10^{-3}, 10^{-4}\}$ , respectively. The hypergradient is clipped by value
Table 1: FedL2P complements existing FL methods by improving on the finetuning process. Experiments on CIFAR10  $(e = 15)$ .
α	Approach	+FT (BN C)	+FedL2P
1000(↓ heterogeneity)	FedAvg	63.04±0.02	65.13±0.02
PerFedAvg(HF)	34.58±0.13	47.58±0.01
FedBABU	65.00±0.07	66.49±0.03
1.0	FedAvg	61.42±0.13	65.76±0.31
PerFedAvg(HF)	44.85±0.28	50.2±1.26
FedBABU	68.92±0.11	70.71±0.28
0.5	FedAvg	62.34±0.14	68.45±0.5
PerFedAvg(HF)	52.43±0.16	55.05±0.53
FedBABU	72.26±0.1	72.87±0.42
0.1(↑ heterogeneity)	FedAvg	79.15±0.07	80.28±0.07
PerFedAvg(HF)	77.31±0.05	77.68±0.13
FedBABU	79.50±0.08	79.58±0.04
Table 2: Experiments on CIFAR-10 using pretrained model trained using FedAvg [46]. Both initial & personalized accuracies are learnt and personalized on the same set of clients.
α	Epochs (e)	Global Accuracy	+FT (BN C)	+FT (BN G)	+FT (BN I)	+L2P	+FedL2P
1000(↓ heterogeneity)	5	65.13	64.35±0.03	62.14±0.13	56.58±0.08	53.61±0.12	64.53±0.06
15	65.13	63.04±0.02	59.85±0.04	55.72±0.03	58.41±0.38	65.13±0.02
1.0	5	60.19	59.72±0.18	63.45±0.04	50.8±0.06	55.81±0.03	66.05±0.09
15	60.19	61.42±0.13	63.23±0.15	54.94±0.07	61.77±0.25	65.76±0.31
0.5	5	57.12	58.22±0.02	67.16±0.11	50.33±0.01	59.79±0.09	68.96±0.09
15	57.12	62.34±0.14	67.4±0.06	58.12±0.07	65.28±0.23	68.45±0.5
0.1(↑ heterogeneity)	5	44.86	68.04±0.05	78.73±0.04	61.91±0.06	74.26±0.29	80.33±0.13
15	44.86	79.15±0.07	78.97±0.07	75.94±0.0	78.6±0.21	80.28±0.07
$[-1, 1]$ ,  $Q = 3$ , and  $\psi = 0.1$  in Alg. 2. The maximum number of communication rounds is set to 500, and over the rounds we save the  $\lambda$  value that leads to the lowest validation loss, averaged over the participating clients, as the final learned  $\lambda$ . The fraction ratio  $r = 0.1$  unless stated otherwise, sampling  $10\%$  of the total number of clients per FL round. We focus on non-IID labels and feature shifts while assuming that each client has an equal number of samples. Finally, to generate heterogeneity in label distributions, we follow the latent Dirichlet allocation (LDA) partition method [24, 61, 52]:  $y \sim \text{Dir}(\alpha)$  for each client. Hence, the degree of heterogeneity in label distributions is controlled by  $\alpha$ ; as  $\alpha$  decreases, the label non-IIDness increases, and vice versa.
4.1.1 Datasets
CIFAR10 [31]. A widely-used image classification dataset, also popular as an FL benchmark. The number of clients  $C$  is set to 1000 and  $20\%$  of the training data is used for validation.
CIFAR-10-C [21]. The test split of the CIFAR10 dataset is corrupted with common corruptions. We used 10 corruption types<sup>5</sup> with severity level 3 and split each corruption dataset into  $80\% / 20\%$  training/test sets. A subset  $(20\%)$  of the train set is further held out to form a validation set. Each corruption type is partitioned by  $Dir(\alpha)$  among 25 clients, hence  $C = 250$ .
Office-Caltech-10 [19] & DomainNet [51]. These datasets are specifically designed to contain several domains that exhibit different feature shifts. We set the no. of samples of each domain to be the smallest of the domains, random sampling the larger domains. For Caltech-10, we set  $r = 1.0$  and  $C = 4$ , one for each domain and thus partitioned labels are IID. For DomainNet, we set  $C = 150$  of which each of the 6 domain datasets are partitioned by  $Dir(\alpha)$  among 25 clients, resulting in a challenging setup with both feature & label shifts. Following FedBN [35], we take the top 10 most commonly used classes in DomainNet.
Speech Commands V2 [60]. We use the 12-class version that is naturally partitioned by speaker, with one client per speaker. It is naturally imbalanced in skew and frequency, with 2112, 256, 250 clients/speakers for train, validation, and test. We sampled 250 of 2112 training clients with the most data for our seen pool of clients and sampled 50 out of  $256 + 250$  validation and test clients for our unseen client pool. Each client's data is then split  $80\% / 20\%$  for train and test sets, with a further  $20\%$  of the resulting train set held out to form a validation set.
4.1.2 Baselines
We run all experiments three times and report the mean and SD of the test accuracy. We consider three different setups of fine-tuning in our experiments as below. As basic fine-tuning (FT) is used ubiquitously in FL, our experiments directly compare FT with FedL2P in different non-IID scenarios. We are agnostic to the FL method used to obtain the global model or to train the meta-nets.
FT (BN C). Client BN Statistics. Equivalent to setting  $\beta = 1$  in Eq. 2, thus using client statistics to normalize features during fine-tuning. This is similar to FedBN [35], and is adopted by FL works such as Ditto [34] and PerFedAvg [15].
FT (BN G). Equivalent to setting  $\beta = 0$  in Eq. 2. BN layers use the pretrained global model's BN statistics to normalize its features during fine-tuning.
FT (BN I). BN layers use the incoming feature batch statistics to normalize its features during fine-tuning. This setting is adopted by FL works such as FedBABU [48].
L2P. Our proposed method without FL;  $\lambda$  is learnt locally given the same compute budget before being used for personalization. Hence, L2P does client-wise HPO independently.
4.2 Experiments on Marginal Label Shift
We evaluate our approach based on the conventional setup of personalized FL approaches [9, 45], where a global model,  $\theta_{g}$ , is first learned federatedly using existing algorithms and then personalized
${}^{5}$  brightness,frost.jpeg_compression,contrast,snow,motion_blur,pixelate,spackle_noise,fog,saturate
Table 3: Personalized test accuracies of CIFAR-10-C, Office-Caltech-10, Domainnet  $(e = 15)$
α	Dataset	+FT (BN C)	+FT (BN G)	+FT (BN I)	+L2P	+FedL2P
1000(↓ heterogeneity)	CIFAR-10-C	59.58±0.03	57.03±0.08	55.77±0.12	58.80±0.13	59.97±0.22
Caltech-10	80.97±0.33	36.02±25.21	81.43±2.16	75.52±3.83	88.85±0.89
DomainNet	52.17±1.55	30.55±1.07	50.47±0.88	45.04±1.56	54.38±0.45
1.0	CIFAR-10-C	67.37±0.08	66.45±0.03	63.56±0.07	66.63±0.09	68.83±0.15
DomainNet	62.27±0.58	44.15±0.11	59.4±0.7	54.75±0.12	63.77±0.44
0.5	CIFAR-10-C	74.92±0.08	75.24±0.17	71.38±0.01	74.48±0.06	76.78±0.22
DomainNet	71.39±0.97	49.81±1.98	68.94±0.71	66.38±0.78	72.64±0.3
0.1(↑ heterogeneity)	CIFAR-10-C	87.25±0.06	88.5±0.02	83.93±0.04	87.93±0.31	89.23±0.15
DomainNet	86.03±0.47	69.41±1.95	85.35±1.14	83.93±1.02	86.36±0.45
to the same set of clients via fine-tuning. To this end, given the CIFAR-10 dataset partitioned among a group of clients, we pretrained  $\theta_{g}$  following best practices from [23] using FedAvg and finetune it on the same set of clients. Table 2 shows the personalized accuracy of the various fine-tuning baselines (Section 4.1.2) and FedL2P using  $e = 5\& 15$  local epochs on groups of clients with varying label distribution,  $P_{i}(y)$ ;  $\alpha = 1000$  represents the IID case and  $\alpha = 1.0, 0.5, 0.1$  represents more heterogeneous case. As observed in many previous works [48, 27], increasing label heterogeneity would result in a better initial global model at a expense of personalized performance. Our method instead retains the initial global performance and focuses on improving personalized performance.
We also show that in many cases, especially for clients with limited local compute budget  $e = 5$ , utilizing the pretrained model's BN statistics result (BN G) can be more beneficial as CIFAR-10 consists of images from the same natural image domain; in contrast, previous works mainly use either the client's BN statistics (BN C) or the incoming feature batch statistics (BN I) to normalize the features. This strategy is discovered by FedL2P, as illustrated in Fig. 2a where the learned  $\beta$  is 0 for all BN layers of all clients. For the IID case in particular, FedL2P learns a sparsity<sup>6</sup> of 1.0, learning rate  $\eta = 0$ , for all layers in all clients, forgoing fine-tuning and using the initial global model as the personalized model. For  $\alpha = 1.0$  and 0.1, FedL2P learns highly sparse models similar to recent works that proposed fine-tuning only a subset of hand-picked layers [48, 17, 11, 2, 37] to obtain performance gains. Lastly, L2P performs worse than some standard fine-tuning baselines as it meta-overfits on each client's validation set, highlighting the benefits of FL over local HPO.
FedL2P's Complementability with previous FL works. As our proposed FedL2P learns to improve the FT process, it is complementary, not competing, with other FL methods that learn shared model(s). Hence, besides FedAvg, we utilize FedL2P to better personalize  $\theta_{g}$  pretrained using PerFedAvg(HF) [15] and FedBABU [48] as shown in Table. 1, where we compare FedL2P against the most commonly used FT approach, BN C. Our results show that applying FedL2P to all three FL methods can lead to further gains, in most cases outperforming FT in each respective method. This performance improvement can also bridge the performance gap between different methods. For instance, while FedAvg+FT has worse performance than FedBABU+FT in all cases, FedAvg+FedL2P obtained comparable or better performance than FedBABU+FT for  $\alpha = 1000$  & 0.1.
(a) CIFAR10  $(e = 15)$
(b) CIFAR10C
(c) DomainNet
(d) Office-Caltech-10
Figure 2: Locality, spread, and skewness of FedL2P's learned hyperparameters,  $\beta$  and  $\eta$ , of different layers across clients and the model's sparsity of all clients for each personalization scenario.
4.3 Personalizing to Unseen Clients
Unseen during Pretraining. We evaluate the performance of FedL2P on CIFAR-10-C starting from the pretrained model trained using FedAvg on CIFAR-10 in the IID setting  $\alpha = 1000$  (Section. 4.2) as
Sparsity refers to the percent of parameters whose learned learning rate for FT is 0.
Table 4: Experiments on Speech Commands. Both the pretrained model and meta-nets are only learned on the seen pool of clients.
Client Pool	Epochs (e)	Global Accuracy	+FT (BN C)	+FT (BN G)	+FT (BN I)	+L2P	+FedL2P
Seen	5	82.11	70.07±0.63	2.98±0.00	65.97±0.43	64.79±1.84	87.77±0.47
15	82.11	74.15±0.70	2.98±0.0	68.98±0.19	65.80±1.66	84.74±0.38
Unseen	5	81.62	62.76±1.83	2.85±0.00	64.88±0.66	-	87.85±0.18
15	81.62	68.76±3.40	2.85±0.00	67.28±0.46	-	84.60±0.35
shown in Table. 3. These new clients, whose data is partitioned from CIFAR-10-C, did not participate in the pretraining of the global model and, instead, only participate in the training of the meta-nets. As added noise would impact the feature shift among clients, Fig. 2b shows FedL2P learns to use  $\beta = 1$  as compared to sole use of  $\beta = 0$  in the CIFAR-10 case (Fig. 2a); some clients benefit more by using its own BN statistics. Hence, the performance gap for non-IID cases between BN C and BN G is significantly smaller as compared to CIFAR-10 shown in Table. 2.
Unseen during Learning of Meta-nets. We evaluate the case where some clients are not included in the learning of the meta-nets through our experiments on the Speech Commands dataset. We first pretrain the model using FedAvg and then train the meta-nets using our seen pool of clients. The learned meta-nets are then used to fine-tune and evaluate both our seen and unseen pool of clients as shown in Table. 4. Note that L2P is not evaluated on the unseen pool of clients as we only evaluate the meta-nets learned on the seen pool of clients. We see that fine-tuning the pretrained model on each client's train set resulted in a significant drop in test set performance. Moreover, as each client represents a different speaker, fine-tuning using the pretrained model's BN statistics (BN G) fails entirely. FedL2P, on the other hand, led to a significant performance boost on both the seen and unseen client pools, e.g. by utilizing a mix of pretrained model's BN statistics and the client's BN statistics shown in Appendix F.
Table 5: Ablation study for FedL2P with  $e = {15}$  .
α	Dataset	+FT (BN C)	+FT (BN G)	+FedL2P(BNNet)	+FedL2P(LRNet)β=1	+FedL2P(LRNet)β=0	+FedL2P
1000(↓ heterogeneity)	CIFAR-10	63.04±0.02	59.85±0.04	62.35±0.24	62.62±0.21	65.11±0.02	65.13±0.02
CIFAR-10-C	59.58±0.03	57.03±0.08	59.57±0.13	60.09±0.02	59.30±0.11	59.97±0.22
Caltech-10	80.97±0.33	36.02±25.21	88.12±1.18	85.50±5.76	42.59±22.87	88.85±0.89
DomainNet	52.17±1.55	30.55±1.07	53.39±0.85	55.59±2.76	44.43±3.46	54.38±0.45
1.0	CIFAR-10	61.42±0.13	63.23±0.15	63.75±0.04	64.67±0.06	64.61±0.49	65.76±0.31
CIFAR-10-C	67.37±0.08	66.45±0.03	68.1±0.07	68.62±0.07	67.82±0.1	68.83±0.15
DomainNet	62.27±0.58	44.15±0.11	62.73±0.51	63.69±0.43	diverge	63.77±0.44
0.5	CIFAR-10	62.34±0.14	67.4±0.06	67.59±0.15	68.81±0.05	68.01±0.29	68.45±0.50
CIFAR-10-C	74.92±0.08	75.24±0.17	76.36±0.08	76.86±0.06	76.11±0.07	76.82±0.19
DomainNet	71.39±0.97	49.81±1.98	70.99±1.15	72.74±0.51	diverge	72.64±0.30
0.1(↑ heterogeneity)	CIFAR-10	79.15±0.07	78.97±0.07	79.47±0.2	80.24±0.09	80.39±0.15	80.28±0.07
CIFAR-10-C	87.25±0.06	88.5±0.02	88.6±0.1	89.08±0.04	89.14±0.13	89.23±0.15
DomainNet	86.03±0.47	69.41±1.95	85.87±1.31	85.78±0.6	diverge	86.36±0.45
4.4 Personalizing to Different Domains
We evaluate our method on Office-Caltech-10 and DomainNet datasets commonly used in domain generalization/adaptation, which exhibit both marginal and conditional feature shifts. Differing from the conventional FL setup, we adopted a pretrained model trained using ImageNet [13] and attached a prediction head as our global model. Similar to CIFAR-10-C experiments, we compare our method with the base
lines in Table. 3. Unlike CIFAR-10/10-C, Office-Caltech-10 and DomainNet consist of images from distinct domains and adapting to these domain result in a model sparsity of 0 as shown in Fig. 2c and Fig. 2d. Hence, a better personalization performance is achieved using the client's own local BN statistics (BN C) or the incoming batch statistics (BN I) than the pretrained model's (BN G). Lastly, each client's personalized model uses an extent of the pretrained natural image statistics as well as its own domain statistics, for both datasets.
Further Analysis To further investigate how well the hyperparameters returned by our FedL2P-trained meta-nets capture the domain-specific information, we analyze the clustering of local features, namely  $\pmb{\xi} = (\xi_1,\dots ,\xi_B)$  and  $\pmb{x} = (E(x_{0}),SD(x_{0}),\dots ,E(x_{M - 1}),SD(x_{M - 1}))$  (i.e., the inputs to the meta-nets), and the resulting respective hyperparameters,  $\beta$  and  $\eta$  (i.e., outputs of the meta-nets), among clients. Specifically, we compute the similarity matrix using the Euclidean distance between the features/hyperparameters of any two clients and perform spectral clustering<sup>7</sup>, using the
Table 6: ARI values between the clustering of clients by their domains and inputs/outputs of the BNNet and LRNet.
Dataset	α	BNNet	LRNet
Input (ξ)	Output (β)	Input (x)	Output (η)
Caltech-10	1000	1.0	1.0	1.0	1.0
DomainNet	1000	0.65	0.64	0.76	0.77
1.0	0.70	0.59	0.77	0.74
0.5	0.52	0.53	0.72	0.63
0.1	0.27	0.52	0.65	0.59
<sup>7</sup>We use the default parameters for spectral clustering from scikit-learn [50].
(a) Office-Caltech-10
(b) DomainNet  $(\alpha = 0.5)$
Figure 3: Cluster distance maps. Each block represents normalized distance, where the distance of the block  $(j,k)$  is measured as the average of the Euclidean distances between all pairs of clients'  $\beta$  that belong to domain  $j$  and domain  $k$  (similarly for  $\pmb{\eta}$ 's), normalized by the within-domain distance (see text). An off-diagonal block  $>0$  indicates that the corresponding clusters are better aligned with the true domains.
discretization approach proposed in [58] to assign labels on the normalized Laplacian embedding. We then compute the Adjusted Rand Index (ARI) [25] between the estimated clusters and the clusters partitioned by the different data domains as shown in Table 6. We also visualize the alignment between the estimated clusters and the true domains in Fig. 3, where each block  $(j,k)$  represents the normalized average Euclidean distance between all pairs of clients in domain  $j$  and  $k$ . Specifically, we divide the mean distance between domain  $j$  and  $k$  by the within-domain mean distances and take the log scale for better visualization:  $\log \left(\frac{d(j,k)}{\sqrt{d(j,j)}\sqrt{d(k,k)}}\right)$  where  $d(j,k)$  is the mean Euclidean distance between  $j$  and  $k$ . Thus, a random clustering has distance close to 0 and  $>0$  indicates better alignment between the clustering and the true domains.
As shown, for DomainNet, both BNNet and LRNet consistently preserve the cluster information found in their inputs,  $\xi \& x$ , respectively. However, perfect clustering is not achieved due to the inherent difficulty. For instance, the real and painting domains share similar features, resulting in similar hyperparameters; the cross-domain distance between real and painting is  $\sim 0$  in log-scale in Fig. 3 and hence indistinguishable from their true domains. In contrast, the clients' features and resulting hyperparameters of the Office-Caltech-10 dataset are perfectly clustered (ARI=1) as visualized in Fig. 3a and Appendix. F.
To elucidate the individual impact of BNNet & LRNet, we run an ablation study of all of the datasets used in our experiments and present the results in Table. 5, where CIFAR10 adopts the pretrained model trained using FedAvg. As BNNet learns to weight between client's BN statistics (BN C) and pretrained model's BN statistics (BN G), running FedL2P with BNNet alone leads to either better or similar performance to the better performing baseline. Running LRNet as a standalone, on the other hand, can result in further gains, surpassing the use of both BNNet and LRNet on some benchmarks. Nonetheless, it requires prior knowledge of the data feature distribution of the client in order to set a suitable  $\beta$ , of which  $\beta = 1$  uses BN C and  $\beta = 0$  uses BN G. Our approach assumes no knowledge of the client's data and learns an estimated  $\beta$  per-scenario and per-client using BNNet.
In this paper, we propose FedL2P, a framework for federated learning of personalization strategies specific to individual FL scenarios and datasets as well as individual clients. We learned meta-nets that use clients' local data statistics relative to the pretrained model, to generate hyperparameters that explicitly target the normalization, scaling, and shifting of features as well as layer-wise parameter selection to mitigate the detrimental impacts of both marginal and conditional feature shift and marginal label shift, significantly boosting personalized performance. This framework is complementary to existing FL works that learn shared model(s) and can discover many previous hand-designed heuristics for sparse layer updates and BN parameter selection as special cases, and learns to apply them where appropriate according to the specific scenario for each specific client. As a future work, our approach can be extended to include other hyperparameters and model other forms of heterogeneity, e.g. using the number of samples as an expert input feature to a meta-net.
This work was supported by Samsung AI and the European Research Council via the REDIAL project.