SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[53] Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[54] Mehdi Setayesh, Xiaoxiao Li, and Vincent WS Wong. Perfedmask: Personalized federated learning with optimized masking vectors. In The Eleventh International Conference on Learning Representations.
[55] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, 2021.
[56] Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXiv:2006.16765, 2020.
[57] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.
[58] X Yu Stella and Jianbo Shi. Multiclass spectral clustering. In IEEE International Conference on Computer Vision, volume 2, pages 313-313. IEEE Computer Society, 2003.
[59] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394-21405, 2020.
[60] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.
[61] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pages 7252-7261. PMLR, 2019.
[62] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021.
[63] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-id data. arXiv preprint arXiv:1806.00582, 2018.
[64] Aoxiao Zhong, Hao He, Zhaolin Ren, Na Li, and Quanzheng Li. Feddar: Federated domain-aware representation learning. In International Conference on Learning Representations, 2023.
[65] Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. Single-shot general hyper-parameter optimization for federated learning. In *The Eleventh International Conference on Learning Representations*, 2023.
A Derivation of Equation (7)
From Eq (5), we know that:
$$
\frac {d \mathcal {L} _ {i , T} \left(\theta_ {i} ^ {*} , \lambda\right)}{d \theta} = 0 \tag {11}
$$
Based on the implicit functional theorem (IFT), we get that if we have a function  $F(x,y) = c$ , we can derive that  $y'(x) = -F_x / F_y$ . Therefore, plus the Eq (11) into the theorem, we can get:
$$
\frac {d \theta^ {*}}{d \lambda} = - \frac {\partial_ {\theta} \left(\frac {d \mathcal {L} _ {i , T} \left(\theta_ {i} ^ {*} , \lambda\right)}{d \theta}\right)}{\partial_ {\lambda} \frac {d \mathcal {L} _ {i , T} \left(\theta_ {i} ^ {*} , \lambda\right)}{d \theta}} = - \left(\partial_ {\theta} ^ {2} \mathcal {L} _ {T} (\theta , \lambda)\right) ^ {- 1} \partial_ {\lambda \theta} \mathcal {L} _ {T} (\theta , \lambda) \tag {12}
$$
B Positioning of FedL2P.
Table 7 shows the positioning of FedL2P against existing literature. Note that this list is by no means exhaustive but representative to highlight the position of our work. Most existing approaches obtained personalized models using a personalized policy and local data, often through a finetuning-based approach. This personalized policy can either 1) be fixed, e.g. hand-crafting hyperparameters, layers to freeze, selecting number of mixture components, number of clusters or 2) learned, e.g. learning a hypernetwork to generate weights or meta-nets to generate hyperparameters. These approaches are also grouped based on whether this personalized policy is dependent on the local data during inference, e.g. meta-nets require local client meta-data to generate hyperparameters.
In order to adapt to per-dataset per-client scenarios, many works rely on storing per-client personalized layers, which are trained only on each client's local data. Unfortunately, the memory cost of storing these models scales with the number of clients, C, restricting previous works to small scale experiments. We show that these works are impractical in our CIFAR-10 setup of 1000 clients in Appendix. C. Moreover, most existing methods rely on a fixed personalized policy, such as deriving shared global hyperparameters for all clients in FLoRA, or they do not depend on local data, such as FedEx which randomly samples per-client hyperparameters from learned categorical distributions. Hence, these methods do not adapt as well to per-dataset per-client scenarios and are ineffective at targeting both label and feature shifts. Lastly, although pFedHN and pFedLA target both label and feature shift cases, they do not scale well to our experiments as shown in Appendix. C.
Table 7: Positioning of FedL2P with existing FL approaches. C is the total number of clients, M is the number of layers in the model, B is the number of BN layers in the model, D is the number of data domains, H is the number of hidden layers in the hypernetwork.
FL Approach	Learns Shared Model(s)	Personalized Layers	Personalized Policy Obtained by?	Personalized Policy Data Dependent?	Targets Label Shift	Targets Feature Shift	Memory Cost	Scale to Large Networks
FedProx [33]PerFedAvg [15]pFedMe [59]Ditto [34]MOON [32]FedBABU [48]	Yes	No	Fixed	No	Yes	No	O(M)	✓
PerFedMask [54]	Yes	No	Fixed	No	Yes	No	O(M)	✓
FedBN [35]	Yes	Yes	Fixed	No	No	Yes	O(CB)	✓
FedPer [2]FedRep [11]APFL [14]LG-FedAvg [37]IFCA [17]	Yes	Yes	Fixed	No	Yes	No	O(CM)	✗
FedDAR [64]	Yes	Yes	Fixed	No	Yes	Yes	O(DM)	✗
FLoRA [65]	Yes	No	Fixed	No	Yes	No	O(M)	✓
FedEx [30]	Yes	Supported	Learned	No	Yes	No	O(M)	✓
FedEM [43]	Yes	No	Fixed	No	Yes	No	O(M)	✓
FedFOMO [62]FedMe [44]	No	Yes	Fixed	No	Yes	No	O(CM)	✗
pFedHN [55]pFedLA [40]	No	Yes	Learned	Yes	Yes	Yes	O(CMH)	✗
FedL2P (Ours)	No	Supported	Learned	Yes	Yes	Yes	O(M)	✓
Most importantly, all existing FL approaches shown in Table 7 can use finetuning either to personalize shared global model(s) or as a complementary personalization strategy to further adapt their personalized models. Since our proposed FedL2P focus on better personalizing shared global model(s) by learning better a personalized policy which leverages the clients' local data statistics relative to the given pretrain model(s), our approach is complementary to all existing FL solutions that learn shared model(s); we showed improvements over a few of these works in Table 1.
Use Cases of FedL2P Mainstream FL focuses on training from scratch, but we focus on federated learning of a strategy to adapt an existing pre-trained model (whether obtained by FL or not) on an unseen group of clients with heterogeneous data. There are several scenarios where this setup and our solution would be useful:
1. Scenarios where it's expensive to train from scratch for a new group of clients, e.g. adopting FedEx [30] from scratch for a new group of clients would require thousands of rounds to retrain the model and HP while our method takes hundreds to learn two tiny meta-nets (Appendix. C).
2. Scenarios where there is a publicly available pre-trained foundation model that can be exploited. This is illustrated in Section 4.4 where we adapt a publicly available pretrained model trained using ImageNet on domain generalization datasets.
3. Scenarios where it's important to also maintain a global model with high initial accuracy - often neglected by previous personalized FL works.
Note that our approach also does not critically depend on the global model's performance. Even in the worst case where the input statistics derived from the global model are junk (e.g., they degenerate to a constant vector, or are simply a random noise vector), then it just means the hyperparameters learned are no longer input-dependent. In this case, FedL2P would effectively learn a constant vector of layer-wise learning rates + BN mixing ratio, as opposed to a function that predicts them. Thus, in this worst case we would lose the ability to customize the hyperparameters differently to different heterogeneous clients, but we would still be better off than the standard approach where these optimization hyperparameters are not learned. In the case where our global-model derived input features are better than this degenerate worst case, FedL2P's meta-nets will improve on this already strong starting point.
C Cost of FedL2P
Computational Cost of Hessian Approximation. We compare with hessian-free approaches, namely first-order (FO) MAML and hessian-free (HF) MAML, both of which are used by PerFedAvg, and measure the time it took to compute the meta-gradient after fine-tuning. Specifically, we run 100 iterations of each algorithm and report the mean of the walltime. Our proposed method takes 0.24 seconds to compute the hypergradient, 0.12 seconds of which is used to approximate the Hessian. In comparison, FO-MAML took 0.08 seconds and HF-MAML took 0.16 seconds to compute the metagradient. Hence, our proposed method is not a significant overhead relative to simpler non-Hessian methods. It is also worth noting that the number of fine-tuning epochs would not impact the cost of computing the hypergradient.
Memory Cost. In our CIFAR10 experiments, the meta-update of FedL2P has a peak memory usage of 1.3GB. In contrast, existing FL methods that generate personalized policies require an order(s) of magnitude more memory and hence only evaluated in relatively small setups with smaller networks. For instance, pFedHN [55] requires in a peak memory usage of 17.93GB in our CIFAR10 setup as its user embeddings and hypernetwork scale up with the number of clients and model size. Moreover, they fail to generate reasonable client weights as these techniques do not scale to larger ResNets used in our experiments. APFL [14], on the other hand, requires each client to maintain three models: local, global, and mixed personalized. Adopting APFL in our CIFAR10 setup of 1000 clients requires over 134GB of memory just to store the models per experiment, which is infeasible.
Communication Cost. For each FL round, we transmit the parameters of the meta-nets, which are lightweight MLP networks to from server to client and vice versa. Note that transmitting the global pretrained model to each new client is a one-time cost. Office-Caltech-10, DomainNet, and Speech Commands setups take a maximum of 100 communication rounds,  $0.24\%$  of the pretraining cost, to learn the meta-nets. CIFAR-10 and CIFAR-10-C, on the other hand, can take hundreds of
rounds up to a maximum of 500 rounds,  $0.38\%$  of the pretraining cost. In contrast, joint model and hyperparameter optimization typically takes thousands of rounds [30], having to transmit both the model and the hyperparameter distribution across the network. In summary, FedL2P incurs  $<1\%$  additional costs on top of pretraining and forgoes the cost of federatedly learning a model from scratch, which can be advantageous in certain scenarios as listed in Section B.
Inference Cost. During the fine-tuning stage, given the learned meta-nets, FedL2P requires 2 forward pass of the model per image and one forward pass of each meta-net to compute the personalized hyperparameters. This equates to 0.55GFLOPs per image and would incur a minor additional cost of  $4.4\%$  more than the regular finetuning process of 15 finetune epochs.
D Pretrained Model and Setup Details
We use the Flower federated learning framework [6] and 8 NVIDIA GeForce RTX 2080 Ti GPUs for all experiments. ResNet-18 [20] is adopted with minor differences in the various setups:
CIFAR-10. We replaced the first convolution with a smaller convolution  $3 \times 3$  kernel with stride  $= 1$  and padding  $= 1$  instead of the regular  $7 \times 7$  kernel. We also replaced the max pooling operation with the identity operation and set the number of output features of the last fully connected layer to 10. The model is pretrained in a federated manner using FedAvg [46] or FedBABU [48] or PerFedAvg(HF) [15] with a starting learning rate of 0.1 for 500 communication rounds. For PerFedAvg, we adopted the recommended hyperparameters used by the authors to meta-train the model. The fraction ratio is set to  $r = 0.1$ ; 100 clients, each of who perform a single epoch update on its own local dataset before sending the updated model back to the server, participate per round. We dropped the learning rate by a factor of 0.1 at round 250 and 375. This process is repeated for each  $\alpha = 1000, 1.0, 0.5, 0.1$ , resulting in a pretrained model for each group of clients. We experiment with various fine-tuning learning rates  $\{1.0, 0.1, 0.01, 1e - 3, 1e - 4, 1e - 5\}$  and pick the best-performing one,  $1e - 3$  for all experiments; the initial value of  $\tilde{\eta}$  in FedL2P is also set at  $1e - 3$ .
CIFAR-10-C. We adopted the pretrained model trained in CIFAR-10 for  $\alpha = 1000$  and used the same fine-tuning learning rate for all experiments.
Office-Caltech-10 & DomainNet. We adopted a Resnet-18 model that was pretrained on ImageNet [13] and provided by torchvision [41]. We replaced the number of output features of the last fully connected layer to 10. Similar to CIFAR-10 setup, we experiment with the same set of learning rates and pick the best-performing one,  $1e - 2$  for our experiments.
Speech Commands. We adopted the setup and hyperparameters from ZeroFL [52]; a Resnet-18 model is trained using FedAvg for 500 rounds using a starting learning rate of 0.1 and an exponential learning rate decay schedule with a base learning rate of 0.01. We use the base learning rate for fine-tuning for all experiments.
E Architecture & Initialization Details
We present the architecture of our proposed meta-nets, BNNet and LRNet. Both networks are 3-layer MLP models with 100 hidden layer neurons and ReLU [1] activations in-between layers. BNNet and LRNet clamp the output to a value of  $[0,1]$  and  $[0,1000]$  respectively and use a straight-through estimator [4] (STE) to propagate gradients. We also tried using a sigmoid function for BNNet which converges to the same solution but at a much slower pace. We initialize the weights of BNNet and LRNet with Xavier initialization [18] using the normal distribution with a gain value of 0.1. To control the starting initial value of BNNet and LRNet, we initialize the biases of BNNet and LRNet with constants 0.5 and 1.0, resulting in initial values of  $\sim 0.5$  and  $\sim 1.0$  respectively. We also tried experimenting BNNet with different initializations by setting the biases to  $[0.2,0.5,0.8]$  and got similar results.
Relative Clustered Distance Maps. We present an extension of Fig. 3 for both the inputs,  $\xi$  &  $x$ , and outputs,  $\beta$  &  $\eta$ , of the meta-nets in Fig. 5.
Table 8: Comparison between using FOMAML+ and IFT in FedL2P for Office-Caltech-10 and Domainnet  $(e = 15)$
α	Dataset	+FT (BN C)	+FEdL2P (FOMAML+)	+FEdL2P (IFT)
1000	Caltech-10	80.97±0.33	83.20±1.92	88.85±0.89
DomainNet	52.17±1.55	52.70±0.17	54.38±0.45
1.0	DomainNet	62.27±0.58	63.14±0.38	63.77±0.44
0.5	DomainNet	71.39±0.97	71.37±0.79	72.64±0.3
0.1	DomainNet	86.03±0.47	86.22±0.16	86.36±0.45
Figure 4: Locality, spread, and skewness of FedL2P's learned hyperparameters,  $\beta$  and  $\eta$ , of different layers across clients and the model's sparsity of all clients for each personalization scenario.
Learned Personalized Hyperparameters. We present the learned hyperparameters for the other client groups not shown in Fig. 2 in Fig. 4.
Comparison with FOMAML. We remark that off-the-shelf FOMAML [16] (as an initial condition learner) is not a meaningful point of comparison because our problem is to fine-tune/personalize pre-trained models. Therefore, to compare with FOMAML, we focus on learning our FedL2P meta-nets with FOMAML style meta-gradients instead of IFT meta-gradients. In order to apply FOMAML to learning rate optimization, this also required extending FOMAML with the same trick as we did for our IFT approach as shown in Eq. 9 & 10. However, FOMAML ignores the learning trajectory except the last step, which may result in performance degradation over longer horizons. We term this baseline FOMAML+. Table. 8 reports results on the multi-domain datasets as described in Section. 4.4, keeping the same initial condition and meta representation (LRNet and BNNet meta-nets), and varying only the optimization algorithm. Our approach of using IFT to compute the best response Jacobian, outperforms FOMAML+ for  $\alpha = 1000, 1.0, 0.5$  and has comparable performance for  $\alpha = 0.1$ .
(a) Office-Caltech-10  $(\alpha = 1000)$
(b) DomainNet  $(\alpha = 1000)$
(c) DomainNet  $(\alpha = 1.0)$
(d) DomainNet  $(\alpha = 0.5)$
(e) DomainNet  $(\alpha = 0.1)$
Figure 5: Cluster distance maps. Each block represents normalized distance between two domains (e.g., Caltech vs. DSLR), where the distance of the block  $(j, k)$  is measured as the average of the Euclidean distances between all pairs of clients'  $\beta$  that belong to domain  $j$  and domain  $k$  (similarly for  $\eta$ 's). We normalize distance by the within-domain distances (see text), and take log for better visualization. Hence, an off-diagonal block greater than 0 indicates that the corresponding clusters are better aligned with the true domains.