Federated Learning for Inference at Anytime and Anywhere
Authors: Zicheng Liu
Da Li
Javier Fernandez-Marques
Stefanos Laskaridis
Yan Gao
Łukasz Dudziak
Stan Z. Li
Shell Xu Hu
Timothy Hospedales
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning

Abstract: Federated learning has been predominantly concerned with collaborative training of deep networks from scratch, and especially the many challenges that arise, such as communication cost, robustness to heterogeneous data, and support for diverse device capabilities. However, there is no unified framework that addresses all these problems together. This paper studies the challenges and opportunities of exploiting pre-trained Transformer models in FL. In particular, we propose to efficiently adapt such pre-trained models by injecting a novel attention-based adapter module at each transformer block that both modulates the forward pass and makes an early prediction. Training only the lightweight adapter by FL leads to fast and communication-efficient learning even in the presence of heterogeneous data and devices. Extensive experiments on standard FL benchmarks, including CIFAR- 100, FEMNIST and SpeechCommandsv2 demonstrate that this simple framework provides fast and accurate FL while supporting heterogenous device capabilities, efficient personalization, and scalable-cost anytime inference.

Under review as a conference paper at ICLR 2023
FEDERATED LEARNING FOR INFERENCE AT ANYTIME AND ANYWHERE
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning has been predominantly concerned with collaborative training of deep networks from scratch, and especially the many challenges that arise, such as communication cost, robustness to heterogencous data, and support for diverse device capabilities. However, there is no unified framework that addresses all these problems together. This paper studies the challenges and opportunities of exploiting pre-trained Transformer models in FL. In particular, we propose to effi- ciently adapt such pre-trained models by injecting a novel attention- based adapter module each transformer block that both modulates the forward pass and makes an early prediction. Training only the lightweight adapter by FL leads to fast and communication effficient learning even in the presence of heterogencous data and devices. Extensive experiments on standard FL benchmarks, including CIFAR- 100, FEMNIST and SpeechCommandsv2 demonstrate that this simple framework provides fast and accurate FL while supporting heterogenous device capabilities, efficient personalization, and scalable-cost anytime inference. Our anonymous code for reviewing can be found here
INTRODUCTION
Federated learning (FL) was proposed by McMahan et al. (2017) new paradigm for distributed learning in which user data privacy is protected. Following the introduction of the FL setting, subse- quent work focused on addressing the emerging challenges that arise due to FL constraints, such as
communication cost Mishchenko et al. (2022), ( verse device hardware Horvatth et al. (2021): Rapp et al. (2022). For example, to reduce the commu- nication cost, ideas borrowed from model compression, such as quantization Alistarh et al. (201 7):
data heterogeneity Li et al. (2020) and supporting di-
Fu et al. (2020), sparsification [Stich al.] (2018) and pruning Yu et al. (2021); Jiang al. (2022) have been successfully applied; to mitigate the non-lID issue of data heterogeneity, different model training recipes for optimization Li et al. (2020); Wang et al. (2020b), model initialization Nguyen et al. (2022) and architecture design Qu al. (2022) have also been proposed.
A new question has now emerged for FL community; Can we benefit from the recent success of large-scale centralized pre- pre-training of foundation models [Bommasani et al. (2021) Although con- temporary federated learning has predominantly been concerned with collaborative training of deep models from scratch McMahan al.) trained models, it has been observed by Qu et al. (2022) that fine-tuning pretrained vision transform- ers (ViT) significantly improves FL performance for various image recognition tasks and enables great robustness to the data heterogeneity among clients. Despite being an important step forward, fine-tuning the whole pre-trained ViT can be problematic due to the heavy communication cost of exchanging large numbers of model parameters and the weak capabilities of on-device training for many client devices. In this paper, we address this problem by reframing FL as parameter-efficient (PE) downstream learning task. This is in line with the recent PE-based adaptation developments in centralized vision and natural language processing methods. This line of parameter-efficien adap- tation research includes adapters Rebuffi et al. (2017); Houlsby et al. (2019); Tomanek et al. (202
(2017): Li et al. (2020), neglecting publicly available pre-
prompt tuning Li and Liangl 22021); Lester et al. 1). bias-only fine-tuning Zaken et al. (2021) and We contribute
adaptor suited for FL under the foundation model regime, which is designed for the requirements of adaptation to fit client devices at anytime (under different compute and memory budgets) and anywhere (under severe data heterogeneity).

---

Under review as a conference paper at ICLR 2023
Accumulator Laye
ane Sires wnggeagge
70
Average val accuracy over layers
Average val accurscy over leyees
50
Fine-turing
use sere edeasggit ofoeggrgrggeee
50-
40
40-
30-
30
20-
20
Accumulator
10-
10-
Layerwise MLP
Fine
10"
10"
Cumulative communicaton cost (#params in million)
Cumulative communication cost (Aperams in millions)
(a) Non-IID case: a 0.1
(b) Near-IID case: a 1000
The effectiveness of Accumulator bbased federated FM adaptation under anytime and any- where setting in terms of communication cost and classification performance. The experiments conducted on CIFAR- 100. Full details can be found Section [ Each point corresponds to an evaluation during FL, where the cumulative communication cost measures the communication of gradients w.r.t. model parameters between clients and the server. We would like to emphasize that our Accumulator converges faster (less communication cost) than the baselines regardless of the data heterogenity condition and b) performs as well as the upper bound full-model fine-tuning.
Figure I:
Given a pre-trained Transformer model, e.g. ViT Dosovitskiy et al. 2020). DeiT Touvron et al. (202T) and AST Gong al. (202T), we re-wire its feature extraction pathway tackle the anytime and any anywhere challenges. Specifically, we keep track of the CLS tokens after each self-attention transformation and make use of the history of previous CLS tokens to revise the current CTS to- ken by lightweight Transformer, which is termed Accumulator. Our Accumulator has an order of magnitude fewer parameters than a pre-trained Transformer model and the only module train- able during the local forward and backward propagations; therefore, the training and communication efficiencies can be both significantly improved. To show this, we make a comparison between Accu- mulator and standard early-exit (Laskaridis al. 2020) model (Layer-wise MLP, by inserting for each self-attention block MLP classification head) and the full-model fine-tuning (Qu et al. 2022]
Nguyen al. 2022). The comparisons for non-IID, and IID cases are presented in Figure) which clearly show that our method is more efficient in reaching certain target performance in both cases. In addition, due the efficient optimization enabled by our Accumulator, user personalization for
a particular client can be conducted efficiently with even better performance than fine-tuning the whole model.
The contributions of our work are the follows:
We take a different perspective to the existing FL literature and propose parameter- efficient learning method adapt the pre-trained Transformer FMs in FL scenarios.
We propose novel parameter-efficient adapter, which modulates all layers of pre-trained Transformer FM and allows flexible early predictions (anytime inference).
Extensive experiments on standard FL benchmarks, including CIFAR- 100, FEMNIST and SpeechhCommnddv2. show our method can improve global accuracy, personalization and communication efficiency with excellent robustness to data and compute heterogeneities.
RELATED WORK
2.1 FEDERATED LEARNING
Though federated learning is still emerging topic, there are already many works published in the literature. There exist two general FL settings, cross-device McMahan et al, (2017) and cross- silo Heikkilä et al. (2020) FL. In this paper, focus the former. The main research focuses in this setting are designing systems to solve communication efficiency, data and system heterogeneity problems. Researchers have proposed different techniques to improve communication efficiency.
2

---

Under review as a conference paper at ICLR 2023
E.g. Allistrr al. (2017): Fu et al. (2020) proposed to use the quantized model parameters or gradients during the FL communication. Similarly, Stich et al. (2018) sparsified, and Yu et al. (202T): Diang et al. 2022 pruned the training model into smaller cardinality reduce communi- cation cost. And some researchers tried to address this by progressive model training Wang et al.) (2022). Another main focus is on data heterogeneity. In contrast centralized training where the learner has access to the whole distribution, each worker having access to a biased distribution in non-i.idd. FL negatively affects convergence and final model accuracy. In attempts to alleviate this, people proposed to add proximal regularization in the local training termed FedProx Li et al. 20200, Alternatively, some normalized averaging technique mitigate the inconsistency between different clients is proposed in Wang et al. (2020b). Interestingly. more recently, researchers found that model initialization (pre-trained v.s. random) plays an important role reducing the detrimental impact of heterogenienty Nguyen et al. (2022). and so does model architecture (Transfomer v.s. CNN) Qu ct al. (2022). System heterogeneity is also a concern in cross-device FL, where different participants may have different hardware resources and thus be unable to perform the same amount of learn- ing. Some researchers used nested-dropout Rippel et al.] (2014) as a nicely fit method to address the varied of computational constraints between different clients Horvath et al. (2021). as different dimension from considering early-exit networks Laskaridis et al. (2020).
Our work differs from the existing literature. Rather than training from scratch, we focus on the challenges and opportunities of adapting a pre-trained Transfomer FM by FL.. While this might seem exacerbate communication bottlenecks and system heterogeneity issues above, we show that with our novel adapter module we can simultaneously ameliorate all the above challenges of communication cost, heterogeneous device capabilities, and difficulty of federated learning on non- i.i.d. data. Our work goes substantially beyond Qu et al. (2022) and (Nguyen et al., 2022), who just focus on conventional federated fine-tumi Lne-tuning (which we treat as baseline). to support communication efficient federated fine-tuning with support for heterogeneous device capabilities thanks to our novel adaptation module.
2.2 PARAMETER- LEARNING
The idea of parameter-effiiien learning of adapters was first proposed in Rebuffi et al.) (2017) for adapting single model into multiple datasets. It has since been extended into various problems, including few-shot learning [i et al. (2022) and ASR Toomaek et al. (202T). Especially as the pre- trained large-scale FMs model sizes are soaring significantly, parameter-efficient (transfer) learning has become important in NLP. Instead of fine-tuning the full pre-traine model, people lean toward designing different small set modules for adapting the pre-trained FMs into downstream tasks Li and Liang| (202T): Lester et al. (202T); Houlsby et al. (2019). [L and Liang (202T) found that tuning the prompt input of the rre-trained language model enables excellent performance
downstream tasks. While Houlsby et al. (2019) found that fine-tuning some injected adapters can be more effective than fine-tuning the top layers of pre-trained NLP model. In this work, make the first attempt at parameter-efficiene adaptation in an FL context, developing novel transformer based adaptation module specifically customized for this task.
PRELIMINARIES
3.1 FEDERATED LEARNING
Let us consider typical setting of FL with K devices, where a local device has N private training examples denoted by {(x,, f. Vi N. with x, the input image and yj the target label. The learning objective following McMahan et al. (2017) aims at finding model parameter w that minimizes the weighted average loss over all local devices:
K w. = arg min Σ
N. C,W,,, C((w) = N Σ (FF(
(1)
K where a₁ N.EE N₁. (xF y,) is the task-specific loss function and Fw(( is the formed model. The main difference that renders the problem difficult is we can only compute (ww)) on device protect the privacy for the user. The common setup (see more details in Beutel et al.
3

---

Under review as a conference paper at ICLR 2023
2020 introduces server to receive gradients sent from each client device, and therefore brings two major challenges: communication cost and data heterogeneity.
Personalization. Minimizing Eq. explicitly optimizes the generalization of the shared global model w across all clients. However, the model performance on individual clients might still be sub-optimal, especially under client data heterogeneity. Clients often care more about personalized performance (i.e., overfitting to client data). Therefore, given the outcome of global federated model learning in Eq. | denoted by w", each client can then further fine-tune the parameters locally to obtain personalized parameter Wi
N {(Fw- (x)),
Wy = arg min ac
(2)
3.2 CHAAMETER-EFFICEER LEARNING
Parameter-efficien learning a typical strategy for adapting pre-trained FMs downstream tasks. The full model size of an FM is often much larger than the size of downstream task data making the fine-tuning prone to overfitting. Not mentioning the back- pack-propagation over the full FM is extremely
expensive. To this end, multiple PE learning methods Houlsby al. (211);; Rebuffi et al. (2017)
Tomanek et al. (202T): Li and Liang (202T); 1) Lester et al. (202 Zaken et al. (2021) have been
proposed for fast adaptation of FMs, in which the learning objective is typically formulated as
1 M
WPB arg min
f( ww. wa.. (x)), 8/).
where WFM corresponds the frozen pre-trained foundation model, wpE the weight associated the introduced parameter-efffcien module and {(x, y) are the M data pairs from the downstream task of interest. Our goal in this paper is to design such lightweight module for Transformer-based FMs in the context of FL.
4 ADAPTING TRANSFORMER FMS INTO FEDERATED LEARNING
The motivation of our work is to treat federated learning as a downstream task for adapting pre- trained Transformer FM. In the following sections, we will introduce the two main modules, includ- ing a pre-trained Transformer and our attention- adapter Accumulator, and how Accumulator modulates the outputs of Transformer to support the early predictions. The overview of model architecture is depicted in Figure 2
4.1 TRANSFORMER MODEL.
A Transformer model typically consists of a sequence of residual blocks of multi-head head self-attention (MSA), each followed by residual block of feed-forward multilayer perceptron (MLP) with Layer- Norm (LN) applied to both MSA and MLP blocks. Denote by x the input, p the positional encoding,
and z' 1 2l. I the intermediate tokens, the feed-forward pass of Transformer can be
formalized as
z Tokenizer(x) P.
(4)
z MSA(LN(((-1) 2/11, 1=1...
(5)
z' MLP(LN(Z')) +Z' 1 I....L
(6)
4.2 ACCUMULATOR
To adapt a preitranned Transformer model, we inject our Accumulator into each self-ateenioo block followed by shared MLP head to enable early predictions. Formally, by collecting the history of
Note that there only Accumulator that handles the outputs from all layers.
4

---

Under review as a conference paper at ICLR 2023
Early Exit 0
Early Exit
Early Exit
Client token
Shared
Shared
Accumulato
0
CLS token
Self- Self-attn attn Block 0
Self-attn Block 1
Self-attn Block
Model architecture of the proposed Accumulator. Given a frozen Transformer FM (in grey), the Accumulator aggregates the information from the history of the CLS tokens through the transformation, which yields better feature representation for early exits at different layers. The early exit slots are numbered (there are three exits in this example), and the corresponding path- ways are denoted by different line types (c.g., dotted for exit 0 and dashed for exit 1). A residual connection is introduced to each early exit between the CLS token at the exit and the output of the Accumulator. A special CLS token (or referred to as Client token) is introduced to be the first token of the Accumulator, which can be tuned for personalization while freezing the other parts.
Figure 2:
CLS tokens, we replace zcts by
2 1/41 with h' Accumulator, ((zetient 2d1s + Po:..... Rcss P().
(7)
where the Accumulator parameterized by Φ is another Transformer (randomly initialized) with single self-attention block as defined by Eq. and hi41 means the / + element h'. In our Accumulator, the positional embedding p' = Po: .......... .P embeds the layer information from a pre- trained Transformer. Note that the modified ecss will be again used by the pre-trained Transformer, specifically, the next self self-attention block, produce z/+1.
For early predictions, instead of taking the modified <' we find it is better to make use of the client token and the original Z/1 specifically, the task-specific prediction at exit 7 is given by
g' MLLP-headp (h/s + =dwe)
(8)
where MLP-head parameterized by 1 is the early exits shared across all layers.
4.3 FEDERATED LEARNING ACCUMULATOR
Following the PE optimization in Eq 3 the learnable parameters wPF (Φ,, v) reduces to the weights of the Accumulator and the MLP head.
Learning. Now the learning problem in Eq. becomes
K
M
arg XIin COTE:
(00) Us). 15))
M
1=1
where 4 is the layer of an early exit for client i, which is set for each inference in client i. Assuming the client has the budget using up to L, layers, there are two schemes for choosing the value of 1; a) 4 L and b) 1, is taken uniformly at random from the range (1...., L,}. We attribute this property our formulation as anytime accommodate clients with different computing and memory capacities.
We find in Table that increases from single black 3 blocks yields little improvement in accuracy hut significantly heavier communication burden.
5

---

Under review as a conference paper at ICLR 2023
Personalization. There are two choices: fine-tuning the whole Accumulator (ó. eb) the Client token Sehient only. Given that they are both lightweight, the personalization is less likely to overfit even under an extremely low data regime.
Given an FL trained model in client i, it can infer labels g not only from block I, according its capability but also labels gl as long as 4 is smaller than the budget constraint L, This property can be very useful in the case when a cellphone suffers from low battery, which enables the user to set small budget 4 < L; to allow energy-efficient inference.
Inference.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Dataset Settings and Implementation. To verify the efficacy of our proposed FL frame- work, we conduct experiments on the standard FL benchmarks with Flower codebase Beutel et al.. (2020), including CIFAR- 18-0100 Krizhevsky et al. (2009). FEMNIST Cohen et al. (2017) and Speech- Command Warden (2018) as downstream tasks of pretrained DeiT Touvron et al. (2021) ) and AST Gong al.] (2021). J. We use the original test set in CIFAR-100
the global test dataset and split 10,000 images from the original training set as the personal validation set for each client when conducting the personalization experiments on CIFAR-100-C. We simulate three different data partitions for CIFAR-100, including one IID-data partition (a = 1000.0), 0), and two non-lID data partitions(a = 1.0, 0.1) by LDA partitioning following prior works Karimireddy et al. (2020): Wang et al. (2020a) with 100 clients. FEMNIST has total of 80,000 images (grayscale) of 62 different character classes (10 numeric, 26 lowercase, and 26 uppercase). For FEMNIST, we partition the training data in two ways, one III) and one non-III, as in Cohen et al. (2017) with 381 clients. The Speech Commands v2 dataset consists of 105,829, 16KHz I-second long audio clips of a spoken word. And we conduct "up". "down", "left", "right", "on", "off", "stop", "go" and one class "unknown" and a class "si- lence" which has no spoken word in the clip. The dataset has three disjoint sets of speakers: 2112 for training, 256 for validation and 250 for the test.
experiment with the 12-classes version, with 10 classes "yes", "no".
For each round in FL, sample 10% of clients for training CIFARI00 and FEMNIST, and 1% on SpeechCmdv2. We use pretrained DFIT-small Touvron et al. (2021) model with x 16 patches on IISVVRC-0122 Fei- et al. (2010) the foundation model, which be used for both image and speech recognition - i.c. the backbone of AST Gong et al. (2021).
All experiments are conducted with Pytorch on single Nvidia Tesla V100 GPU with results re- peated three times and reporting the mean and std. All models are trained using an SGD optimizer with a cosine annealing learning rate schedule. The detailed recipe can be found in the Appendix
Early Exits. Each client has its own computing capability. According to their capability, they can support certain amount of FLOPs for one inference, or one FL round. For example, given a pretrained Transformer, some clients may be able to pass the input data through all layers of the Transformer to train classifier. Some could only pass through half of them to the same end. Specifically, how many layers of a pre-trained transformer can be passed through during the local model training for a client depends on the client's computing capability. One can train a classifier head to make predictions for each layer to support the different compute capabilities among clients. These layer-wise classifier heads are often called early exits Lastsrass et al. (202T); Leontiadis et al. (2021). For example, we can straightforwardly train 12 carly exit classifiers for given pretrained DeiT-small, which has 12 layers. And then, those exits can be used for different clients accordingly. Briefly, we can treat 12 exits corresponding to 12-tiers of client compute capability namely, the different levels of computing budgets in terms of anytime inference.
Baselines. We run all our FL experiments with the FedAvg McMahan et al. (2017) strategy. We compare with all the baselines, namely, i) Fine-luming Qu et al. (2022), where the whole pretrained vision transformer is finetuned end-to-end in FL; ii) Layer-wise Lineur (L.W. W.W. Linear), where we append a learnable linear head after each transformer block to be learned by FL. while keeping the rest frozen; iii Layer-wise MLP (L.W. MLP), where we append two-layer MLP with GELU and Random Dropout after each transformer block, with the rest frozen. iv) Our Accumulator, where We also tried with shared linear head among layers, hut achieved worse results.

---

Under review as a conference paper at ICLR 2023
Table 1: Conventional FL performance.
CIFAR-100
FEMNIST
Method
SpeechCmdV2
IID (1000.0)
Non-lID (1.0)
Non-lID (0.1)
HID
Non-IID
Fine-tuning
85.05
84,91
84.25.11.14
86,87.10.74
85.28
98,22
Lincar head
74.37=00.42
73.85=0,74
72.79=0.883
74.02+1.33
72.35=0.52
69.47
PAA
83.28+00.67
82.57=1.12
81,41=1.228
12.3
81.60=0.66
94.74
MLP head
76.49-10.43
75,6920.81
74,44
74.2611.66
72.87
74.33
PA
84.55=00.69
13.87=1.25
82.33=#1.24
82,49:11.88
81,5440.78
95.63
Accumulator
84.90:10.75
8..3411.33
83,31:1.554 84.02:11.32
79.12:11.66
78.0550.75
93.24
PA
85.35:10.49
85.11=1.28
83.68:1.755
82.2410.84
95.27
Table 2: Anytime FL performance.
CIFAR 100
FEMNIST
Method
SpeechCmdV2
IID (1000.0)
Non-IID (1.0)
Non-IID (0.1)
IID
Non-IID
Fine-tuning
1..24+11.1
66.95+ 1.93
60.05+2..51
76.43+11.45
75.82+2.42
93.71
L. Linear
36.42.10.80
36.06.1..35
34.00.1..77
35.92.1..23
15.49
65.10
47.33-10.95
17.9411.53
47.11:1192
57.36:11.45
1..78.1.89
84.16
L.W. MLP
37.62.10.76
37.84
38.22.11.34
35.27.1..99
34.54
64.57
48.65.10.89
48.39.11.02
47.9411.211
55.2111148
54.42.12.1
83.19
Accumulator
64.28:t1.01
23.02:+:.799
57.34++2.23
1..47
75.12++..6
84.66
PA
65.23.11.26
64.28.11.88
58.40.11.67
76.55.11.63
76.03
88.30
we append one shared Accumulator and one shared MLP into each transformer block, with the rest frozen. We also compare the variants of L.W. Linear, MLP, and Accumulator with Parallel Adapters (PA), as proposed in He et al. (2021), injected into the feed-forward MLP networks of the pretrained Transformer their efficient and effective adaptation He et al. (2021).
Evaluation Settings. We evaluate all the methods in four different settings, including (1) Con- ventional Clients train the local models by using only the final exit of pretrained transformer, and the trained global model will be evaluated at a test set using only the final exit. (2) Anytime FL- Clients train the local models by using random exit each iteration, and the trained global model will be tested at each exit. (3) multi-tier FL Clients train the local models by using a specific early exit determined by the tier of each client. The trained global model will be tested each exit, and (4) Personalization, where the FL-trained model from setting (3) will be further finetuned using the local data in each client.
5.2 EXPERIMENTAL RESULTS
Conventional Federated Learning. Results in Table show the comparison between all com- petitors. The results show that fine-tuning works the best among all methods. This is unsurprising, as has access to the large combined dataset of all clients and can use this to tune the whole model, but incurs the most local training and communication costs. Among the other competitors, that our Accumulator achieves the best results already when used alone. Parallel adapters are quite effective in enabling federated adaptation of the pretrained Transfomer, boosting the performance of base methods L.W. Linear and MLP significantly in all settings, especially on SpeechCmdv2. Nevertheless, our Accumulator is complementary with parallel adapters, achieving the best result in all cases, with some results even surpassing the Fine-tuning method, such as on CIFAR- 100 with IID and NID- (1.0). The results demonstrate the efficacy of our proposed Accumulator to adapt pretrained Transformer model into FL at anywhere under any type of data heterogencity.
can see
Anytime Federated Learning. In this setting, we will consider the early exit situations. For each layer of pretrained DeiT, we train early exit, such linear head, an MLP head and our Accumulator. At each local training iteration, an exit layer index E j0...... ..1 will be sampled randomly, and only that exit (and accumulator where relevant) will be trained. For fine-tuning,
append the layer-wir wise MLP heads and train all parameters during the FL training. After training,
the global model will be evaluated the test set at all exits. The results in Table ? and Figure

---

Under review as a conference paper at ICLR 2023
Alpha=0.1
Alpha=1.0
Alpha=1000.3
80-
30-
70-
Accuracy
60-
40-
40-
30
Fre-taning
30
Accumaster
Arcumaaste
Arcumuiter
Accumdator7AA
Accumdato-1A
10
15
12
704
Accuracy
30
40-
20
1: MPP
30
LW NL7
LWl MLP
20-
LW MI
Accumdatrr11
Accamdator
Accimdator
Ascumdatrr11
10
Assimaidor
10
12
10 12
Early Exit Layer
Early Exit Layer
Early Exit Layer
Figure 3: Test accuracy of each exit trained on CIFAR- 100 with anytime federated learning (Top) and multi-tier constraint (Bottom). The memory and FLOPs cost of each exit is given in Table 7
Table 3: Multi-tier based FL performance.
CIFAR100
FEMNIST
Method
SpeechCmdV2
IID (1000.0)
Non-lID (1.0)
Non-1ID (0.1)
11D
Non-IID
Fine-tuning
58.17:0.552
56.70=0.83
42.47:1144
74.79:10.75
74.3621.36
93.03
L.W. Linear
32.43-10.34 30.9810.60
21.33.1097
35.68:10,54
35.1711.01
65.14
PA
46.96:10.41
45.1420.68
33.94:11003
57.06:10.60
54.67+1.15
83.23
L.W. MLP
36.57-10.32
34,3810,58
24.37.10.93
33.12.10.59
32.88
64.84
+ PA
51.33:10.47
49.2910.66
36.29:11007
51.88:10.67
51.39+1.27
82.28
Accumulator
58.27+00.56
57.32+0.88
47.26:11.18
12.14:00.72
11.33++.399
85.21
PA
59.33:10.72
58.48-10.9
48.55.11.30
72.67.10.81
72.05:11.57
87.42
top show the average and budget-wise performance over all exits. Without surprise, we can see that global fine-tuning achieves the best at substantial comms cost. Among the more efficient competi- tors, we can see that our Accumulator outperforms L.W. Linear and MLP significantly when no extra adapters are used. With parallel adapters, all methods enjoy performance boost, leading our Accumulator to the best performance in all cases, outperforming fine-tuning in few situations.
Multi-tier based Federated Learning. More realistically, there is a certain level system het- erogeneity among client devices. Individual devices in FL training have certain level of computing capability and their associated early exit should be persistently fixed in both training and testing. Results in the setting evaluated and reported in Table 3 where
that most results dropped to some extent compared with Table This is expected due to the existence of system het- erogeneity among clients. And most observations Table are similar to the previous tables. One interesting observation is that in this harder scenario, our Accumulator outperforms the fine-tuning baseline consistently in all situations in CIFAR-100. Figure 3 bottom shows the corresponding results for clients of different tiers on CIFAR- 100 after training by different algorithms. The accu- mulator+adapter architecture performs most favourably. This again shows our Accumulator works for anytime inference in harder and more practical scenario. We also provide communication efficency in Appendix
Personalization. Since clients' class and data distributions are typically different and diverse, adapting the globally trained model through per-elient personalization is an interesting task for FL. Based on the pertained model of multi-tier federated learning as the most practical scenario, we can adapt the model to the local data of each client efficiently. Specifically, the multi multi-tier FL pretrained model is fine-tuned for ten epochs for each client. We noisy data for personalization and the final test to simulate personal data distributions. For the corrupted CIFAR- 100, we simply use CIFAR- C. (Hendrycks and Dietterich 2019). To corrupt SpeechCmdy2 dataset, add 60% background
8

---

Under review as a conference paper at ICLR 2023
Table 4: Personalized performance on CIFAR-I00-C and corrupted SpeechCmdv2 with multi-tier constraints after FL training on clean data (figures in bracket show relative improvement to perfor- mance before personalization).
Update part for
CIFAR-100-C
SpeechCmdV2
personalization
IID (1000.0)
Non-IID (1.0)
Non-IID (0.1)
Bkg. Noise (0.6)
Full Model
34,30+ 185
37.75++2.2 (-76
26.94+3. 14
(0.2
L.W. Linear +PA
34.67 22 16-32100
37.55:t: 1. 1.7.5::t1.79..55500 79.
26.63:12.22
(622440
80.82
PA Only
34.061: 1.01
37.02:1 (
25.96 1. 1,58
81.40
L.W. MLP PA
31.6911442
36.95:1:2.05(13446
22.30 2100:229961.22 (2026))
80.59
PA Only
33.01=11.1..3.221
37.38 (.34
26.59: (2/424555)
80.82.+6.55)
Accumulator PA
38.23+11722-34.5
40.05-+42.344-10m
31.26++2. 28.
85.46-6647
Only
37.33+ 11.33+1.37---1 (377
39.45++ 1,48 vn
30.24++1
1.69
85.40-6641
CI fent Token Only
15.38+0.96.-1100
47.02+ 13:45mm
38.25 1.30
85.24(-620)
Table 5: Left: parameter number of different methods. Right: Ablation study of Accumulator.
Method
Parameter Num
Mean performance
Full Fine-tuning
30.62M
Full Accumulator
48.55
Parallel Adapter
0.60M
#sclf-attn blocks=3
48.86 (+0.31)
Layer-wise Linear
0.46M
No Replace
45.81 (-2.74)
Layer-wise MI.P
8.95M
No Residual
43.34 (-5.21)
Accumulator
3.17M
No Parallel Adapter
47.93 (-0.62)
Client Token
0.38K
Linear Head
(-3.95)
noise into each validation speaker and split them into two sets for personalization and the final test. From the results in Table 4 we can see that when tested on the corrupted data, the performances of all methods degraded. Now, petitors, including updating the full model during personalization. More interestingly. updating the cl innt token, which has extremely small set of parameters, i.e. 0.38K parameters according to Table 5 (Left), in our Accumulator, works overall the best among all situations. And also, when comparing the test accuracy gains after personalization, we can see our Accumulator produces the most, especially when the Client token is solely used for personalization fine-tuning.
Accumulator shows outstanding performance among all com-
Ablation Study, In order verify the contributions of different components proposed in our Accumulator, ablation studies were conducted based on the multi-tier based FL setup on CIFAR- 100 (a 0.1), the hardest scenario. T (right), the full accumulator indicates a transformer with depth=1, class token parallel connection, token replace mechanism, parallel adapter, and an MLP classification head. Among them, we find that parallel connection influences performance the most. This is mainly because DeiT's original class tokens are well learned on ImageNet, which are directly beneficial for model predictions reduces the effect of modulating the extracted features from DeiT. Thus the performance drops noticeably. And we also tried larger depth, i.e. 3, in the Accumulator, but seemed not to produce much performance gain. Our Accumulator works complementarily with parallel adapters used but apparently does not rely on them to have excellent performance.
the downstream tasks. Removing token replacing
6 CONCLUSIONS
We explored the challenges and opportunities of leveraging pre-trained models for downstream fed- erated learning tasks. While this might appear raise communication costs and preclude diverse device capabilities from participating, we show that after introducing a novel parameter-efficien adapter module we can simultaneously capture the benefits of communication efficient adaptation, non-lID robustness, support for diverse device capabilities, and robust personalization.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. (2017). Qsgd: Communication-
efficient sgd via gradient quantization and encoding. Advances in neural information processing systems, 30.
Beutel, D. J., Topal, T., Mathur, A., Qiu, X., Parcollet, T., and Lane, N. D. (2020). Flower: A friendly federated learning research framework. arXiv preprint vrXiv:2007.13390
Bommasani, R., Hudson, D. A.. Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021). On the opportunities and risks of foundation models. arXiv preprint 108.02200...228..
Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. (2017). Emnist: Extending mnist to handwrit- ten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 2921- 2926. IEEE.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Trans- formers for image recognition scale. arXiv preprint arXiv:2010.1199 1929.
Fei-Fei, L., Deng, J., and Li. K. (2010). ImageNet: Constructing large-scale image database. Journal of Vision, 9()): 1037-10777
Fu, F., Hu, Y., He, Y., Jiang, J., Shao, Y., Zhang, C., and Cui, B. (2020). Don't 't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript. In International Con- ference Machine Leurning, pages 3304-3314. PMLR.
Gong, Y., Chung, Y.-A., and Glass, J. (2021). AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571-575.
He, J., Zhou, C., Ma, X., Berg- Kirk-Kirkattrik, T., and Neubig, G. (2021). Towards a unified view of parameter-effiiient transfer learning. arXiv preprint arXiv:21 10.04366.
Heikkilä, M. A., Koskela, A., Shimizu, K., Kaski, S., and Honkela, A. (2020). Differentially private cross-silo federated learning. arXiv preprint arXiv:2007.0555:
Hendrycks, D. and Dietterich, T. (2019). Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint 1903 12261.
Horvath, S., Laskaridis, S., Almeida, M., Leontiadis, I., Venieris, S., and Lane, N. (2021). Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout, Advances in Neural Information Processing Systems, 34: 1486686--22889
Houlsby, N., Giurgiu, A., Jastrzebkk, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficiee transfer learning for nlp. In International Conference Machine Learning, pages 2790-2799 PMLR.
Jiang, Y., Wang, S., Valls, V., Ko, B. J., Lee, W.-H., Leung, K. K., and Tassiulas, L. (2022). Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Net- works and Learning Systems.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020). Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-52 5143. PMLR.
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
Laskaridis, S., Kouris, A., and Lane, N. D. (2021). Adaptive inference through carly-exit networks: Design, challenges and directions. In Proceedings of the 5th International Workshop on Embed- ded and Mobile Deep Learning, pages 1-6.
Laskaridis, S., Venieris, S. I., Kim, H., and Lane, N. D. (2020). Hapi: Hardware- aware progressive inference. 2020 /EEE/ACM International Conference On Computer Aided Design (ICCAD), pages 1-9. IEFE.
10

---

Under review as a conference paper at ICLR 2023
Leontiadis, I., Laskaridis, S., Venieris, S. L., and Lane, N. D. (2021). It's always personal: Using early exits for efficient on-device enn personalisation. Proceedings of the 22nd International Workshop Mobile Computing Systems and Applications, pages 15-21.
Lester, B., AI-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efffcient prompt tuning. arXiv preprini 2rXiv: 104.08691.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020). Federated opti- mization in heterogeneous networks. Proceedings of Machine Learning and Systems. 2:429-450
Li, W.-H., Liu, X., and Bilen, H. (2022). Cross-domain few-shot learning with task- adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7161-7170.
Li, X. L. and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint
McMahan, B., Moore, E., Ramage, D.,, Hampson, S., and y Arcas, B. A. (2017). Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273- 1282. PMLR.
Mishchenko, K., Malinovsky, G., Stich, S., and Richtárik, P. (2022). Proxskip: Yes! local gradient steps provably lead communication acceleration! finally! arXiv preprint 2rXiv:2202.09357.
Nguyen, J., Malik, K., Sanjabi, M., and Rabbat, M. (2022). Where to begin? exploring the impact of pre-training and initialization in federated learning. arXiv preprint 220..37700.15887.
Qu. L.. Zhou, Y.. Liang, P. P., Xia, Y., Wang. F., Adeli, E., Fei-Fei, L., and Rubin, D. (2022). Re- thinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
Rapp, M., Khalili, R., Pfeiffer, K., and Je (2022). Distreal: Distributed resource-aware learn- ing in heterogeneous systems, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8062-8071.
Rebuffi, S.-A., Bilen, H., and Vedaldi, A. (2017). Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30.
Rippel, O., Gelbart, M., and Adams, R. (2014). Learning ordered representations with nested dropout, In International Conference on Machine Learning, pages 1746-1754. PMLR.
Stich, S. U., Cordonnier, J.-B., and Jaggi, M. (2018). Sparsified sgd with memory. Advances in Neural Information Processing Systems, 31.
Tomanek, K., Zayats, V., Padfield, D., Vaillancourt, K. and Biadsy, (2021). Residual adapters for parfmeter-efficient ast adaptation to atypical and accented speech. arXiv preprint arXiv:2109.06952
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. (2021). Training data-efficient image transformers & distillation through attention. In Meila, M. and Zhang, T., editors. Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR.
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. (2020a). Federated learning with matched averaging. arXiv preprint arXiv:2002.064444
Wang, H.-P., Stich, S., He, Y., and Fritz, M. (2022). Progfed: effective, communication, and com- putation efficient federated learning by progressive training. In International Conference Ma- chine Learning, pages 23034-23054 PMLR.
Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. (2020b). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 1-7623.
11

---

Under review as a conference paper at ICLR 2023
Warden, P. (2018). Speech commands: A dataset for fimited-vocabulaty speech recognition. arXiv preprint MXXi::880.002000
Yu, S., Nguyen, P., Anwar, A., and Jannesari, A. (2021). Adaptive dynamic pruning for non-iid federated learning. arXiv preprint arXiv:2106.0663 06921.
Zaken, B., Ravfogel, S., and Goldberg, Y. (2021). Bitfit: Simple parameter-efficint fine-tuning
for transformer-basee masked language-models. arXiv preprint 206.::2006.10/99
12

---

Under review as a conference paper at ICLR 2023
Table 6:
Transmitted message size (# communication round model parameters (M) required to reach a target performance with multi-tier FL on CIFAR-100 and FEMNIST (best and second best).
L.W. Linear +Adapter
L.W. MLP +Adapter
Accumulator +Adapter
Fine-tuning
L.W. Linear
L.W. MLP
Accumulator
CIFAR-I00 100
90x30.62
1500x0.46
140x 1.06
610x8.95
165x9,55
40x3.17
100x3.77
FEMNIST
60x30.44
0.20
1.08
1500x8.77
240x9.37
40x
80x3.59
Table 7: The computational and memory budgets across all exit layers with Deit-S as the foundation model. The memory and compute footprint increases linearly with the layer at which the early exit triggered. During training. the memory peak can be largely reduced if the base model is kept frozen, allowing this way the participation of more constrained devices.
Early Exit Layer
0
2
3
4
5
6
7
8
10
11
FLOPP((BB)
0.21
0.40
0.57
0.67
0.95
1.13
1.31
1,50
1.69
1.88
2.07
2.67
Para
2.54
4.34
6.15
7.96
9.77
11.57
13.38
15.19
16.99
18.80
20.60
22.41
Mem. Peak(MM)
123
179
236
293
350
407
464
522
580
638
670
Mem. Peak(MB) Iroeen
62
134
170
206
242
279
315
352
A IMPLEMENTATION DETAILS
Hyperparameter Settings The base setup is the same for all experiments, and we basically did not include any extra training tricks to be able to demonstrate that our proposed framework can be successfully adapted with Anytime and Anywhere. For all datasets, the initial learning rate for the SGD optimizer set to 5e-3, and the batch size and training epoch on the client is set to 10 and respectively. The SandomResizedCeze FedAvg McMahan et al. 2017) is the default aggregation algorithm.
with scale=(0.05,, 1.0) is used training. The
Conventional Federated Learning In this setting, the last layer of the foundation model is set as the only exit. The full foundation model can be used for training the additional parameters, e.g. L.W. MLP, and Accumulator, etc. The baseline Fine-tuning indicates all parameters of the foundation model with an extra MLP head are free for tuning. The total training round is set to 500.
Anytime Federated Learning The major difference from the conventional FL is that the exit layer E [1, 12] for each client is randomly sampled from a uniform distribution for each training iteration. The total training round set to 1500.
Multi-tier based Federated Learning Exit layer is no longer produced by random sampling, but a fixed value that represents the current computing power of the client. In other words, each client will be assigned a permanent 1 during the anytime FL., The distribution of clients' exit layer is balanced. The total training round is set to 1500.
Personalization
Following the data corruption methods in Hendrycks and Dietterich 2019),
apply a corruption policy with different severity s [0, 5) as unique style for each client's local data. Then, fine-tune the pre- pre-trained multi-tier model with predefined exit layer these data for 10 local epochs.
B ADDITIONAL EXPERIMENTS
Communication Cost of Different Methods. Table the communication cost of different methods to reach a target test accuracy, which is set the best performance of the L.W. Linear head here. We can see from the results that our Accumulator as such achieves the lowest cost to reach the target by only cost 40 3.17 and 40 2.99 message size for CIFAR-100 (a 0.1) and FEMNIST(Non-III),
13


Reviews:
[{"review_id": "znN4SlBHnW-", "reviewer": "Reviewer_pRUk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors proposed to model clients in a federated learning (FL) setup as downstream tasks for large pre-trained models. The main component introduced by the authors is an accumulator module which is shared across the layers of a transformer and is learned based on a client token, CLS token and positional embeddings. To make the model suitable for clients with varying computational constraints the authors suggest to utilize exit points of the transformer. The authors evaluated their method against several natural baselines in this setting on common FL benchmarks. ", "summary_of_the_review": "Overall I think that using transformers in FL systems is a nice idea, but I think the paper has several flaws in the writing and experiments which makes it hard for me to recommend acceptance.", "strength_and_weaknesses": "Strength:\n1. As far as I know, and admittedly it is quite hard to keep track of all the papers in FL recently, this paper is the first to suggest a method for using transformers in FL systems.  \n2. The solution is relatively simple.  \n3. The comparison to other approaches that also use transformers is comprehensive, although I think that the overall experimental section is not sufficient. \n\nWeaknesses:  \nI found several problems with the paper and writing. I will group them into categories:\n1. Writing:  \na. It seems that the authors assume prior knowledge of concepts that are more common in the NLP/Vision-NLP communities (e.g., tokenizer, positional encoding, ViT architecture, CLS token, residual adapter). I believe that for many readers who engage in FL these concepts are not familiar (for example, FL is also very popular in the signal processing community). Hence, I think that an appropriate background should have been given in the appendix and the paper is incomplete without it.  \nb. The writing of the paper at times is very confusing. For example, already in the introduction, the authors present results with the accumulator, but it is not clear what it is and to what exactly it is compared until the method section. The authors refer to Fig. 2 on page 4 which in its description is talking about early exit, yet the explanation for what is early exit appears only on page 6.\n\n2. Experiments:  \na. Although, as I stated, the comparison within the realm of transformers is good, I think that the authors should try harder to compare to other methods as well. For example, using standard methods with a network that is pre-trained on some datasets (e.g., ImageNet). At the current state, it is hard to decouple the effect of the transformers, the pre-trained model, and the proposed solution.  \nb. The experiments with the corrupted CIFAR dataset are not clear to me. Which datasets are used and for what exactly? Also, I am aware of other studies that used corrupted CIFAR-100 in FL (see [1]), so it is not clear to me why not use these datasets and or compare to them if possible.  \nc. The authors show that their method has a low communication cost, which is great. I would like to understand better two other aspects of computationally which are also important in FL. How much memory does it take to train and make predictions with the proposed approach, and how much time does it take (wall-clock time) to make predictions on CPU and GPU? I wonder how it is compared to the standard FedAvg with simple architectures as commonly used.  \nd. Perhaps I missed it, but I didn't find an explanation on how did you search over the hyper-parameters of the model, or if a validation set was used (aside from the CIFAR-C experiments). In general, more details on the exact experimental section are missing. In the absence of code (which is a major flaw in the submission in my opinion) this information is important to assess the validity of the experiments.\n\n3. Minor:  \na. What is LDA partitioning? It is not mentioned in Hsu et al. (2019) paper as far as I can tell.  \nb. I think that Fig. 2 is not entirely clear.\n\n[1] Achituve, I., Shamsian, A., Navon, A., Chechik, G., & Fetaya, E. (2021). Personalized Federated Learning with Gaussian Processes. Advances in Neural Information Processing Systems, 34, 8392-8406.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qBtnVFgEzz", "reviewer": "Reviewer_Fwvb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to exploit pre-trained Transformer models in federated settings. Specifically, a novel attention-based adapter module is applied to collect information from each transformer layer/block, and then make an early prediction without needing all blocks.", "summary_of_the_review": "The paper proposes a lightweight adapter to be combined with a pre-trained Transformer. This design can not only to be applied to solve the NLP downstream tasks and also to be aligned to the Federated task. However, the learning objective of personalization is not clearly defined to be matched to the federated setting. ", "strength_and_weaknesses": "Strength\n\n1. This paper presents a technically sound method for efficient fine-tuning under a federated learning framework, which can reduce communication overheads and enable privacy preservation. Specifically, it regards a fixed pre-trained model as the feature extractor and uses a learnable aggregator (called Accumulator) with client indicators for deep representation learning.\n\n2. Empowered by the neural structure, the proposed method supports \"early exit\", which can speed up inference and suit a variety of devices. \n\n3. Compared with adequate baselines and competitors, the proposed method shows the best performance across several datasets. \n\nWeakness\n\n1. My biggest concern is the novelty of the proposed neural architecture. Essentially, regardless of the framework of federated learning, this work aims to introduce a new adapter for efficient fine-tuning. However, such an accumulator-style adapter can be seen as a simplified version of the Ladder Side-Tuning by  [1]. The simplification is done by only considering sequence-level features. \n\n2. This paper claims the proposed method can alleviate severe data heterogeneity to achieve the purpose of \"anywhere\", but I did not see any specific neural design or learning method to target the data heterogeneity issue. \n\n3. The paper is not well-written, making it not easy to follow and understand. For example, the passage under the title of Section 4 said that \"our attention-based adapter Accumulator\", but \"Adapter\" and \"Accumulator\" are regarded as different modules in the experiments. Meantime, please fix writing typos, e.g., \"Competiters\". The symbols in Eq 3 should be formally defined, such as M and alpha.\n\n4. The definition of personalization is not accurate. Personalization in the federated setting is learning many personalized models simultaneously rather than learning only one model for a client. Therefore, the definition of Eq (2) and (9) are not typical federated personalization objective functions. \n\n5. The experiment cannot support its claims. For example, in Table 1, the performance on FEMINST dataset is relatively too low compared to other work in the conventional FL settings.\n\n\nSome recent papers [2,3,4] could be discussed to enhance the related work section by introducing federated learning on pre-trained models. It is worth noting that because the target topic is very new while the related papers are published recently, thus, the missing of these references won't impact the rating of this submission. \n\nREFERENCE:\n\n[1] Sung et al., LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning.\n\n[2] Yuanyishu Tian, et al., When Federated Learning Meets Pre-training, ACM TIST 2022\n\n[3] Yue Tan, et al., Federated Learning from Pre-Trained Models: A Contrastive Learning Approach, arXiv 2022\n\n[4] Orion Weller, et al., Pretrained Models for Multilingual Federated Learning, NAACL 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jhMkG_0ooh", "reviewer": "Reviewer_eyXk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the federated learning problem. In particular, this paper aims to provide a unified framework for communication cost, robustness to data heterogeneity, and other challenges. To tackle these problems, this paper proposes an attention-based adapter module at each transformer block. In the federated optimization, only the lightweight adapter is trained. Empirical evaluation provides an analysis of the performance of heterogenous device capabilities, efficient personalization, and scalable-cost anytime inference.", "summary_of_the_review": "In summary, this paper focuses on an interesting problem, and the evaluation in computation and communication is solid.\n\nHowever, it would be better to have a better presentation of the paper with strong support for the claims.", "strength_and_weaknesses": "This paper raises an interesting topic in FL. However, this paper's presentation and evaluation do not provide enough support for its claims.\nIt would be better to provide justifications in the following domains.\n\n1. Pre-trained models: in most of the practical FL settings, the client device cannot afford the memory to store the pre-trained models. How does the proposed approach generalize to FL from scratch?\n\n2. Transformer models: this paper studies the transformer models for vision and acoustic models. Is it possible to provide an analysis of the language models, such as next-word prediction on the StackOverflow dataset? \n\n3. Device diversity: How does the proposed approach support the diverse device capabilities? Should each client maintain the same pre-trained transformer models? If so, is the device capabilities diverse enough?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xuxb_H5NoPZ", "reviewer": "Reviewer_sDyc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an efficient way to fine tune foundation models in a federated learning setting. The authors propose a light-weight transformer adapter, called accumulator, weight of which are only trained during the federated learning as opposed to all the weights in the original foundation model. The authors claim that the propose method outperform other state-of-the-art methods for efficient fine tuning of FMs in a federated learning setting. ", "summary_of_the_review": "The paper provides a novel approach to take the problem of training large foundation model in a federated setting manner. With some more explanation on experimental results it should be a good paper for ICLR audience. ", "strength_and_weaknesses": "The paper is quite well written in terms of clarify of the objective and applications. I also found the paper quite substantial in content as it tries to address both early exits (for inferencing)  as well as fine tuning of FMs in a single approach.\n\nHere are some of the main concerns or ways in which the paper can be improved...\n\na) What is the unit of the compute budget in Fig. 3? In its absence it's difficult to gauge what kind of devices can be supported with various techniques.\n\nb) Combining Table 1 and Table 5, it's not clear if the small improvement in accuracy is worth the additional number of parameters exchanged for the proposed approach as compared to the two other baselines. How do you justify that?\n\nc) I am quite surprised to see that there is very little difference in accuracy in IID vs non-IID setting in all cases in Table 1. Normally we see significant drop in accuracy of final model when trained in a non-IID federated learning setting. Can you comment on that?\n\nd) It would have been good to see some more baselines such as using a knowledge distillation based approach to train a smaller model from FM and aggregating that through FL. I wonder if you have any insights how such a baseline will perform. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject