Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to exploit pre-trained Transformer models in federated settings. Specifically, a novel attention-based adapter module is applied to collect information from each transformer layer/block, and then make an early prediction without needing all blocks.
strength_and_weaknesses: Strength

1. This paper presents a technically sound method for efficient fine-tuning under a federated learning framework, which can reduce communication overheads and enable privacy preservation. Specifically, it regards a fixed pre-trained model as the feature extractor and uses a learnable aggregator (called Accumulator) with client indicators for deep representation learning.

2. Empowered by the neural structure, the proposed method supports "early exit", which can speed up inference and suit a variety of devices. 

3. Compared with adequate baselines and competitors, the proposed method shows the best performance across several datasets. 

Weakness

1. My biggest concern is the novelty of the proposed neural architecture. Essentially, regardless of the framework of federated learning, this work aims to introduce a new adapter for efficient fine-tuning. However, such an accumulator-style adapter can be seen as a simplified version of the Ladder Side-Tuning by  [1]. The simplification is done by only considering sequence-level features. 

2. This paper claims the proposed method can alleviate severe data heterogeneity to achieve the purpose of "anywhere", but I did not see any specific neural design or learning method to target the data heterogeneity issue. 

3. The paper is not well-written, making it not easy to follow and understand. For example, the passage under the title of Section 4 said that "our attention-based adapter Accumulator", but "Adapter" and "Accumulator" are regarded as different modules in the experiments. Meantime, please fix writing typos, e.g., "Competiters". The symbols in Eq 3 should be formally defined, such as M and alpha.

4. The definition of personalization is not accurate. Personalization in the federated setting is learning many personalized models simultaneously rather than learning only one model for a client. Therefore, the definition of Eq (2) and (9) are not typical federated personalization objective functions. 

5. The experiment cannot support its claims. For example, in Table 1, the performance on FEMINST dataset is relatively too low compared to other work in the conventional FL settings.


Some recent papers [2,3,4] could be discussed to enhance the related work section by introducing federated learning on pre-trained models. It is worth noting that because the target topic is very new while the related papers are published recently, thus, the missing of these references won't impact the rating of this submission. 

REFERENCE:

[1] Sung et al., LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning.

[2] Yuanyishu Tian, et al., When Federated Learning Meets Pre-training, ACM TIST 2022

[3] Yue Tan, et al., Federated Learning from Pre-Trained Models: A Contrastive Learning Approach, arXiv 2022

[4] Orion Weller, et al., Pretrained Models for Multilingual Federated Learning, NAACL 2022

clarity,_quality,_novelty_and_reproducibility: The paper’s major contribution is a lightweight framework for Transformer, and then is applied to a Federated setting. The paper is well written. There is no source code to evaluate the reproducibility. 

summary_of_the_review: The paper proposes a lightweight adapter to be combined with a pre-trained Transformer. This design can not only to be applied to solve the NLP downstream tasks and also to be aligned to the Federated task. However, the learning objective of personalization is not clearly defined to be matched to the federated setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an efficient way to fine tune foundation models in a federated learning setting. The authors propose a light-weight transformer adapter, called accumulator, weight of which are only trained during the federated learning as opposed to all the weights in the original foundation model. The authors claim that the propose method outperform other state-of-the-art methods for efficient fine tuning of FMs in a federated learning setting. 
strength_and_weaknesses: The paper is quite well written in terms of clarify of the objective and applications. I also found the paper quite substantial in content as it tries to address both early exits (for inferencing)  as well as fine tuning of FMs in a single approach.

Here are some of the main concerns or ways in which the paper can be improved...

a) What is the unit of the compute budget in Fig. 3? In its absence it's difficult to gauge what kind of devices can be supported with various techniques.

b) Combining Table 1 and Table 5, it's not clear if the small improvement in accuracy is worth the additional number of parameters exchanged for the proposed approach as compared to the two other baselines. How do you justify that?

c) I am quite surprised to see that there is very little difference in accuracy in IID vs non-IID setting in all cases in Table 1. Normally we see significant drop in accuracy of final model when trained in a non-IID federated learning setting. Can you comment on that?

d) It would have been good to see some more baselines such as using a knowledge distillation based approach to train a smaller model from FM and aggregating that through FL. I wonder if you have any insights how such a baseline will perform. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and novel.
summary_of_the_review: The paper provides a novel approach to take the problem of training large foundation model in a federated setting manner. With some more explanation on experimental results it should be a good paper for ICLR audience.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the federated learning problem. In particular, this paper aims to provide a unified framework for communication cost, robustness to data heterogeneity, and other challenges. To tackle these problems, this paper proposes an attention-based adapter module at each transformer block. In the federated optimization, only the lightweight adapter is trained. Empirical evaluation provides an analysis of the performance of heterogenous device capabilities, efficient personalization, and scalable-cost anytime inference.
strength_and_weaknesses: This paper raises an interesting topic in FL. However, this paper's presentation and evaluation do not provide enough support for its claims.
It would be better to provide justifications in the following domains.

1. Pre-trained models: in most of the practical FL settings, the client device cannot afford the memory to store the pre-trained models. How does the proposed approach generalize to FL from scratch?

2. Transformer models: this paper studies the transformer models for vision and acoustic models. Is it possible to provide an analysis of the language models, such as next-word prediction on the StackOverflow dataset? 

3. Device diversity: How does the proposed approach support the diverse device capabilities? Should each client maintain the same pre-trained transformer models? If so, is the device capabilities diverse enough?

clarity,_quality,_novelty_and_reproducibility: The approach in this paper is well-introduced. The evaluation on the computation budget as well as the choice of adapter is also well-presented.
summary_of_the_review: In summary, this paper focuses on an interesting problem, and the evaluation in computation and communication is solid.

However, it would be better to have a better presentation of the paper with strong support for the claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors proposed to model clients in a federated learning (FL) setup as downstream tasks for large pre-trained models. The main component introduced by the authors is an accumulator module which is shared across the layers of a transformer and is learned based on a client token, CLS token and positional embeddings. To make the model suitable for clients with varying computational constraints the authors suggest to utilize exit points of the transformer. The authors evaluated their method against several natural baselines in this setting on common FL benchmarks. 
strength_and_weaknesses: Strength:
1. As far as I know, and admittedly it is quite hard to keep track of all the papers in FL recently, this paper is the first to suggest a method for using transformers in FL systems.  
2. The solution is relatively simple.  
3. The comparison to other approaches that also use transformers is comprehensive, although I think that the overall experimental section is not sufficient. 

Weaknesses:  
I found several problems with the paper and writing. I will group them into categories:
1. Writing:  
a. It seems that the authors assume prior knowledge of concepts that are more common in the NLP/Vision-NLP communities (e.g., tokenizer, positional encoding, ViT architecture, CLS token, residual adapter). I believe that for many readers who engage in FL these concepts are not familiar (for example, FL is also very popular in the signal processing community). Hence, I think that an appropriate background should have been given in the appendix and the paper is incomplete without it.  
b. The writing of the paper at times is very confusing. For example, already in the introduction, the authors present results with the accumulator, but it is not clear what it is and to what exactly it is compared until the method section. The authors refer to Fig. 2 on page 4 which in its description is talking about early exit, yet the explanation for what is early exit appears only on page 6.

2. Experiments:  
a. Although, as I stated, the comparison within the realm of transformers is good, I think that the authors should try harder to compare to other methods as well. For example, using standard methods with a network that is pre-trained on some datasets (e.g., ImageNet). At the current state, it is hard to decouple the effect of the transformers, the pre-trained model, and the proposed solution.  
b. The experiments with the corrupted CIFAR dataset are not clear to me. Which datasets are used and for what exactly? Also, I am aware of other studies that used corrupted CIFAR-100 in FL (see [1]), so it is not clear to me why not use these datasets and or compare to them if possible.  
c. The authors show that their method has a low communication cost, which is great. I would like to understand better two other aspects of computationally which are also important in FL. How much memory does it take to train and make predictions with the proposed approach, and how much time does it take (wall-clock time) to make predictions on CPU and GPU? I wonder how it is compared to the standard FedAvg with simple architectures as commonly used.  
d. Perhaps I missed it, but I didn't find an explanation on how did you search over the hyper-parameters of the model, or if a validation set was used (aside from the CIFAR-C experiments). In general, more details on the exact experimental section are missing. In the absence of code (which is a major flaw in the submission in my opinion) this information is important to assess the validity of the experiments.

3. Minor:  
a. What is LDA partitioning? It is not mentioned in Hsu et al. (2019) paper as far as I can tell.  
b. I think that Fig. 2 is not entirely clear.

[1] Achituve, I., Shamsian, A., Navon, A., Chechik, G., & Fetaya, E. (2021). Personalized Federated Learning with Gaussian Processes. Advances in Neural Information Processing Systems, 34, 8392-8406.
clarity,_quality,_novelty_and_reproducibility: * I think that the paper is not clear enough (see Strength And Weaknesses section). 
* There is some novelty, mainly in the idea, yet there is no real technical novelty. 
* It is not reproducible (see Strength And Weaknesses section).
summary_of_the_review: Overall I think that using transformers in FL systems is a nice idea, but I think the paper has several flaws in the writing and experiments which makes it hard for me to recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a federated learning algorithm in the setting where each client is learning a transformer model. With the goal of efficient training, the paper presents a method which takes pre-trained transformers and adds an attention-based adapter module at each transformer block. The paper also attempts at learning a personalized model at each client.

The paper received mixed reviews initially. Some of the key concerns raised by the reviewers (some of which are shared by me too) were:

- The idea of using adapters in federated learning has been used in prior works.
- The idea of solving the data heterogeneity problem used in this paper doesn't involve much architectural novelty
- The personalization scheme used in the paper is rather basic, and the paper lacks clarity about how it is being done.
- The paper lacks clarity at various places.
- It also feels that the paper is trying to do too many things at once - handling heterogeneity, personalization, early exit - and each of these aspects individually is not looked at rigorously in the paper.

The authors' response was discussed. However, even after the discussion, some of the concerns still lingered and, in the end, no reviewer championed the paper strongly. 

The paper is also missing some relevant recent works on federated learning and personalization using adapters, such as "Federated Learning with Partial Model Personalization" (Pillutla et al, ICML 2022)

Considering these aspects, the paper falls short of the acceptance criterion. The paper does have some interesting ideas such as early exit. However, in terms of overall contribution (technical novelty and experimental evaluation), it falls short. The authors are advised to incorporate the reviewers' feedback and more clearly place their work in comparison to the existing work on this topic.

The paper also has some other minor issues, such as

- The notation w_* in Eq 2 (personalization) is confusing. I suppose  w_*  refers to the global model which is then fine-tuned to get the optimal client-specific model. However, the way it is written, Eq 2 doesn't quite reflect that. In the rebuttal, the authors try to answer this (after Reviewer Fwvb raised this issue) but the lack of clarity still remains.
- The acronym "FM" is used at various places but hasn't been defined (I guess it means "foundation models" but should be defined).
justification_for_why_not_higher_score: Reasons discussed in the meta-review.
Decision: Reject