Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper explores leveraging adapter in vision transformers. Similar to adapter-transformer in NLP, this is done by replacing the MLP layer in each transformer block with a adapter-MLP module. In finetuning, only a small number of parameters in the adapter-MLP is updated.  For image classification datasets, the proposed method AdaptFormer is on par or of lower performance than fully-finetuned model. For video datasets, where training set is usually in smaller scale than image classification datasets, this proposed parameter-efficient tuning technique achieves good performance, outperforming fully-funetuned models. Authors also conduct ablation study, e.g., the effect of #layers and the size of intermediate dimension of adapter-MLP module on the performance.
strengths_and_weaknesses: Strengths:
The proposed methodology is clearly presented. The ablation study and exploration is comprehensive. The parameter-efficient performance on video dataset is impressive.

Weakness:
1. In my opinion, the contribution of the proposed method is incremental. The adapter architecture is essentially the same as the one in the adapter paper [1]. Applying adapter in vision-related tasks, e.g., VQA and vision-grounding, have been explored in [2] and [3]. Actually, many different parameter-efficient tuning techniques other than adapter, e.g., Hyperformer and Compacter, have also been explored in these previous works.

2. The adapter module proposed in [1] is originally designed for NLP. Considering the spatial and channel-wise properties of vision inputs, I'm wondering have authors ever considered other adapter architectures that can be potentially more suitable for vision modeling?

Reference:
[1] Parameter-Efficient Transfer Learning for NLP, 2019
[2] VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks, CVPR
[3] Adaptive Fine-tuning for Vision and Language Pre-trained Models, NeurIPS
questions: Have you tested on ImageNet dataset, as in the original ViT paper?
limitations: Authors discussed it briefly in the conclusion section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an efficient adaptation approach for vision Transformer (ViT) that enables downstream tasks with limited parameters to tune. Specifically, the MLP in ViT is replaced by a new AdaptMLP, which keeps the original MLP branch but introduces a simple bottleneck branch for tuning. During the finetuning stage, the original MLP, multi-head self-attention and layernorms are frozen. In contrast, the weights of the additional branch in AdaptMLP and the last linear layer for classification are updated. Extensive experiments on both image and video recognition tasks indicate that the proposed approach achieves on-par or even better performance compared with a full-tuning setting and a significant improvement over the prior arts such as VPT.
strengths_and_weaknesses: 
Strengths:
1. This paper proposes a lightweight bottleneck branch to enhance the MLP in ViTs for efficient task transfer. The proposed approaches reuse (almost) all weights from pretrained ViTs that maximize the utilization of pretraining.
2. The proposed AdaptMLP is simple and can be easily applied to multiple variants of ViTs since the MLP module is usually kept intact in many ViT variants.
3. Empirical experiments show that the proposed AdaptFormer is efficient and effective: On image and video recognition tasks, the AdaptFormer obtains way better performance than VPT and linear probing. With less than 2% extra parameters, the AdaptFormer achieves on-par or even better performance than the full-tuning method, which requires all weights to be updated in finetuning.
4. Ablation study shows that the hyperparameter for the AdaptMLP (e.g., bottleneck middle dimension) is not very sensitive to the downstream tasks. It reduces the burden of engineering work on finding good combinations of hyperparameters in pretraining and finetuning. 
5. The paper is well written and easy to understand.

Weaknesses:
1. The efficiency of the proposed method is mainly measured in parameters throughout the manuscript. However, it remains unclear how much computation this method introduces in finetuning and inference. Thus, it would be better to include comparisons for finetuning time and additional latency during inference. 
2. Line 289 mentions that the accuracy reaches saturation when middle dimension = 64. Is the same middle dimension used for all tasks, or each task still has its own middle dimension? I am curious about this because the ablation study claims that the performance is insensitive to the (proper) choice of the middle dimension.
3. In line 216 Initialization of AdaptFormer, it said that "the weights of down-projection layers are initialized with Kaiming Normal [33], while the biases of the additional networks and the weights of the up-projection layers are configured with zero initialization." I wonder if there is any reason behind such choice of initializations.
4. Table 2(c) scaling factors s is not well analyzed in the paper.
5. Other minor issues: Line 308-309 ourperforms -> outperforms, methhod -> method
questions: See Weaknesses for most questions.

I have an additional question/suggestion here: In AdaptMLP, the MLP and the bottleneck module are parallel. Is it possible to combine them into a single MLP (e.g., unifying the activation function and then merging the weights)?  It would be more efficient during the inference in downstream tasks since GPUs/TPUs are more optimized for a single branch rather than parallel branches.
limitations: The authors have discussed limitations (e.g., the ability to extend to tasks beyond recognition) and claimed no potential societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper propose a simple method to adapt a pre-trained model to down-streaming tasks with most of the parameters frozen. The authors propose to use a SE attention like extra branch to adjust the MLP block when fine-tuned on downstream tasks. Overall, the paper is well written and the experiments are comprehensive. However, some of the experiment settings are not clear at the moment. 
strengths_and_weaknesses: Strength:

1. the paper is well motivated. Adaptation of large pre-trained ViT models is an important topic to make the recent popular ViT models more useful.
2. the paper is well written and easy to follow. The presentation is clear and the proposed method is well illustrated.
3. the experiments results are promising. The proposed method achieves significantly better results over the full finetuning baselines.

Weakness:

1. the settings of the full fine-tuning are not clear. The results of AdaptFormer uses pre-trained weights on ImageNet-21k. Does the full-tuning baseline also uses the pre-trained weights on large dataset? 

2. I was not clear on why the proposed method could achieve better results with most of the backbone parameters frozen. With more parameters tunable, the full-tuning should have higher representation capability. This counter intuitive results are not well explained in the paper.
questions: 1. Based on the results shown in table 1, the ViT-B model only achieves 85.99% accuracy on CIFAR100. However, in other papers such as DeiT, a pre-trained ViT model can easily achieve 90%+ accuracy.

2. Why the proposed methods achieve much better accuracy than the full-tuning baseline with much less number of tunable parameters?

3. The main motivation of the paper is the huge memory and computation cost of the traditional full-tuning method. How much of memory and computations could the proposed method save on those aspects?

4. Since the authors are using the ImageNet-21K pretrained weights, why no experiments are conducted on ImageNet-1k dataset?
limitations: NA.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper introduces an effective adaptation approach for vision Transformer which avoids the training cost of large-scale parameters in downstream task fine-tuning. In detail, the AdaptMLP is devised by including an encoder-decoder branch into the original MLP layer in the self-attention block. During the fine-tuning of the downstream task, only the parameters of the AdaptMLP module are optimized and all other parameters are frozen. Experiments of the inner-domain (image/video) pretraining task with five different downstream datasets and the cross-domain pretraining task between ImageNet-21k and SSv2 datasets, validate the proposed approach.
strengths_and_weaknesses: [Strengths]

1.	The paper forms well and the technical implementation details are clear. 
2.	It is a good direction to reuse the parameters of the vision Transformer pre-trained on the ImageNet for the downstream tasks. The problem is very close to the real-world application and much valuable in the research community. 
3.	The performances on the image and video classification tasks give some clues that the proposed AdaptMLP module can learn some task-specific knowledge to enhance the feature learning. 

[Weakness]

1.	The biggest issue for this work is about the design of AdaptMLP. The rationale behind the newly-proposed encoder-decoder branch is unclear. The technical part from 152-165 only shows the implementation details and it is very hard to judge the technical contributions for this module. 
2.	The evaluated downstream image datasets (e.g., CIFAR-100 and SVHN) are very small and the relative image contents are not diverse. The classification performances on these two datasets are saturated. Thus, the evaluations are not very convincing in my opinion.  
3.	In the experimental section, there should be more analysis rather than only describing the results. For instance, in the investigation of the middle dimension in the encoder-decoder block, there is no point that gives some insights to readers how reasonable the choice it is. The dimension of 64 is the best and I don’t know why.
4.	The t-SNE visualization in Figure 8 cannot verify the effectiveness of AdaptFormer compared to the full-finetune since the (c) and (a) are almost the same.  

questions: 1.	What is the rationale behind the design of the AdaptMLP module? Dose the structure of encoder-decoder perform the best? How about other structures for the design of this module? Is there any empirical reason or theory support such design?
2.	How about the generalization ability of the AdaptFormer on large-scale image datasets, e.g., NUS-WIDE?
3.	Why the Transformer pre-trained on ImageNet-21k performs such poor (41.5%) on Something-Something V2 datasets? In action recognition research filed, the model pre-learned on image datasets (i.e., ImageNet) will even perform better than the model pre-learned on video dataset (e.g., Kinetics). The performance is questionable. 

limitations: The major limitation of this work is that the rationale behind the AdaptMLP module is unclear. There is no theory or reference mentioned in the paper that validates the reasonability of such module design. Only listing the performance gains is not enough. The evaluation is not very convincing since the downstream image datasets are very small and not very diverse.
review_title: null

Meta-review: metareview: Authors introduce lightweight parallel FC layers to the MLP layers for fine-tuning, freezing all other parameters of the model. Experiments across 5 datasets (3 Image: CIFAR-100, SVHN, Food-101 -- 2 Video: SSv2, HMDB51) show that performance is similar to full-finetuning while changing less than 2% of the parameters. Some performance improvements are noted in the video domain.

Pros:
- [R/AC] Paper is well written
- [R/AC] Ablation study is comprehensive
- [R/AC] Performance is impressive
- [R/AC] Efficiency gains for finetuning are significant.
- [R/AC] Modifications are simple and can be applied to variety of ViT architectures.

Cons:
- [R/AC] The novelty is low. In regards to its motivation, the work shares a great deal of similarity to LORA. In regards to its technical approach, only minor adjustments are made in comparison to LORA, with the studied application domain being vision. 
- [R/AC] Authors do not provide sufficiently clear description on how learning parameters were chosen across AdaptFormer and full fine-tune approaches. Were parameters independently optimized, or fixed? 
- [AC] Method seems very sensitive to weight initialization and scaling mixing parameter s. 
- [R] Efficiency was mainly measured as parameters. More information regarding latency and FLOPS should be added. Authors have addressed this concern by including additional information, and should ensure that this new information is in the final manuscript.
- [R] Motivation behind different weight initialization schemes is not clear. Authors have sufficiently answered with more information.
- [R] Need more analysis on why 64 dimensions is the optimal point. What is the explanation? Authors didn't really answer this question other than providing an ablation. Insight behind result is not clear.
- [R] Evaluation is not thorough enough. Some of the datasets used are saturated. Authors added one dataset (NUS-WIDE) to appendix.
- [R] Rationale of design is unclear. [AC] This approach is from a class of similar approaches previously studied. Reviewers also answered this part well.


Reviewer consensus is that this work in its current form is borderline, but the majority of reviewers lean toward accept. AC has concerns about novelty of the method and missing details about how finetuning parameters were chosen across the various experiments. AC will maintain reviewer tendency toward accept, though will comment that the work can be improved if full details about how fine-tuning parameters were chosen across experiments, and that authors optimized those parameters independently in each experiment.

AC Rating: Borderline Accept
confidence: Less certain
Decision: Accept