Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a simple linear adapter for parameter-efficient fine-tuning. Unlike previous methods that involve complex non-linear operations or prompting, they explore linear feature scalability of vision transformers and propose LIFTs-ADA. Furthermore, their linear adapters can be merged or subsumed into the original modules of the network during inference stage making them not requiring any additional parameters and FLOPs. Their experiments show that their proposed method performs at par or better than previous competing methods such as VPT with substantially less number of parameters during fine-tuning for downstream tasks.



strengths_and_weaknesses: Strengths

1. Method is simple, effective and easy to use.
2. Inference stage does not require any additional parameters and FLOPs, thereby reducing computational costs. They also provide numbers (#params and #FLOPs) for comparison with another parameter-efficient fine-tuning method such as VPT.
3. Authors have performed a comprehensive set of experiments with various datasets and downstream tasks to support their method and claims.
4. Authors also perform a series of ablations to understand the effectiveness of locations for placing their linear adapters.

Weaknesses
1. Paper is poorly written in some places. Some specific lines 103, 175, 176, 177. In general the paper lacks the written maturity of a Neurips paper.
2. Lack of theoretical grounding. But their simplicity of idea and empirical effectiveness seem to be sufficient according to me.
3. Lack of study on robustness and OOD performance.

questions: 1. Please review and explain this section better (lines 175-177)

Here, we consider that g is a linear function for its simplicity of the linear function but the lack of comprehensive investigation and the properties of the linear function make it possible to be merged with other operations of the network in the inference phase.

I do not understand this line. Looks like 2 unrelated sentences have been merged.

2. The claim (lines 27-27) that the you do not have to store the pretrained seperately for each fine-tuned model seems flawed. If the Linear adapters are subsumed once into the pre-trained network during inference stage, they you would have to do this for each downstream task and store a different model for each downstream task. 
limitations: 1. Lack of results on robustness or OOD performance.
2. Writing quality at some places is not up-to the standards.
3. Lack of theoretical grounding (but this can be excused).
4. For each downstream task, you still have to store a separate model since the parameters of the linear adapters are subsumed into original parameters of the pre-trained network thus updating the parameters effectively.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this submission, instead of optimizing all the parameters to adapt to a new task, the authors propose a LInear Feature Scalability (LIFTs) method that keeps all the model parameters from the pre-trained network but only learns the task-specific linear feature scalability adaption layer for each block. In this way, the authors alleviate the dilemma of useful information elimination and expensive computation burden as in previous all-parameter-optimized fine-tuning approaches, making the model readily and effectively reusable across various tasks. Experiments on both the fine-grained visual classification and general classification datasets convincingly demonstrate the superiority of the proposed LIFTs over prior works. Further results on segmentation and detection are also provided in the supplement.
strengths_and_weaknesses: [Strengths]
- The proposed module is plug-and-play, which is easy to be incorporated in various architectures. The motivation is clear. The authors analyze the flaws of the existing full-parameter fine-tuning methods in details, and then demonstrate their solution of LIFTs accordingly.

- The detailed algorithm workflow with the key code is provided, which makes it easier to understand the method details. Detailed discussions on the complexity are given in the method section, which is critical to understanding the contributions of the proposed LIFTs. The paper is easy to follow. The illustrations are quite clear, by firstly giving a definition of the key term of the feature scalability in this paper.

- Extensive ablation studies and visualization results are provided, by exploring the impacts of various designs. 

[Weaknesses]
- Please explicitly denote in Tab. 4 on what device the running time is obtained.

- The authors are suggested to use the same font across all the figures. 

- It will be great if the authors could discuss the limitations of the proposed method, which may showcase the future directions on this line of research to some extent.

- Grammars: Ln 329 “ … which makes the network applied to downstream tasks not require any additional parameters and FLOPs …”


questions: - Could you analyze the limitations of the proposed method? It will be helpful for follow-up research on this line of fine-tuning schemes with the feature scalability.
limitations: Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A new fine-tuning method for vision transformer is proposed in this manuscript. The authors have explored the Linear Feature Scalability of vision transformers, which defined as the fine-tuning performance with only linear transformation of the feature map. A simple module named LIFTs-ADA is designed for a fine-tuning, where some additional linear transformation operations are injected after the MLP and MSA. Re-parameterization is designed to make the fine-tuning parameter-free during inference. It has shown its efficiency on different tasks and models.

strengths_and_weaknesses: Strengths:
1, The definition of feature scalability and the observation that applying only a linear transformation of the feature map can lead to great performances on different downstream tasks are interesting. 
2, The proposed method is quite simple yet efficient for vision transformers with respect to the naive fine-tuning and vision prompt fine-tuning. The additional FLOPS of the proposed method is indeed lower than that of the prompt-based method.
3, The Re-parameterization trick can neurally integrate the linear transformation learned for different tasks into the original linear layer of transformer, leading to a zero-overhead inference.
4, The ablation studies about different configuration of ADA is good.

Weaknesses:
1, Some parts of the manuscript is unclear. For example, line 195, A_2^2 should be A_i^2. It is unclear whether the transformation A is different among different attention heads within the same layer.
2, For the Fig. 3 (b), it is recommended to provide details explanation about how the similarity matrix is computed. Also, it would be interesting to show the dissimilarity matrix of the linear transformation before and after fine-tuning, since they are comparable after re-parameterization.
3, For Tab. 2, why the Tuned/Total rates are so different between NABirds dataset and others?
questions: Please refer to the weaknesses part
limitations: The limitations are mainly about the unclear statement of some parts and the insufficient analysis about the learning behaviors (the different visualizations of the linear transformation).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a parameter-efficient transfer learning method for vision transformer, named LInear Feature Scalability (LIFTs). It inserts a linear-structured adapter layer to the transformer block, while keeping other model parameters unaltered. The micro and macro insertion positions are carefully searched through experiments. The trained linear adapter layer can be fused into the original network structure by applying the reparameterization tricks, without additional computational overhead at inference time. LIFT shows promising results on a variety of evaluations.
strengths_and_weaknesses: Strengths:
1.	Simple but efficient design. The linear-structured adapter can be easily implemented and plugged into any existing vision transformer structure without hustle. 
2.	The proposed LIFT adapter takes the idea of reparameterization method, thus avoid the additional inference cost. 
3.	The performance improvement is encouraging.

Weakness:
1.	Since the paper propose a new adapter-based architecture for transfer learning, please compare the computational complexity (Table 1) and model performance (Table 3 and Table 4) with original adapter-tuning and recent vision adapter structures. I do not see the results in the paper.
questions: 1.	As the paper put a lot of effort on the where to insert the LIFT block, the current version seems to a simplified Neural Architecture Search with grid search strategy. For example, 3 macro designs, called LIFTs-Shallow, LIFTs-Shared and LIFTs-Deep are evaluated. But there are still other hybrid micro and macro designs that are untested. I am wondering whether the author tried some NAS techniques for such combinational optimization problem.

2.	In table 3, why not include the ViT-B structure for CIFAR-100 and ImageNet1k experiments? Since the Visual prompt tuning was originally designed for ViT-structured network, I believe it is more appropriate and important to report the results for CIFAR and ImageNet with ViT, since it provides a fair test bed for two methods.
limitations: Please provide some discussion on the limitations and societal impact in the paper.
review_title: null

Meta-review: metareview:  
 This paper provides a simple method to avoid full fine-tuning of vision transformers, namely very simple linear adapters that can be trained and then subsumed into the existing linear layers during inference, which is an interesting characteristic as it prevents added computation during inference (unlike the use of regular adapters as used in NLP). Overall the reviewers appreciated the simplicity and intuition of the method, the improvement in performance over other competing methods such as VPT, the avoidance of computation overhead during inference, and comprehensive experiments. 

  There were some concerns, however, related to the writing of the paper and clarity, robustness on OOD data, complexity analysis/details of runtime, and lack of theoretical justification. Many of these were addressed by the authors, including nice robustness/OOD results which add to the experimental validation. After the rebuttal, the reviewers all agreed on acceptance. While the method is still empirical (the hypothesis related to distribution matching seems unsubstantiated, and there are many other ways to do that, and so should probably not be in the paper), the paper has a strong empirical execution that uses a simple method to deal with limitations that have significant societal/deployment consequences. As a result, I recommend accepting this paper. 
confidence: Certain
Decision: Accept