Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the adaptation of the well-trained transformer to downstream tasks on resource-limited devices. The authors introduced grouped connections with re-parameterization technology into the training process. The experimental results on serval vision tasks demonstrate the effectiveness of the proposed method.

strength_and_weaknesses: + Compared to the baselines and other methods, the proposed methods could bring some improvements in accuracy and computation cost.
+ Some ablation experiments are introduced to prove the effectiveness of the proposed method.

The main concerns are listed below. 
- Strictly restricted application scenario. The author introduces the application scenario in the abstract where the resource-limited devices cannot store a full copy of parameters but could provide computation to train/fine-tune the large models. It is a bit strange the devices with the ability to train DL models do not have sufficient storage.
- The proposed method is irrelevant to Transformer. It is closer to the re-parameterization technology. 
- Some recent literature on the adaptation of Transformers is missing.

[a] AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition, NeurIPS 2022.
clarity,_quality,_novelty_and_reproducibility: I'm Ok with the quality, clarity, and originality of the work. 
summary_of_the_review: As written in the Strength And Weaknesses, the strictly restricted application scenario lowers the practical value of the proposed method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a consolidator that modifies the pretrained model with the addition of a small set of tunable parameters to store the task-specific knowledge while freezing the backbone model during adaptation. A two-stage consolidation process is designed by merging corresponding parameters in the training-storage phase and loading-inference phase. Extensive experiments are conducted on various downstream tasks and the results outperform state-of-the-art methods with fewer stored parameters but superior performance. 
strength_and_weaknesses: Strength:

(1) The authors propose a basic module, a consolidator for efficient transfer learning.

(2) A two-stage consolidation process is designed. 

(3) Extensive experiments on various downstream tasks show state-of-the-art performance.

Weaknesses:
This work aims to design a mergeable adapter for efficient transfer learning and proposes a two-stage consolidation process by merging corresponding parameters. I have some questions as follows. 

(1)  In Figure 1, the authors split channels and use zero padding, which might cause a heavy computational cost. Thus, the training time/speed should be shown for comparison with other tuning methods. Besides, I notice that the authors insert a consolidator in the FC layer in Figure 1. Will performance be further enhanced if the consolidator is inserted in other locations? Some recent tuning methods [1, 2] should also be discussed and compared in Table 1. 

(2)  Some other tuning baselines are missing in Table 4. The authors should conduct more experiments with other tuning methods for fair comparisons. 

(3)  The consolidator is designed for a linear layer. Can it be applied to other operational structures, such as self attention. Further, the current consolidator module is designed mainly in the vision transformer. In Table 1, Table 4 and Table 5, the proposed method obtains promising performance compared to the Full tuning. Push it forward, can it be extended to other model families such as CNNs or MLP architectures [3,4,5]? These will be interesting experiments. Due to the rebuttal time constraints, the authors do not need to add these extension experiments, but feel free to discuss them. 

(4)  There exists a large number of typos in the current version, the authors are able to polish it further. 

[1] ‘AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition’, in NeurIPS2022. 

[2] ‘Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning’, in NeurIPS2022. 

[3] ‘A ConvNet for the 2020s’, in CVPR2022.

[4] ‘AS-MLP: An Axial Shifted MLP Architecture for Vision’, in ICLR2022.

[5] ‘RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality’, in CVPR2022. 

clarity,_quality,_novelty_and_reproducibility: It should not be hard to reproduce this work because the authors describe the specific details for training and hyper parameters. The authors also propose consolidation process, which is also original in efficient transfer learning. 
summary_of_the_review: This work proposes a two-stage consolidation process for efficient transfer learning. It outperforms state-of-the-art methods with fewer stored parameters but superior performance. I tend to vote marginally above the acceptance threshold. Some additional experiments should be conducted to further improve the quality of this manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new fine-tuning method for vision transformers that only trains and stores very few parameters. The fine-tuning method achieves both efficiency and effectiveness through its design of mergeable multiple branches and parameter shifting. Extensive experiments show the method surpasses other adapter methods and full fine-tuning on several transfer learning tasks. 
strength_and_weaknesses: Strength:
1. Consolidator is a very effective method. Compared with other adapter methods and full parameter fine-tuning methods, Consolidator can achieve an almost optimal transfer learning effect.
2. Compared to the adapter methods, the design in this paper can be merged with the original model so that the inference phase does not increase the cost of memory and computation time.

Weakness:
1. The prior of ChannelReorder operation is strong but the explanation of its effectiveness is weak. 
2. The parameters in all tables are stored parameters for Consolidator but not trainable parameters.
3. It should be possible to save time compared to the full parameter finetune during training, I wonder why this is not reported, nor is it compared to other baseline training time.
4. The value of saving storage space is not so great, especially for the base-level model, and there are few experiments on the Large model.
5. There are slightly more spelling errors and grammatical errors. P3 workss; P4 ti, thw; P5 the all the; P7 accomodate.

clarity,_quality,_novelty_and_reproducibility: See Strength And Weaknesses
summary_of_the_review: See Strength And Weaknesses
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed **consolidator** to efficiently fine-tune transformers. Consolidator has multi-branches structures, which can be merged into a single matrix and saved on the disk. Extensive experiments demonstrate the efficiency and effectiveness of consolidator on various vision tasks.
strength_and_weaknesses: #### Strength

1. The design of the consolidator is parameter-efficient since its mergeable property. By extending a single branch to multiple branches, the consolidator has better flexibility and transferability.

2.  The two-stage process maximizes the adaptation capacity without extra inference costs.

3. The paper is well-written and easy to follow.

#### Weaknesses

1. The novelty of this work is limited. The mergeable design is proposed by LoRA [1] for efficient parameter finetuing. Besides, what's the main difference between the **Channel Reorder** and **shuffle** operator in ShuffleNet[2]?

2. Some recent related works are not discussed, e.g. [3][4][5].

3. The improvement is marginal compared with NOAH in Table 1.


[1] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv 2021.

[2] Zhang, Xiangyu, et al. "Shufflenet: An extremely efficient convolutional neural network for mobile devices." CVPR 2018.

[3] Bahng, Hyojin, et al. "Exploring visual prompts for adapting large-scale models." arXiv 2022.

[4] Chen, Shoufa, et al. "AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition." NeurIPS 2022.

[5] Jie, Shibo, and Zhi-Hong Deng. "Convolutional bypasses are better vision transformer adapters." arXiv 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow.

A typo in Sec 3.2: **aim ti**.

There is no source code available so I can not evaluate the reproducibility currently.

summary_of_the_review: See above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission received mixed reviews. The raised issues include restricted applications, limited novelty, generalizations to other encoder backbones, and insufficient comparisons to other adapters while also a lack of large model evaluations. During the rebuttal phase, the authors addressed all the raised issues one-by-one. After carefully checking all the reviews and rebuttals, the AC feels that most of the raised issues, including novelty discussion upon prior arts, more experimental validations, and technical presentations, are significantly addressed. From the methodology itself, this is not a straightforward design of parameterization. The grouped connections, channel reorder, and other paradigms are carefully proposed to gradually improve the reparameterization process. Considering the well-addressed issues and the technical contribution, the AC recommends acceptance. The authors shall recheck all the raised issues and make sure them to be solved in the final version.
justification_for_why_not_higher_score: Combing several designed modules makes a system-like algorithm, which is suitable for a poster.
justification_for_why_not_lower_score: The raised issues are addressed in the technical part and experimental parts. The proposed method is somewhat novel and new.
Decision: Accept: poster