Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework to adapt large video-based models to down-stream tasks with a parameter-accuracy trade-off. It analyzes different PETL techniques and investigates the importance of fine-tuning position of their methods. In order to better transfer prefix-tuning from NLP to vision task, it compares differences between NLP and video data regarding data structures and pre-training mechanisms.  
strength_and_weaknesses: Strengths:
● The paper is well written and well structured.
● The authors considers the situation of insufficient computing resources. Under the corresponding experimental settings, the experimental results demonstrate the superiority of the method proposed in paper.  
Weakness:
● As one of the baselines, Adaptformer can achieve excellent performance (59.02 on SSv2 and 55.69 on HMDB51) after larger batch size training.  Why does reducing the batch size has such a big impact on performance of Adaptformer? It is uncertain whether authors have explored the hyperparameters of Adaptformer under small batch size setting. In other words, authors can increase the batch size to compare the proposed method and Adaptformer.
● The paper only shows the experimental results but does not elaborate on the reasons for the gains brought by the method.
Minors:
● There are some minor mistakes in the experimental table: first two lines in Table 1 appear to have numerical errors where "0.11M" is inconsistent with"0.18M" in Table 3, and "Tune FC layer" and "Full-tuning" should be swapped.
clarity,_quality,_novelty_and_reproducibility:  
summary_of_the_review: The proposed method seems to achieve the best performance under the condition of limited computing resources. This paper should demonstrate the superiority of the method under fairer conditions or give sufficient reason analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a module inserted into a pre-trained model for parameter efficient fine-tuning. The authors empirically combine adapter, adapt pre-fix tuning, and verify the effectiveness of the design through a series of experiments. The proposed method achieves very competitive results and has the potential to outperform full fine-tuning.
strength_and_weaknesses: ## Strength

1. This paper proposed a design for parameter efficient fine-tuning and demonstrate its effectiveness in a video setting.

1. The authors conducted some analyses to (partially) validate the design choices of the proposed method.

## Weakness

1. This is mainly an empirical paper. Therefore, the reviewer would expect more extensive and rigorous experiments to demonstrate the generalizability and effectiveness of the proposed method. For example, different architectures, and other combinations of pre-training and fine-tuning datasets.

1. Some conditions are missing in Table 1:
    - Section 2.2 and Figure 2 mentioned three types of most common baselines, but the prompt tuning baseline is missing
    - In CV or NLP, typically more trainable parameters lead to better performance. Therefore, the authors should compare their method with baselines under similar trainable parameters.
    - Since the method is the combination of adapter, PATT (a variant of prefix-tuning), and fc layer fine-tuning, the author should show the results of different combinations of baselines.

1. Some critical ablations of the proposed method are missing:
    - Since the final method is the combination of adapter, PATT, and fc layer fine-tuning, the authors should ablate this.
    - The proposed PATT is a variant of prefix-tuning. Thus the author should also ablate the design of PATT. Specifically, (1) with or without the weight $s$, (2) weighted addition v.s. concatenation of $K_p$ and $V_p$, and (3) $Z^{l-1}$ vs some trainable parameters as the input to Equation 9 for generating $K_p$ and $V_p$.

1. It would also be good to analyze why the proposed design is better (Eg. mathematically, from the perspective of gradient propagation, or anything else). Otherwise, it looks like the authors simply enumerate some possible/common design choices and do a manual architecture search.
clarity,_quality,_novelty_and_reproducibility: This paper clearly explains the proposed method as well as previous relevant baselines such as prefix-tuning and adapter. Since this is more of an empirical paper, the novelty and contribution would mainly come from the quality of the experiments. However, some crucial experiments are missing as described in the Weakness section of the review. The proposed method is not complicated so should not be hard to reproduce. Nevertheless, the authors do not submit code or promise to release code upon acceptance.
summary_of_the_review: At first glance, the proposed method seems to be a simple yet effective design that combines and adapts previous techniques of parameter efficient fine-tuning. However, after close examination, some critical experiments and ablations are not provided to support this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Motivated by recent studies on parameter-efficient learning of Transformers in NLP (He et al., 2022) and vision domain (Jia et all., 2022), the authors proposed to extend them to video models. Numerous ablation studies are conducted to find the best configuration for incorporating parameter efficient approaches in video Transformers, which turned out to be consistent with the previous observation (He et al., 2022) where the self-attention and MLP layers in Transformers are modified with a prefix-tuning-like approach and adaptor, respectively. The proposed method is evaluated in two video downstream tasks and demonstrated encouraging results.
strength_and_weaknesses: Strength:
- The motivation for applying parameter-efficient learning to videos is sound and reasonable.
- Ablation studies and empirical results are strong and convincing.

Weakness:
1. Novelty
The ideas presented in the paper–steering the Transformer via a prefix-tuning-like modification to MHSA and adaptor-like modification to the MLP–are already explored by He et al., 2022, which leads to a very similar model. Although there are some differences in details, such as how we incorporate prefix-tuning in self-attention, these are rather minor and can be considered as the special case of the prior work (He et al., 2022). Although I agree that extending the observations to videos can be valuable, the impact is still not significant considering that success in the NLP domain tends to easily extend to the visual domains.


2. Experiments
- Although the same method is applicable to images, the paper is evaluated only on videos. To understand the benefits and limitations of the proposed method better, it would be great to see the results in image downstream tasks as in the prior works (e.g., Chen et al., 2022).

- The comparisons to AdaptFormer (Chen et al., 2022) in Table 2 need more careful considerations. It is not easy to directly compare the two methods (AdaptFormer and the proposed method) since the number of parameters are different (AdaptFormer uses fewer parameters). Considering the results from Chen et al., 2022 that the performance of AdaptFormer also increases proportionally to the number of parameters, it would be more desirable to roughly match the parameters and compare the performance in multiple scales. 


3. Presentations
- There are some minor presentation issues. 
  1) the first dimension of B (in the line above Eq.(2)) should be t->p. 
  2) in Table 1, the first two rows should be switched. 

- It would also be interesting to consider comparisons with much more parameter-efficient alternatives such as BitFit (Ben-Zaken et al., 2022).

clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written. The novelty of the work on the other hand is limited due to the overlap to the prior work (He et al., 2022).
summary_of_the_review: Overall, I lean towards rejecting this paper since the ideas and results from the paper largely overlap with the prior work (He et al., 2022) and the experiments only demonstrate the results in videos.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The goal of this paper is to reduce the parameter usage when adapting pretrained models to video tasks. To this end, the authors propose a new parameter-efficient method, which adds the learnable parameters to generate additional keys and values. Unlike the prior Prefix-tuning, the authors further reduce trainable parameters by introducing an autoencoder-like modules for generating keys and values. In the experiment, the authors show the improvements when adapting a Kinetics 400 pretrained models to SSv2 and HMDB51 datasets. 
strength_and_weaknesses: **Strengthes**
- The paper presentation is good, and the authors clearly show how the proposed method is designed with easy-to-understand figures (Figure 2 and 3).
- The authors presented several analyses on the hyper-parameters (Table 2, 3, 4, and 5).

**Weaknesses**
- Paper title does not properly describe the paper content: Since the a large portion of the paper deals with the video data and video recognition tasks, I would suggest the author either include more different types of vision tasks or narrow the title to "for video recognition tasks".


- In section 2.3, "prefix implementation in NLP can be regarded as providing conditional information for the downstream tasks, which is similar with the pre-training process where words are masked". The sentence is confusing to me, since the connection between the prefix tuning and masking text pretraining does not look similar to me. I would suggest to elaborate this motivation and provide a more reasonable motivation why the prefix tuning is suitable for video tasks. 

- CLIP is not one of parameter-efficient methods, so it should be removed from Section 2.2

- Less convincing experiments: 

1. Missing an important baseline - ST-Adapter [1]: Since the paper addresses the video tasks, one important baseline (ST-Adapter [1] ) should be included in the experiment. ST-Adapter present several video benchmarks, including SSv2 and K-400 datasets with different pretrained weights.

2. *Comparison to AdapterFormer*: Although the authors compare AdapterFormer, there are still some concerns in the comparison. First, the experimental setup of AdapterFormer paper is different from the proposed method, since AdapterFormer uses different settings (pretrained on IN-21k and adapt to SSv2). I would suggest the authors also include their experimental setups (same backbone, same pretraining datasets, and same downstream tasks) to provide a complete comparison. It is unclear whether the baseline is properly tuned and well reproduced. Also, the authors show the number of parameters of AdapterFormer is only 1.73M, while the AdapterFormer, which is based on adapter, can certainly be improved by increasing the hidden state size of adapters. 

3. Experiment setups are still limited. The authors only show a single model architecture, video SwinTransformer, and I would suggest the authors extend to other model architecture. I would be interested in how the proposed method performs under ViT with ImageNet-21k perform on K400.

Ref:
[1] "ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning", Pan et al., NeurIPS 2022
clarity,_quality,_novelty_and_reproducibility: - Clarity, Quality:
I think the paper presentation is clear, and the proposed method is easy to understand. 

- Novelty:
Problem novelty: ST-Adapter [1] and AdapterFormer [2] have addressed the same problem settings.
Method novelty: Since the proposed method is based on Prefix tuning [3], the way they introduce the learnable parameters have been explored by the prior work. The only interesting point is they further reduce the parameter usage by using a autoencoder-like module rather than additional linear layers to produce keys/values, although this method is similar to LoRA [4].

Ref:
[1] "ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning", Pan et al., NeurIPS 2022
[2] "AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition", Chen et al., NeurIPS 2022.
[3] "Prefix-Tuning: Optimizing Continuous Prompts for Generation", Li et al.
[4] "LoRA: Low-Rank Adaptation of Large Language Models", Hu et al.
summary_of_the_review: While the authors propose a new parameter-efficient method for video classification tasks, there exist some concerns and questions in the experiment sections. More experimental setups and baselines should be considered for a more complete and convincing empirical results. Given the current status of the paper, I would suggest the score "marginally below the acceptance threshold".
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
This paper looks at the problem of parameter-efficient transfer learning for computer vision. Inspired by prior methods, the authors combine several known principles including adapters and pre-fix tuning for self-attention and MLP layers. Results are demonstrated across video vision tasks, including with self-supervised pretrained models, demonstrating improved parameter-efficiency and in some cases improvements over full finetuning. 

Overall, the reviewers appreciated the writing of the paper, the importance of the topic, and the results demonstrated (though caveats forthcoming). However, a range of weaknesses were identified, including: 1) The comparisons to baselines such as Adaptformer and VPT, including hyperparameter settings and fair comparisons (uEYB), 2) The fact that the method itself is not intrinsic to videos, yet did not present results on image datasets (uEYB,EH6p), 3) Explanations or analysis for why the method can work well (GJuJ, oYYz), and 4) Lack of extensive and rigorous experimentation to demonstrate the effectiveness and generality of the approach, including missing conditions and experiments/ablations (oYYz,EH6p), 4) Novelty with respect to prior findings in these areas (uEYB,EH6p). The authors provided extensive rebuttals, including new experiments on image-based datasets. 

However, after extensive discussions there are still considerable concerns that have not been addressed. Ultimately, this is an empirical paper with a simple method that combines ideas from prior works and demonstrates results empirically. Ignoring incremental novelty and the concurrent work, such empirical methods still need strong motivation/descriptions of where and why the method works, rigorous empirical methodology and experimental results (including rigorous implementations of baselines with equal hyper-parameter tuning budgets given to all methods), and demonstration that the method is general across many application areas (or at least description of why it's not). In its current state, the paper does not live up to this however. Specifically, some examples of remaining concerns include: 1) Unclear motivation: It's not clear how the method benefits video tasks in particular (which was the focus of the paper), 2) The image-based experiments (and indeed the video-based ones as well, based on the initial reviews) highlight a number of concerns including potentially weak implementations of baselines (e.g. using VPT-shallow and/or not matching the number of parameters across these baselines and the proposed method), and 3) Still lacks extensive experiments across many architectures, downstream tasks, etc. Either the method should be well-motivated to the limited tasks (videos) or more extensive experiments should be done to show the generality of the method. 

Given these significant concerns highlighted during the reviews and ensuing discussion, and the fact that these points hit directly the experiments that are crucial for an empirical study, I cannot recommend acceptance in its current state. 
justification_for_why_not_higher_score: There are a number of concerns about exactly the empirical contributions of the paper, either targeted towards videos or shown more generally for vision tasks, and hence further work is needed to tighten the approach and experimental methodology.
Decision: Reject