Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new Spatio-Temporal Adapter (ST-Adapter) for parameter-efficient fine-tuning on video tasks. With a much smaller trainable parameter, ST-Adapter can match or even outperform the strong full fine-tuning strategy on K400 and SSv2 datasets.

strengths_and_weaknesses: Pros:
1. It's a good attempt to introduce an Adapter for Spatio-Temporal modeling using a 2D pretrained model.
2. The proposed ST-Adapter are effective. It can match or even outperform the strong full fine-tuning strategy with a much smaller trainable parameter.
3. SOTA results on K400 and SSv2 using a CLIP pre-trained model.

Cons:
1. Lack two important comparisons. The new added ST-Adapter has a  quite different structure from common practice for extending a 2D backbone to Spatio-temporal, e.g, an additional temporal self-attention in Figure1. So it would be better to disentangle the effect of the new structure (e.g., DW 3D-CONV which may have a complementary impact on a full Transformer architecture) and learning strategy (only tuning some new added modules, which is also called Adapter). So, exp1 [finetuning the (b) of Figure 1] and exp2 [using (a) as an Adapter of Figure 1] would be valuable.
2. CLIP pre-trained model and ImageNet-21K pretrained model performs very differently. For example, Partial Fine-tuning using CLIP pre-trained model on K400 only performs a little worse than Full Fine-tuning (80.1 vs 81.0) while this gap is quite large when using ImageNet-21k pre-trained model (61.7 vs 76.9). It seems that the CLIP learns many video concepts which may narrow the gap between pre-training and video finetuning, especially for K400 which is largely biased to spatial appearance. So it may be more suitable to conduct an ablation study using ImageNet-21K pretrained model or using SSv2 dataset.
3. Similar to 2, it would be helpful to add the ImageNet-22K pre-trained results in Table 2 and Table 3

questions: See Strengths & Weakness.
limitations: No.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The goal of this paper is to perform parameter-efficient learning on image-to-video adaptation tasks. Specifically, the authors propose a Spatio-Temporal Adapter, which applies an additional depth-wise 3D convolution in adapters, to incorporate temporal and spatial information with only a limited amount of parameters in the model fine-tuned. In the experiment, the authors showed that, compared with full fine-tuning and other fine-tuning techniques, the proposed ST-adapter can achieve better results in Kinetics-400 and Something-Something-v2 datasets.
strengths_and_weaknesses: Strengths:
- The motivation is reasonable. Since the video pre-training data or pretrained models are limited or hard to obtain, it is ideal to use pretrained models/data from other modalities and adapt to video domains.

- The paper is easy to understand, and the presentation is clear and good.

Weaknesses:
- Since the major difference between the prior work [1] and the proposed method is depth-wise 3D convolutional layers in the hidden state, it is important to compare the prior work and show that such a design makes a difference in the performance. 

- A major weakness of this paper is the experimental setup. As mentioned in the prior empirical study [2] on parameter-efficient methods, a common pitfall or mistake of the existing parameter-efficient works is only presenting validation set results while doing early-stoping on the validation set. This makes the reported results of the parameter-efficient methods biased on the validation set and higher than the full fine-tuning. A more reasonable experimental setup is to either report test set results or split the training set and perform early-stoping on the held-out set. It is unclear whether this paper follows the proper experimental setup. 

- In Table 1, only showing the number of fine-tuned parameters is a bit misleading, as some methods including ST-adapter introduces **extra** parameters which makes the comparison unfair due to the different amounts of total parameters in the models across different methods. 

- Comparison to the state-of-the-art methods is confusing. Since the major focus is the effectiveness of the ST-adapter, a fair comparison should be, under the same experiment setups (e.g., same pretrained datasets), how the ST-adapter improves against the prior works. For example, the authors should present the results of (1) ViT-L pretrained on IM-21k and apply ST-Adapter and (2) ViT-L pretrained on Clip and without ST-adapter. In this way, the reader can understand the improvement gains solely provided by the proposed ST-adapter.

[1] "Parameter-efficient transfer learning for nlp", Houlsby et al., ICML '19

[2] "Revisiting Parameter-Efficient Tuning: Are We Really There Yet?", Chen et al.

questions: The following questions are mainly based on the concerns listed in the Weaknesses part. Here are the questions:

- I have some concerns regarding the experimental setup (i.e., doing early-stoping on the validation set and only presenting validation set results for parameter-efficient methods). Could the authors provide the test set results (or doing early-stopping on the held-out set) or provide arguments to alleviate this concern?

- Could the authors present a column to clearly show the number of total parameters across different methods? How does the extra parameter affect the inference time compared to the method without the extra parameters?

- To have a fairer comparison to SoTA, could the authors provide the results under the same setups, such as results of (1) ViT-L pretrained on IM-21k and applying ST-Adapter and (2) ViT-L pretrained on Clip and without ST-adapter?


- Notation is confusing. In line 120, the authors mention $N=H \times W / P^2$ which is the number of patches in a single frame without temporal information. However, in line 173, the authors mention the feature representations will be first ***reshaped*** from $X' \in R^{N \times d}$ to $R^{T \times H \times W \times d}$. Why does $N$ contain the temporal information? There seems to be notation confusion or inaccurate description. 


limitations: I do not see the authors including any discussion on limitations or negative societal impact. The potential limitations of the paper are the inference time of using extra parameters and generalization to other video tasks.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Authors propose an extension to the Adapter [24] framework to build a space-time adapter, which converts a pretrained image model into a video recognition model by finetuning very few parameters on the video task. The proposed adapter is very simple, based upon depthwise 3D convolution, added to each Transformer block. Experiments are shown on Kinetics and SSv2, where the proposed method obtains strong results, often comparable to fully finetuning the network (less so on SSv2, given its temporally challenging recognition task). Moreover, the resulting model is more data and training efficient.
strengths_and_weaknesses: ## Strengths
1. [originality] While the method is a simple extension of "Adapter" [26]: The proposed method is very simple to implement and builds upon a well known Adapter architecture. Nevertheless, it obtains strong results and to my knowledge these results are not well known in the community.
2. [quality] **Comprehensive ablations:** Authors provide comprehensive set of interesting ablations. This includes apples to apples comparisons with reasonable baselines (Table 1), as well as properties of the adapter in Fig 2 and Tab 4, which show STAdapter makes the model training and data efficient.
3. [clarity] **The paper is generally very well written**, easy to follow, with useful and easy to read figures. 

## Weaknesses
1. [quality] **Runtime savings:** While authors show impressive results in reducing parameters being finetuned yet obtaining near SOTA performance, it is unclear what the real-world advantages of such a method is. Does the reduced parameters being trained lead to speed up in training time? If so, by how much? Does that make the training possible on GPUs with less memory? If the proposed approach makes SOTA video models accessible to researchers with constrained resources, the impact of the paper will be much more.
2. [quality] **Other backbones:** The results are largely limited to ViT based models. It's not clear if the approach can also work with other popular architectures, such as MViT and Swin? Swin for instance does come with large scale pretrained models, which would be interesting to adapt to videos.
3. [significance] **Results without CLIP are not that strong**: Authors show impressive results, including 85.6 on Kinetics, while finetuning very few parameters and operating on very few frames from CLIP pretrained model. However, the results with ImageNet-21K are not as strong (although they do obtain improvements over other approaches for parameter efficient finetuning). It's not clear why this is the case. Perhaps more results with other "foundation models" (such as https://github.com/facebookresearch/SWAG) could help decipher if this approach is best applicable to vision-text models or does it work more generally. 
4. [clarity] **SOTA results missing** some higher performing methods like "Multiview Transformers for Video Recognition" (CVPR'22), "Omnivore: A Single Model for Many Visual Modalities" (CVPR'22) etc. While not directly comparable since they use different architectures and pre-training, they obtain better performance in Table 2 and 3 and should be reported when comparing to SOTA.
5. [significance] **More datasets:** Since the proposed method is fairly simple and general, it would be nice to evaluate it more broadly for other video understanding tasks, such as egocentric action classification (eg EPIC, Ego4D), long video classification (eg HowTo100M). Such experiments would further throw light on the advantages and limitations of the proposed adapter.

## Minor
- L251: "arts" -> "art"
- L256 "stat" -> "state"
questions: Please reply to the weaknesses above, especially 1 and 2.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a parameter-efficient image-to-video transfer learning framework. In particular, the authors introduce a spatiotemporal adapter (ST-Adapter) for fine-tuning a pretrained image model to video tasks. In this setting, the parameters of the original network are frozen, and only the parameters of the  ST-Adapter are updated. The authors demonstrate that will only ~8% trainable parameters, they are able to achieve competitive results to previous fully fine-tuned approaches on various video recognition benchmarks. 
strengths_and_weaknesses: Strengths:
+ Timely and relevant topics as video models are getting larger and larger.
+ Strong results with a small number of trainable parameters.
+ Detailed ablation studies for the most part. 
+ The paper is well written and easy to follow.

Weaknesses:
- The technical contribution of the paper can be seen as somewhat limited. Adapters and other parameter-efficient modules (i.e., Lora, prompt-tuning, etc.) have been widely used in other domains such as NLP, images, etc. The authors use a relatively straightforward adaptation of these ideas (with the addition of 3D depthwise convolutions, which are commonly used for efficient video processing) to the video setting. While the technical contribution is small, given that this is the first attempt (to the best of my knowledge) to perform parameter-efficient tuning on video recognition benchmarks, I'm ok with it. 
- I wish the authors validated their proposed ST-Adapter scheme with more backbones (e.g., Swin, Uniformer, Motionformer, etc.) to demonstrate its generality.
- I think it would have also been useful to see more experiments / insights into the design choices for ST-Adapter. While 3D depthwise convolution is a natural choice, the question remains is it the best choice? Have the authors experimented with any other operators (e.g., self-attention, efficient self-attention, etc.)? In my view, it would be useful to see more design-related experiments to verify that this is indeed the adapter scheme that works most effectively.
- Many of the Tables listing GFLOPs are unfair to the proposed method. Is it possible to highlight not only the GFLOPs but also the number of trained parameters? The proposed approach is trained using significantly fewer parameters than the remaining baselines that are fully finetuned.
questions: See the weaknesses section above. 
limitations: No, the limitations were not discussed.
review_title: null

Meta-review: metareview: This paper proposes a new Spatio-temporal adapter for parameter-efficient fine-tuning per video task transferred from a pretrained image model. After the discussion phase, the requested comparisons on full finetuning, finetuning only the temporal attention modules and more backbones are added and the reviewers are satisfied with the rebuttal. Given all the positive scores by reviewers, the meta-reviewers recommend accepting this paper.
confidence: Certain
Decision: Accept