StretchySnake: Flexible VideoMamba for Short and Long-Form Action Recognition

Authors: Nyle Siddiqui, Rohit Gupta, Swetha Sirnam, Mubarak Shah
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Representation Learning, Video Understanding, State Space Models

Abstract:
State space models (SSMs) have very recently been introduced as an alternative deep architecture to transformers, exhibiting competitive or superior performance across various language and vision tasks. However, both SSMs and transformers share certain limitations in the vision domain, namely spatio-temporal inflexibility. Traditionally, deep video models are trained on a fixed resolution and number of frames, often arbitrarily chosen as a trade-off between performance and computational cost. Changing the resolution and/or number of frames a model can ingest usually requires retraining the model, while avoiding re-training by variably changing the weights of a trained model leads to significantly reduced test accuracy. In this paper, we introduce a spatio-temporal flexible training method that encourages a single set of learned weights to adapt well to any input resolution or video length. We achieve this by simply randomly changing the spatial and temporal resolutions of a video during training, and dynamically interpolating the model's weights accordingly. This single change in training not only allows for one model to be applied to both short and long video understanding tasks alike, but also allows for user-specific tailoring of computational cost. We propose and evaluate $5$ different spatio-temporal flexible training methods to find the optimal type for training a video SSM. We then evaluate our best flexibly-trained SSM, which we call StretchySnake, across a variety of short- and long-form action recognition evaluation protocols, such as video retrieval, fine-tuning, and linear probing, and massively outperform the same vanilla video SSM trained in a standard fashion by up to $28$% in some cases. Therefore, our training method can be used as a simple drop-in training technique for any SSM-based video models to strongly improve performance and instill spatio-temporal and compute flexibility.

Decision: reject
Reviewer scores: 1, 6, 6

Reviews and discussion:
<thread>
creation date:
1730564585225

thread_type:
official_review

<thread_object>
creator: reviewer_oh1a

<content>
summary:
This paper aims to train a flexible VideoMamba architecture, which can handle short and long-term video understanding by adapting different video resolution and temporal sampling rate. Accordingly, the paper proposes a simple yet effective stragety based on dynamically interpolating model weights. The authors conduct experiments on four short or long-term video datasets, namely UCF101, HMDB51, COIN and Breakfast, and the results demonstrate that the proposed training method can improve VideoMamba effectively.

strengths:
1. The authors propose a simple yet effective method to adapt different video inputs, which seems to generalize to other domains. 
2. The proposed model can handle both short-term and long-term videos, which advances the applications of VideoMamba.

weaknesses:
1. The authors conduct experiments on UCF101, HMDB51, COIN and Breakfast datasets, which are relatively small datasets. Can you provide results on relative larger datasets, e.g., Kinetics400, something-something-v2. 
2. As stated in Line 299~300, the authors verify the effectiveness of the proposed model mainly based on transfer learning experiments. How about the performance in vanilla fully-supervised setting?
3. Can the proposed training recipe benefits downstream video understanding tasks, e.g., weakly-supervised action localization? 

If the authors can address my concerns above, I am willing to raise my rating.

questions:
1. In Line 347~348, the authors conclude that the best type of st-flexibility modeling is static-tokens, according to the empirical analysis. Can you explain the reason in model design or learning? I think any insights you provide will benefit the society. 
2. As stated in Line 316~317, on Kinetics-400, you flexibly train the proposed model for 12 epochs, while train the vanilla VideoMamba for 50 epochs. Why? Is it not fair? How do the two models perform on Kinetics-400?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655182580

thread_type:
official_review

<thread_object>
creator: reviewer_tetd

<content>
summary:
This paper introduces StretchySnake, a training method for video state space models (SSMs) that enhances spatio-temporal flexibility. The approach involves randomly varying the spatial and temporal resolutions of videos during training, allowing a single model to adapt to different input sizes without retraining. The authors evaluate five different spatio-temporal flexible training methods and find that "static tokens" perform optimally. StretchySnake, their best-performing model, significantly outperforms the standard VideoMamba across various action recognition datasets,

strengths:
In general, the proposed method is somehow simple yet effective. The proposed method explores different settings of making the fixed resolution VideoMamba flexible to dynamic spatial and temporal resolution. Also, the proposed method performs extensive experiments on both short and long action recognition datasets to verify the effectiveness of the proposed method. Various visualization also makes the results more convincing.

weaknesses:
Besides the strengths part, I still have some concerns about the paper:

1. The author starts to fine-tune the model from a self-supervised trained VideoMamba model. Then the author claimed that the static tokens perform best at flexible training, i.e., different time scales and different spatial resolutions. Will the self-supervised training (VideoMAE with token masking) dominate the downstream pattern and performance? I think solely fine-tuning and line-probing a pre-trained fixed-resolution model is less convincing than flexible training a model from scratch and then applying it in the downstream tasks. The author could perform a from-scratch pipeline, even in a smaller dataset to verify the effectiveness of the proposed method to make the result more convincing.

2. The author could have more datasets to convince the model. For example, the author chooses two appearance-based action recognition dataset like UCF101 and HMDB51, and 2 multi-label datasets like BreakFast and COIN. What if the proposed method are tested on some temporal sensitive datasets like Sth-Sth V2? If the author could verify that, I think the proposed paper can be more convincing. 

3. For FLOPs calculation, the author could compare more with test time augmentations (i.e.), with fixed resolution models, but performing more inferences. Applying dynamic resolution to a fixed-resolution model is sometimes unfair here.

questions:
Based on the weaknesses mentioned above, I have the following questions:

1. Is the static token pattern dominated by the upstream self-supervised learning tasks?

2. Will the static-token pattern also performs better on temporal related datasets like Sth-Sth V2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661279362

thread_type:
official_review

<thread_object>
creator: reviewer_gt1y

<content>
summary:
To develop spatio-temporal feature representations that are robust across varying video lengths, the paper exposes pretrained VideoMamba models to random spatio-temporal input resolutions via resizing operations. The proposed method is evaluated on the UCF101, HMDB51, Breakfast, and COIN datasets. Ablation studies are conducted for different spatio-temporal resolutions, and visual analysis for video retrieval is also provided.

strengths:
- The implementation details are sufficiently thorough, enhancing the reproducibility and reliability of this work
- The visual analysis provided is clear and convincing.

weaknesses:
- The intended purpose of Figure 2 is unclear. For instance, the spatial positional embeddings are obscured by other elements, and the temporal positional embedding, defined as $\textbf{ξ}^t$ with shape $1 \times S \times D$ on line 207, is illustrated as having only shape $D$.  On line 194, $S=H \times W \ (p \times p)$, but $S = 128/16=8$, which is very confused. 

- There is a mismatch between Fig.(2) and its caption, where the figure merely depicts a standard tokenization process for videos, while the caption emphasizes 'flexibility,' which is not effectively conveyed in this diagram. 

- The paper uses mathematical symbols inappropriately and inconsistently with established conventions, making it challenging for readers to fully understand the paper.

- There is a lack of comparisons with state-of-the-art (SOTA) methods on action recognition tasks. Additionally, StretchySnake’s performance is surpassed by SOTA methods, such as VideoMAE, which achieves 99.6% top-1 accuracy on UCF101, significantly higher than StretchySnake’s 96.5%.

- The four datasets used in the paper are small-scale. No results on larger scale datasets, such as Kinetics400/600 and Something-Something are provided in this paper.

- Overall, the concept of "st-flexibility" amounts to a straightforward spatio-temporal interpolation of the input, limiting the novelty of the idea and offering minimal contribution to the community.

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732210490878

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Addressing Shared Comments Among Most Reviewers (1/2)

comment:
There were 3 common concerns/comments brought up by 2 or more reviewers. We decided to answer them here for easier discussion and visibility. Individual questions/concerns are still answered in a direct reply to each reviewer, so do please refer to those as well. The 3 common concerns we will answer here are as follows:

1. **Can you show results on larger datasets than the ones in the paper? (R1, R2, & R3)**
	- We acknowledge that UCF-101, HMDB51, COIN, and Breakfast are not the largest action recognition datasets available (the largest in this list being COIN with ~$46K$ total clips). To provide some context, the motivation behind the choice of these datasets were for readily available baselines to compare with, and more importantly, the datasets' specific content. As noted in Lines (300-303), UCF-101 and HMDB51 are short-video action datasets, meaning a model can deduce the action being performed in only a couple frames. Contrastively, COIN and Breakfast are long-video action recognition datasets, meaning the entire action takes place over many minutes, requiring many more frames and long-range video understanding to perform well on. 
	- Regardless, to further show that our results still hold on larger datasets, we perform video retrieval on Kinetics700 with vanilla VideoMamba and StretchySnake (reminder that evaluation details can be found in Section A.2). We are $\textbf{not}$ performing any training, we are simply performing video retrieval with the same models discussed in the main paper. We perform these video retrieval experiments under two different settings - only evaluating on novel classes in K700 that are not present in K400 since K400 is a subset of K700 (resulting in ~$202K$ training videos, ~$15K$ testing videos), and evaluating on all labels in K700 (resulting in ~$520K$ training videos, ~$34K$ testing videos). Even with the massive increase in data as compared to the aforementioned datasets, we show that StretchySnake's training method is scalable and still beats out vanilla VideoMamba by a significant amount, especially on unseen data.
  
## Kinetics400 & Kinetics700 Video Retrieval Results

| Dataset | Model  | New K700 Classes Only | All Classes |
|  :-----:  |  :----: |  :-----:  |  :---------: |
| K400 | VideoMamba | - | $78.0$
| K400| StretchySnake | - | $78.2$ (+$0.2$) 
| K700 | VideoMamba | $51.5$ | $51.3$
| K700| StretchySnake | $59.4$ (+$7.9$) | $57.0$ (+$5.7$)


2. **What are the results like if you train a model flexibly from scratch, rather than loading pre-trained weights first? (R2 & R3)**
	- Firstly, it is important to note that starting with pre-trained weights does not significantly affect our results as the baseline vanilla VideoMamba we compare against is initialized with the same exact weights. Therefore, both StretchySnake and vanilla VideoMamba benefit from the pre-training, yet we still obtain massive performance gains with our proposed training method. We also discuss in our first response to Reviewer TETD that the upstream weights we load before training is standard protocol in previous papers and is somewhat necessary due to VideoMamba's sensitivity during training (as discussed in the original VideoMamba paper).


	- More importantly, we still think investigating how flexible training from scratch can affect performance is a valuable question. Thus, following Reviewer TETD, we train both a vanilla VideoMamba and a flexible VideoMamba "nearly from scratch" on the small UCF-101 dataset. It is "nearly from scratch" as we still load the ImageNet weights of VideoMamba, since training VideoMamba (and video models in general) from complete scratch on videos without first training on images has been shown to be suboptimal and difficult [1]. We provide the results in the table below, where we show that "from scratch" StretchySnake still performs marginally better than "from scratch" vanilla VideoMamba. However, we motivate throughout our paper that flexible training helps mostly with unseen data and generalization to other domains, hence our focus on video retrieval results on both short- and long-term action recognition datasets. Thus, we also perform video retrieval on the same datasets in the paper, and again we see that even training flexibly from scratch still leads to good performance gains in other downstream domains. We hope this further supports the validity and benefit of our contribution.

  

## Video Retrieval Results after Training from Scratch on UCF101

| Model | UCF101 | HMDB51| Breakfast | COIN |
| --- | :------: | :-----------: | :---------------: | :---------------: |
| VideoMamba | $92.0$ | $47.5$ | $22.9$ | $50.1$ |
| StretchySnake | $92.7$ (+$0.7$) | $49.4$ (+$1.9$) | $25.1$ (+$2.2$) | $52.0$ (+$1.9$) |


1. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., & Schmid, C. (2021). Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 6836-6846).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes StretchySnake, a spatio-temporal flexible training method for action recognition which based on the latest VideoMamba architecture. Experiments are done on UCF-101, HMDB-51, Breakfast, and COIN with good results compared with vanilla VideoMamba.

On the positive side:
* **Positive 1**: The proposed method seems simple and yet effective once pre-training on Kinetics and evaluated on small datasets such as UCF-101, HMDB-51, Breakfast, and COIN (with good improvements) thanks to the flexible training input (can be also considered as augmentations). 

On the negative side: 
* **Negative 1** There is no experiments done on large-scale datasets (see details below).
* **Negative 2** There is no direct comparison with latest methods based on CNNs or ViTs on the large-scale datasets.

## Further explanation on Negative points.
* **Negative 1** Although in the authors response, they provided additional experiments on Kinetics-700. However, the experimental setup is video retrieval (where the paper claims action recognition as in its title) and it requires no training while the proposed method is claimed to be flexible **training** approach.
* **Negative 2** There is one place in discussion between the authors and reviewer oh1A, the authors provided results on K-400 with both video-retrieval and classification results. Note that this experiments show very small differences (0.2 and 0.4) between vanilla VideoMamba and the proposed method, this may indicate that the flexible training may be beneficial only to small size datasets and when we have enough data, this *augmentation* (or flexible input sampling) is not very useful. Furthermore, the proposed method only compare with vanilla Video Mamba, not the latest methods of CNN-based or ViT-based.

## Final Recommendation
Although the proposed method has some potential, the current paper does not have experiments on a large-scale dataset (K-400, K-600, or K-700) with **a fair setting** (e.g., classification not a retrieval and compare with all approaches not just VideoMamba). Additionally, it would be more flexible if the proposed approach can be trained from scratch (or at least pre-trained with ImageNet only) to have fair comparison and flexibility for model / training adoption.

With the above observation, AC recommends not to accept the paper in its current form and encourages the author(s) to improve the paper and submit it to future conferences.