MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Authors: Yuhuan Yang
Chaofan Ma
Zhenjie Mao
Jiangchao Yao
Ya Zhang
Yanfeng Wang
Venue: ICML 2025 poster (2025)
Keywords: Video Recognition
PEFT
Mamba
Adapter
Primary area: applications->computer_vision
TL;DR: We propose MoMa, an efficient adapter that integrates spatial-temporal modeling into pre-trained image foundation models for improved video understanding, achieving better performance with lower computational cost.

Abstract: Video understanding is a complex challenge that requires effective modeling of spatial-temporal dynamics. 
With the success of image foundation models (IFMs) in image understanding, recent approaches have explored parameter-efficient fine-tuning (PEFT)  to adapt IFMs for video. 
However, most of these methods tend to process
spatial and temporal information separately,
which may fail to capture the full intricacy of video dynamics. 
In this paper, we propose MoMa, an efficient adapter framework that achieves full spatial-temporal modeling by integrating Mamba's selective state space modeling into IFMs. 
We propose a novel SeqMod operation to inject spatial-temporal information into pre-trained IFMs, without disrupting their original features. 
By incorporating SeqMod into a Divide-and-Modulate architecture, MoMa enhances video understanding while maintaining computational efficiency. 
Extensive experiments on multiple video benchmarks demonstrate the effectiveness of MoMa, achieving superior performance with reduced computational cost. 
Codes will be released upon publication.

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Yuhuan Yang' Chaofan Ma Zhenjie Mao
Jiangchao Yao 12 Ya Zhang 32 Yanfeng Wang
Abstract
video understanding model from scratch is highly inefficient and data demanding. In contrast, image understanding has made significant progress by using image foundation models (IFMs) (Radford et al., 2021; Bao et al., 2021; Wang et al., 2022). This progress has spurred efforts to leverage IFMs for video understanding, as IFMs provide strong pre-trained representations, reducing the reliance on training models from scratch for specific tasks.
Video understanding a complex challenge that requires effective modeling of spatial-temporal dynamies. With the success of image founda- tion models (IFMs) in image understanding, re- cent approaches have explored surameter-effiieen fine-tuning (PEFT) to adapt IFMs for video. How- ever, most of these methods tend to process spatial and temporal information separately, which may fail to capture the full intricacy of video dynam- ies. In this paper, propose MoMa, an effi- cient adapter framework that achieves full spatial- temporal modeling by integrating Mamba's se- lective state space modeling into IFMs. We pro- pose a novel SeqMod operation to inject spatial- temporal information into pre-trained IFMs, with- out disrupting their original features. By incor- porating SeqMod into a Divide-and-Modullte chitecture, MoMa enhances video understand- ing while maintaining computational efficiency. Extensive experiments on multiple video bench- marks demonstrate the effectiveness of MoMa, achieving superior performance with reduced computational cost.
Some early works have attempted to adapt IFMs to video understanding by conducting full parameter training on video data (Xue et al., 2022; Li et al., 2023). These meth- ods, although effective, still require considerable computa- tional resources and data. To alleviate this, a more efficient approach is adapt IFMs using parameter-efficient fine- tuning (PEFT), and keeping most of the parameters frozen. Since IFMs do not explicitly model temporal dynamics dur- ing pre-training, these PEFT-based methods are compelled to incorporate additional modules to capture temporal de- pendencies across frames, often independently of the origi- nal spatial representations. For instance, AIM (Yang et al., 2023a) introduces temporal attention layer within each Transformer block. DualPath (Park et al., 2023) designs two separate adapter modules for spatial and temporal modeling, and DiST (Qing et al., 2023) introduces a parallel temporal encoding branch alongside the spatial one. By processing spatial and temporal information separately, these PEFT designs simplify the full modeling of sequences. As re- sult, patches at different spatial and temporal positions can only interact with each other in an indirect, implicit manner. This is potentially insufficient for capturing the complex dynamics inherent in video data.
1. Introduction
Video understanding is a crucial yet challenging task in computer vision. The key to addressing this challenge lies in learning effective ppatial-tempordd representations from
video data (Tong et al., 2022; Feichtenhofer et al., 2022; Carreira & Zisserman, 2017; Hara et al., 2017; Feichten- hofer et al., 2019; Liu et al., 2022). Since video data ex- hibits complex ppatial-temporal dynamics, training such a
To explicitly capture the full spatial-temporal dynamics while using PEFT, one straightforward method is to apply an additional full attention over the entire ppatial-temporaa sequence after IFMs. However, due the long length of the spatial-temporal sequence, the quadratic complexity of full attention causes scalability issues, resulting in drastic increase in memory usage and inefficiencies in speed when processing multiple frames. In this context, recent low-cost operators like Mamba (Gu & Dao, 2023) offer a promising solution with its selective state space model (SSM), achiev- ing linear complexity for long-term dynamic modeling. This
Cooperrttive Medianet Innovation Center, Shanghai Jiao Tong University 'Shanghai Artíficial Intelligence Laboratory 'School of Artificial Intelligence, Shanghai Jiao Tong University. Corre- spondence to: Jiangchao Yao mamaarkr@@sjt..ddu.oo Yanfeng Wang <wangyanteng6222es @jt.aa sjt..cdu.cn>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
motivates explore how Mamba can be leveraged in PEFT to capture the full spatial-temporal dynamics within IFMs, while maintaining training efficiency.
We conduct extensive experiments and ablations on mul- tiple video understanding benchmarks, showing MoMa's significantly improvements in both performance and com- putational efficiency compared to existing methods.
A naive way to integrate Mamba is to first extract each frame's spatial features using IFMs, then flatten and concate- nate into a spatial-temporal sequence, which is processed by Mamba. However, we found that this approach often suboptimal. Most hybrid architectures require fewer layers of attention compared with Mamba to achieve best perfor- mance (Waleffe et al., 2024; Lieber et al., 2024), which is opposite in our case. Inserting a lightweight Mamba module directly into a pre-trained heavy Transformer-base IFMs may confuse the model, especially when the Trans- former parameters are frozen. To better integrate Mamba into pre-trained Transformer, we propose a sophisticated sequence modulation operation SeqMod for efficiently in- jecting spatial- pptiial-emmorrll information using the lightweight Mamba, i.e., rather than directly modifying the features of IFMs, features are adjusted by learnable scale and bias. This modulation operation prevents interference with IFMs, and through a conditional adjustment mechanism using Mamba, peappal-temporal dynamies can still be explicitly learned, thus preserving training efficiency and flexibility.
2. Related Works
Video Understanding. One crucial aspect of video un- derstanding is capturing the temporal patterns in videos. CNN-based methods introduced 3D convolutions and other temporal modules to handle this (Feichtenhofer et al., 2019; Feichtenhofer, 2020). Compared with the limited receptive field of 3D convolutions, Transformer- based methods with global attention mechanisms (Arnab et al., 2021; Zhang et al., 2021; Li et al., 2022) have achieved promising perfor- mance. These models can capture long-range dependencies across frames, enabling them to better understand the com- plex temporal relationships inherent in video data. While we aim to capture full spatial-temporal dynamics using Mamba architecture for more efficient video understanding.
Image Foundation Models Adaptation. Recently, image foundation models (IFMs) has made remarkable progress. The introduction of self-supervised learning (Oquab et al., 2023; Chen et al., 2020) and multi-modal contrastive Iearn- ing (Jia et al., 2021; Radford et al., 2021; Zhai et al., 2023) techniques enable models to learn effective representations from unlabeled images. And, the come up of large-scale datasets such as LAION-5B (Schuhmann et al., 2022) and COYO-700M (Byeon et al., 2022) has led to even more powerful visual representations. IFMs trained on such web- scale datasets have been shown to be well generalized and effective in wide range of computer vision tasks. Adapting these IFMs to these tasks has carned significant attention. Several approaches have been proposed to adapt IFMs to downstream tasks like detection (Kuo et al., 2022; Gu et al., 202 Minderer et al., 2023; Zhong et al., 2022), segmen- tation (Ma et al., 2022: Yang et al., 2024b; Ma al., 2023b; Liu et al., 2023; Ma et al., 2025), image captioning (Hes- sel et al., 2021; Nukrai et al., 2022; Mokady et al., 2021),
To fully leverage this, we further propose MoMa, frame- work that integrates this modulation operation with Mamba within IFMs for PEFT. Specifically, it incorporates a two- process applied to each layer
stage sovide-and-Mddull of given Transformer- based IFM. In the first Divide stage, we aim enhance the efficiency of the attention mechanism. Rather than processing each video frame individually, we divide each frame into smaller windows and apply local attention within each window. This window-based approach significantly reduces computational overhead by capturing short-term spatial dependencies. Then, in the second Modu- late stage, we focus on capturing the global spatial-temporal dynamics. We employ the strong SeqMod operation to ensure that comprehensive spatial-temporal information is effectively injected through sequences of short-term spatial features. This framework not only improves computational efficiency, but also enables PEFT to capture intricate spatial- temporal relationships in video data using IFMs.
2D/3D generation and reconstruction (Shi et al., 2025; Li et al., 2024c; Xiong et al., 2025; Li et al., 2024a; Ma et al., 2023a), and some trustworthy scenarios (Han et al., 2025; Yang et al., 2023b; 2021). In this work, we follow the re- search direction of EVL (Lin et al., 2022b), ST-Adapter (Pan et al., 2022), AIM (Yang al., 2023a) and DiST (Qing et al., 2023), and aim to adapt IFMs to video understanding tasks.
To sum up, our contributions lie three fold:
We pioneer to use Mamba as an efficient adapter for image foundation models (IFMs). With our sequence modulation operation SeqMod, full -tetoolll dynamics can be captured without interfering the pre-trained IFMs.
State Space Model. Compared to Transformers that based on quadratic-complexity attention, State Space Mod- cls (SSMs) (Gu et al., 2021a) excels at processing long se- quences with linear complexity. And recently, Mamba (Gu & Dao, 2023) distinguishes itself by incorporating a data- dependent selection mechanism and hardware-efficient algo- rithms, further improve its efficiency for sequential model-
We propose a novel framework MOMa, which consists of a Divide and Modulate stage. For each layer of IFM, It first applies window-based spatial local attention, followed by modulation via SeqMod to inject full spatial-temporal information.
2

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
A Attention
Sequence Moduaation (SeiMod)
State Space Model
Y + + y2
Sequenoe Split
Feed Forward
Divide Stage
Modulate Stage
Divide Stage
Modulate Stage
Task Head
*
Y/2
A Attention
Divide Stage
Modulate Stage
Figure 1. Overview of our proposed MoMa. The Divide stage aim to cut down the computational cost by narrowing attention range. We utilize the original CLIP attention layers. but splits the input video sequence into smaller windows and processes attention independently for each window. In Modulate stage, we aim to capture full spatial-temporal dynamies using lightweight Mamba. We first forward the sequence through SSM layer obtain sequences of scale and bias, then use our proposed sequence modulation operation (SeqMod) to inject spatial-tempot information into IFMs without interfering with their pre-trained parameters. Finally, the output fed into CLIP's feed-forward layer. Only SSM layers are trainable through the whole architecture.
ing tasks. vision domain, backbones built on Mamba (Liu et al., 2024; Zhu et al., 2024; Hatamizadeh Kautz, 2024) shows great potential, and introduces new solution in pro- cessing visual data at large scale and high dimension such video (Li et al., 2024b; Chen et al., 2024) and 3D point cloud (Liang et al., 2024; Zhang et al., 2024). The great potential of Mamba motivates a series of works (Teng et al., 2024; Wan et al., 2024; Yang et al., 2024c;a; Pan et al., 2024) on downstream tasks, further demonstrates Mamba's 's better performances and higher GPU efficiency.
CLIP and cannot change drastically. Thus, our focus is on how to maximize advantages of Mamba without disrupt- ing the original pre-training weight.
3. Method
3.1. Preliminary and Overview
This work adopts image foundation models (IFMs) for video understanding tasks. We leverage Mamba as an ef- ficient adapter, due to its linear complexity in long-term spatial-temporal modeling. For IFMs, following previous works (Pan al., 2022; Lin al., 2022b), we use CLIP as representative because of its rich semantic information, strong generalization, and multi-modal learning capabilities. Below, we first provide some preliminaries, and then present our architecture overview based on CLIP.
Unlike previous works which train Kalat-related architec- tures from scratch, aim build hybrid architecture on top of pre-trained IFMs. This approach allows to in- herit the linear complexity advantages of Mamba while fully leveraging the powerful representations learned by IFMs.
Mamba and Attention Hybrid Architectures. Consid- ering the efficiency of the Mamba structure, many meth- ods have tried to combine Mamba with other structures. For example, Jamba (Lieber et al., 2024) and Mamba-2- Hybrid (Waleffe et al., 2024) attempt to construct hybrid language models by interleaving attention, MI.P and Mamba layers. For vision tasks, Mamba Vision (Hatamizadeh & Kautz, 2024) combines Mamba, attention and convolution layers together. PoinTramba (Wang al.. 2024) utilizes both Transformer and Mamba encoders. However, these methods aim to find scratch training model, and thus focus more on architecture design such as the optimal attention- Mamba ratio. In contrast, adapting Mamba to pre-trained model has many constraints. Our architecture builds upon
CLIP. Given a image I mx CLIP first employs a patch embedding layer to encode the image into an initial image embedding, denoted 1 CLIP then consists of series of Transformer layers, where each layer performs two primary operations: Attent ion and FFN. Specifically, for the i-th layer with input feature I the forward process can be written as:
H++ FFN(Attention(TT)
(I)
Since CLIP is primarily an image model, it lacks inherent modeling of temporal information. Common approaches to adapting CLIP for video tasks typically involve adding addi- tional temporal modules to capture the sequential dynamics.
3

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Architecture Overview. Our method achieves parameter- efficient fine-tuning of CLIP by inserting small number of learnable parameters. Specifically, for each Transformer layer, we introduce two important stage: Divide and Mod- ulate, as shown in Figure 1. (1) In Divide stage, we aim to cut down the computational cost by narrowing the atten- tion range in pre-trained CLIP. We achieve this by replacing the original image-level attention into a non-overlapping 2D window-level attention on video sequence. Using this method, we can reduce the computational cost while still able to capture short-term dependencies. (2) In Modulate stage, we aim to introduce full spatial-temporal interaction and capture long-range dependency. We utilize the linear- time complexity State Space Model (SSM) for efficient long sequence processing. Then, to integrate SSM's output with the original CLIP output, we propose a sophisticated se- quence modulation operation to combine sequences from different sources effectively. Formally, the forward process in Equation (1) is modified as:
information locally inside each window. After attention operation, we concatenate the output of each window to obtain new sequence for next stage process. The complete Divide stage can be written as:
Divide(V") = Div sxy))
[ (5)
= isf''...,,s''!,
Complexity Analysis. For video sequence with spa- tial and temporal H x W x T, the computational com- plexity for full spatial-temporal attention is O ((HW))²). And the complexity for frame-by-frune frame spatial attentionis O ((HW)TT). For our Divide stage with window size w w, the complexity can be written as:
o ( HWT (n²2)) = O (w= HHT)- w
(6)
V FEM(Modullate((ILvic(((()))
By restricting the attention range within each window, we reduce the computational burden by large margin and achieve linear time complexity.
where V T HWTXC denotes the input video feature in the i-th layer. In the following sections, we will first detail the Divide stage (Section 3.2) and the Modulate stage (Sec- tion 3.3). Then, Section 3.4 describes the training process. And we provide an in-thpth discussion in Section 3.5.
3.3. Modulate Stage
In this stage, we aim to capture full spatial-temporal dynam- ies using Mamba, thus achieving linear-time complexity.
3.2. Divide Stage
Overview. The main part of this stage is SeqMod, a novel sequence modulation operation to inject lightweight Mamba features into pre-trained Transformer-base than directly modifying the features of IFMs, SeqMod can adjust features by learnable scale and bias sequences.
Similar to self-attention, our Divide stage transforms the input video sequence without altering its shape. Since CLIP model is initially trained for image level understanding, most previous adapter based methods tend to follow this pattern and utilize CLIP attention only process videos frame-by-frame frame (Pan et al., 2022; Yang et al., 2023a; Qing et al., 2023). While we aim to have a more precise control over the reception field of attention mechanism, and reduce the computational overhead accordingly. Thus, we propose to further divide each frame into multiple windows and apply local attention within each window. For input video feature V E #WTXC at the i-th layer, we cut it with fixed 2D window size w x w. Let the total number of window across the video be denoted as N, so that:
IFMs. Rather
In this stage, for any input sequence x', we first use one SSM layer introduced in Mamba (Gu & Dao, 2023) to learn two sequences representing the scale and bias for modulation parameter. Then we conduct SeqMod operation and inject full spatial-temporal information during the process. We will explain in detail below.
SSM Forwarding Layer. We use an SSM layer to introduce full
interaction and capture long-range de- pendency. Figure shows the detailed structure of our SSM forwarding layer. We use the similar architecture as pro- posed in Mamba (Gu & Dao, 2023). To extend the module's ability to output two sequences, for Divide stage output x", we slightly modify the original SSM layer to double the channel number of the output projection layer. After that, we split the output by channel to obtain two output sequences yi and y²2 This can be formulated as:
V + |s) ........... syy, skyl
(3)
s E N = HWT/W².
Here 82,82, ....,,,.................. SN non-overlapping flattened windows in frames of the video. For each window sh we apply self-attention independently:
yí.y SSM(x').
(7)
Attention(s
(4)
Besides, following VideoMamba (Li et al., 2024b), we also conduct multiple times of bidirectional scanning operations through both spatial and temporal dimensions.
Attention is the pre-tr CLIP attention layer. This attention mechanism allows the video sequence to exchange
4

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
S
We can observe from Equation (9) that the core of AdaN consists of three components: scale and bias, and skip con- nection. Note that, both the scale and bias are scalar, and the sequence is modulated globally. This global modu- lation best suits in tasks like style transfer, where image feature should be considered as a whole. However, in our scenario, the spatial-temporal information fine-grained. Condensing the whole sequence output from SSM into single scalar will obviously lose information. Therefore, we introduce sequence modulation operation: SegMod. It extends the scalar scale and bias item into tensors with the same shape as the input sequence, and thus provide a fine-grane fine-grained sequence-to-sequee to-suueeee modulation. Formally, this operation takes a similar form to that in Equation (9),
Spalial Scan
Temporal Scan
B
Lineer
Conv
x
Linear (c>>2c)
y1
BxLxc
BxLx
Limear
Y2
State Space Model
XxLx
and can be expressed as:
Figure 2. Detailed architecture of State Space Model (SSM) for- warding layer. The SSM module designed to learn two sequences Y1 and Y2 that are used in further modulation operation. We con- duct multiple times of bidirectional scanning through both spatial and temporal dimensions. The final projection channel is then doubled, and we split the output into two sequences.
(x.
= Y1 x
Y2 + bias
X
scale
skip conncction
(10)
where Y1 and Y2 are the output of SSM, serving as se- quential scale and bias; @ is element-wise multiplication. The modulated sequence is then fed into the original CLIP feed-forward layer for further processing:
Sequence Modulation Operation (SeqMod). For atten- tion output x' and SSM output vi, we aim to effectively integrate these sequences together. We draw inspiration from the adaptive normalization (AdaN), and design new sequence modulation operation. SeqMod. In the follow- ing, will detail the design of our sequence modulation operation.
v FFN(SegMod(x yx..ixxxx,....
(11)
And the output V is then input for the (i + 1)-th layer of Divide stage.
3.4. Training Process
The concept of adaptive normalization (AdaN) refers to series of methods that adjust the input's mean and variance globally with an outer condition. This concept was first in- troduced in FiLM (Feature- wise Layer Modulation) (Perez al., 2017). At the same time, Huang & Belongie (2017) implemented this adaptation before the network normaliza- tion layer, which led to the term "Adaptive Instance Nor- malization" (AdaIN). Dumoulin et al. (2018) summarized these concepts as "feature-wise transformations", which have since been widely applied across a variety of tasks such as image recognition and generative modeling. For example, StyleGAN (Karras et al., 2018) uses inject style information into images, while DiT (Peebles & Xie, 2022) applies it to achieve text-to-image diffusion task. The general AdaN formula can be written as follows:
We average the output feature from final layer of CLIP model to obtain video representation yo.
y. Average (V=).
(12)
Here L is the total number of CLIP layers.
All CLIP's parameters are kept frozen during training, and only newly introduced SSM layers are trainable. Besides the classification loss, we also add a CLIP distillation loss to keep CLIP's ability on zero-shot understanding and ensure the output feature space is not altered too much. The whole architecture is trained end-to-end.
3.5. Other Fusion Designs
In Figure 3, we show some other designs to integrate two sequences. (a) Weighted average. (b) Element-wise max pooling. (c) Concatenation in channel dimension. Unfor- tunately, the performance of these methods are suboptimal, evidenced in Table 5. Similarly, as evidenced in Yang et al. (2024c), Mamba-based module does not inherently fit Trans- former well. It is likely that directly inserting a completely new Mamba framework into an already trained Transformer may confuse the model, especially when the Transformer
AdaN(x) = x (rxx+(3))) Bc) + x,
(8)
Here, is the input sequence. (-)) is arbitrary module, and Oke, Se, Yo are learnable scaler parameters derived from the given condition. Without loss of generality, here we choose (xx) = x for simplicity. Then the derived form is:
AdaN(x) = Y. x + 3g3
X skip connection
scale
bias
5

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Table Comparison Kinetics-400 (Kay al., 2017) dataset. Views in the format of #frames crrammsxfficmpprrlkkkkttttt We compare
with previous PEFT methods as well as full- parameter- fineneeeee---------e fine--itttteee tuniig methods. Best results are in bold.
Methods
PEFT
Extra
Data
GFLOPs
Param (M)
Param(M)
Tunable
Top-1
Top-S
Views
MVTT-B (Fan et al., 2021)
4095
37
37
81.2
95.1
x4x3x3
TokenLeerner-L/110 (Ryoo al., 2021)
48912
450
450
85.4
96.3
64x4x3
MViTv2-L. (312 1) (Li et al., 2022)
42420
218
218
86.1
97.0
32x3x5
UniFormer-B (Li et al., 2021)
IN-Ik
3108
50
50
83.0
95.4
32x4x3
ViViT-L/16xx FE (Amab al., 2021)
IN-21k
3980
311
311
80.6
92.7
32x1x1
TimeSformer-L (Bertasius al., 2021)
IN-21K
7140
121
121
80.7
94.7
64x1x3
VideoSwin-L. (Liu et al., 2022)
IN-21k
7248
197
197
83.1
95.9
32x4x3
MTV-I (Yan ct al., 2022)
IN-21k
18050
876
876
84.3
96.3
32x4x3
VideoMamba-M (Li et al., 2024h)
CLIP-400M
2424
74
74
83.4
95.9
16x4x3
ActionCIIIP (Wang al., 2021)
CLIP-400M
16890
142
142
83.8
97.1
32x10x3
X-CLIP-L/14 (Ni et al., 2022)
CLIP-40DM
7890
420
420
87.1
97.6
8x4x3
AIM ViT-B/16 (Yang et al., 2023a)
CLIP-400M
1214
97
11
84.5
96.6
16x3x1
DiST ViT-B/16 (Qing et al., 2023)
CLIP-400M
986
112
26
84.4
96.7
16x3x1
MoMa (Ours) ViT-B/16
CLIP-400M
451
97
11
83.7
96.5
8x3x1
MoMa (Ours) ViT-B/16
CLIP-400M
902
97
11
84.8
96.9
16x3x1
EVL ViT-L/14 (Lin et al., 2022b)
CLIP-400M
8088
368
59
87.3
32x3x1
AIM ViT-L/14 (Yang et al., 2023a)
CLIP-400M
5604
341
38
87.3
97.6
16x3x1
DiST VIT-L/14 (Qing al., 2023)
CLIP-400M
4534
343
40
87.6
97.8
16x3x1
MoMa (Ours) ViT-L/114
CLIP-400M
2076
342
39
86.7
96.7
8x3x1
MoMa (Ours) ViT-1./14
CLIP-400M
4152
342
39
87.8
98.0
16x3x1
baselines, Section 4.2 for long video baselines, and Sec- tion 4.3 for zero-shot transfer. Section 4.4 provides ablation studies to analyze our model from multiple perspectives.
(a)
weighted average:
W1
W2 y
(b)
max pooling: max
x
y
Implementation Details. We use the pre-trained CLIP as our base model. We set the split window size SSM hyper-parameters,
For set its hidden state 16, hidden dimension 384 and use gelu activation layer similar with CLIP. We adopt the same prompt in ActionCLIP (Wang et al., 2021). We use 8 Tesla V100 GPUs and fp16 precision for training. We use Adam optimizer with learning rage 3e-4 and weight decay 0.05. Training a model on K400 dataset with 30 epochs takes about 12 hours converge.
(C)
concat channel:
concat
x
Figure 3. Some other designs of sequence fusion operations to integrate two sequences and y. (a) Weighted average operation with hyperparameter w1 and w2; (b) Element-wise max pooling operation; (c) Concatenation channel dimension.
parameters are frozen and unable to adapt to this new struc- ture.
4.1. Standard Video Recognition Benchmarks
Datasets. We first evaluate our method on Kinetics-400 (K400) (Kay et al., 2017) and Something-Somettie (SSv2) (Goyal et al., 2017). K400 is action recognition dataset containing 240K training videos and 20K valida- tion videos for 400 human action categories. Each video is trimmed to have a length of 10 seconds. SSv2 is a more challenging dataset requiring stronger temporal mod- eling (Sevilla-Laare al., 2021). It contains about 168.9K training videos and 24.7K validation videos for 174 classes.
By drawing inspiration from multi-modal fusion techniques, specifically from AdaN, our MoMa introduce Mamba's in- formation in a manner that minimally disrupts the original IFM's forwarding process while still effectively integrat- ing Mamba's processing results. In essence, this method strikes a balance between the power of IFMs and the ef- ficiency of Mamba's linear complexity, enabling effective video sequence understanding without the need discard the benefits of prior pre-training weights.
V2
Table and Table 2 show the comparison of our method with the state-of-the thettrtrttt methods on K400 and S$v2. Notably, we achieve the best performance while requiring substan- tially fewer FLOPs (25.6% off than AIM and 8.5% off than DiST) and trainable parameters than most other approaches.
4. Experiments
We performed a thorough evaluation of our model across several benchmarks: Section 4.1 for standard CLIPP-ddapter
6

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Table Comparison Samething-Somethigg (Goyal et al., 2017) dataset. Views in the format of #frames #ttramese*tmmppaallla>ae We compare with previous PEFT methods as well as full parl-parameter ceetrammter-fie--tee fine-tuning methods, Best results are in bold.
Methods
PEFT
Extra
Data
GFLOPs
Param (M)
Tunable Param(M)
Top-1
Top-S
Views
MVīT-B (Fan al., 2021)
X
510
37
67.1
90.8
32x 1 x3
M-VTv2-B (Li al., 2022)
X
70.5
92.7
40x
MViTv2-L (312 1) (Li et al., 2022)
8484
213
213
73.3
94.1
32x 22x1x3
UniFomer-B (Li et al., 2021)
X
IN-IK
777
50
50
71.2
92.8
32x1x3
ViViT-L/16x2 (Amab et al., 2021)
X
IN-21K
11892
311
311
65.4
89.8
16x4x3
TimeSformer-L (Bertasius et al., 2021)
X
IN-21k
7140
121
62.4
64x1x3
VideoSwin-B (Liu et al., 2022)
IN-21k
963
89
89
69.6
92.7
32x1x1
MTV-B (Yan et al., 2022)
IN-21k
4790
310
310
67.6
90.4
32x4x3
VideoMamba-M (Li et al., 2024h)
CLIP-400M
1212
74
74
71.0
92.7
16x2x3
EVL V VIT-B/16 et al.. 2022b)
CLIP-400M
2047
182
86
62.4
32 1x3
AIM ViT-B/16 (Yang al., 2023a)
CLIP-400M
2496
100
14
69.1
92.2
32x1x3
DiST ViT-B/16 (Qing al., 2023)
CLIP-400M
1972
112
26
70.9
92.1
32x1x3
MoMa (Ours) ViT-B/16
CLIP-400M
902
11
69.3
91.8
16x1x3
MoMa (Ours) ViT-B/16
CLIP-400M
1804
97
11
71.5
92.9
32x
EVL ViT-L/14(Lin et al., 2022b)
CLIP-400M
9641
175
66.7
32x1x3
AIM ViT-L/14 (Yang et al., 2023a)
CLIP-400M
11508
354
50
70.6
92.7
32x1x3
DiST VIT-L/14 (Qing et al., 2023)
CLIP-400M
9068
343
40
73.1
93.2
32x3x1
MoMa (Ours) ViT-L/14
CLIP-400M
4152
342
39
72.2
92.8
16x1x3
MoMa (Ours) VIT-1/14
CLIP-400M
8304
342
39
73.8
93.6
32x1x3
Specifically, on SSv2 dataset where the temporal model- ing more critical, our method outperforms the previous adapter methods like AIM and EVL by large margin, and is also better than DIST. But DiST introduces totally new coder to capture the temporal information, while our method only needs few adaptation layers inside the Transformer.
shows the comparison result. Our method surpasses DiST on both datasets, demonstrating the effectiveness of our method zero-shot transfer learning.
4.4. Ablation Studies
In this section, we introduce some baseline implementations under similar Mamba architecture. We also ablated each crucial part of our architecture. All ablations conducted the Kinetics-400 dataset and CLIP ViT-B/16 backbone with 16 frames as input, unless otherwise specified.
4.2. Long-term Video Recognition Benchmark
Datasets. To further demonstrate the effectiveness of our method in capturing long video sequence, we evaluate our method on Breakfast (Kuehne et al., 2014) and COIN (Tang et al., 2019). Breakfast comprises 1,712 12 videos, encapsu- lating 10 intricate cooking activities over 77 hours. COIN features 11,827 videos across 180 unique procedural tasks, with average duration of 2.36 minutes. Following Video- Mamba's is (Li et al., 2024b) setting, further PEFT our models trained on K400 from Table 1.
Baselines for Sequence Modulation Operation. As dis- cussed in Section 3.5, inserting Mamba module directly into well pre-trained Transformer architecture sub-optimal. Here we compare our method with other implementations to integrate the two sequences. (1) Skip: We skip the whole SSM forward module and use sequence from Divide op- eration directly, which can be seen as lower bound. (2) Add: We add the two sequences in an 1:1 ratio; (3) Max: Replacing addition with element-wise max introduces non- linearity, and this may enhance the model capability; (4) Concat: We concatenate the two sequences in channel and then linear layer reduce the dimension; (5) Raw- AdaN: We use the vanilla AdaN formula which learns scaler scale and bias.
Table 3 shows the comparison result. Our method surpasses those non end-to-end methods by large margin, demon- strating the effectiveness of our method in capturing long- term video understanding. Meanwhile, by conducting PEFT on top of CLIP model, also outperforms VideoMamba.
4.3. Zero-shot Ability
Similar to DiST (Qing et al., 2023), our method also has the ability perform zero-shot tasks. Following DiST's setting, rel-
Table 5 shows the comparison result. We find that sim- ple fusion methods such as Add and Max performs even worse than purely skip the whole SSM block. This indi- cates fusing different output sequences directly may lead information "confusion". especially when the two features
the model trained on K400, and evaluate atively small video recognition datasets: HMDB51 (Kuehne et al., 2011) and UCF101 (Soomro et al.. 2012). Table 4
7

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Table Comparison on long video datasets Breakfast (Kuehne et al., 2014) and COIN (Tang et al., 2019). f32 and fas denote number of frames sampled during training and testing. Best results are in bold.
Model
end-to-end
Archi
Extra Data
Breakfast
COIN
Timeception (Husscin al., 2019a)
X
3D-ResNet Conv.
IN-1K+K400
71.3
VideoGraph (Hussein al., 2019b)
X
I3D Cone.+Atten.
IN-IK+K400
69.5
GHRM (Zhou et al., 2021)
I3D Graph Conv. TimeSformer Atten. w/ KB
IN- IN-1K+K400
75.5
Distant Supervision (Lin et al., 2022a)
INTM
89.9
90.0
ViS4mer (Islam & Bertasius, 2022)
X
Swin-B SSM
IN-21K+K600
88.2
88.4
F32 (Han et al., 2022)
VideoMAE-B
K400
86.8
82.3
T32 (Han et al., 2022)
VideoMAF-B
K400-HTMAA
91.3
87.5
VideoMambayss2 (I. al., 2024h)
VadhoMaMMa-M
K400
94.8
88.3
VideoMambays4 (I.i et al., 2024b)
VidcoMamaa-M
K400
95.8
89.5
MoMa (3u
CLIP VIT-L/14
K400+CLIP-4000
95.1
89.0
MoMa (O4
CLIP VIT-L/14
K400+CLIP-400M)
96.9
90.0
Table Comparison of zero-shot action recognition performance on HMDB51 (Kuehne et al.. 2011) and UCF101 (Soomro et al., 2012) datasets. All models are based on CLIP VIT pre-training.
Table 6. Evaluation of various window sizes during the Divide
stage. In addition to the 2D window, we also evaluate 3D window with dimensions x x 4. Increasing the window size does not always bring positive gain.
Method
Model
HMDB51
UCF101
Window Size
Frame/Sec
Attn(%)
FFN(%)
Top-1
ActionCLIP(Wang et al., 2021)
B/16
40.8
58.3
X-CLIP (Ni et al., 2022)
B/16
44,6
72.0
Full (40 30)
8.2
39.7
41
69.2
DiST (Qing et al., 2023)
B/16
55.4
72.3
16 x 16 8 x 8 (Ours)
10.4
31.7
52
70.3
DiST (Qing et al., 2023)
L/14
57.5
74.9
14.0
23.9
70
70.1
MoMa (Ours)
B/16
56.2
74.0
x 4 X 4 (3D)
14.0
23.9
70
68.2
MoMa (Ours)
L/14
59.1
76.2
4x1
15.1
18.8
75
67.5
Table Comparison of different sequence modulation opera- tions. We implement three baseline methods: "Add", "Max" and
Thus, the input feature scale is 40 x 30 16 (patches). Beside 2D window, we also explore 3D window.
"Concat" as mentioned Figure 3. "Skip" is also provided as lower bound. Our operation "SeqMod" outperforms all baselines.
Table 6 shows the comparison result. The divide opera- tion brings both speed up and performance gain. Speed: Divide the window from full to increase in speed (175% fps) since attention operation dom- inates the computation. However, further dividing brings marginal improvement when FFN becomes the bottleneck. Performance: Conducting full attention on the whole image performs worse than window dividing. That's may because CLIP originally trained on attention sequence. Understand- ing long sequence is out of its training scope.
Methods
Operation
Top-1
Top-5
Skip
72.4
90.8
brings significant
Add
x + y max y)
69.3
88.7
Max
70.2
89.6
Concat
Lincar( [z. v))
75.5
92.5
Raw-AdaN
Cty i + By
78.9
94.1
SegMod
V1
x 1/2
84.8
96.5
misaligned have inconsistent distributions. While "Concat" preserves more information by adding learnable linear layer, it does not guarantee that the modification of the sequence is orthogonal. Raw-AdaN, on the other hand, uses a "soft gating" mechanism, and changes to the sequence's 's features are orthogonal to the original CLIP feature outputs. This helps avoid altering the feature distribution of the orig- inal CLIP too much, reducing interference with pre-trained knowledge. Built on top of this mechanism, "SeqMod" operation shows powerful sequence fusion capabilities, far surpassing all baselines.
Different Layer Design Pattern. We offer several design of architectural patterns below: (1) [TM]12: Alternating sequence of Transformer and SSM layers. each repeated 12 times (our current design). (2) [T] [ 12 consecutive Transformer layers followed by 12 SSM layers, with the SSMs functioning as the decoder rather than being inserted as an adapter in the middle of the encoder.
(3) [T]6[TM]6. No modulation for the first half of the Transformer layers. (4) [TTM]6 One modulation layer is inserted between every two transformer layers
Window Size in Divide Operation. We here explore the impact of window size in the Divide operation on SSv2 dataset. We upscale the video resolution to 640 x 480 x 16.
Table 7 shows the result. We find that: (1) Adapting only the latter half of the backbone is sub-optimal, Since learn to recognizing temporal information in the early stages of
8

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Table Ablation study on various layer design patterns. We found that the best performance is achieved by uniformly alternat- ing between transformer and mamba layers.
AIM
Ours
UMT
Inference Speed
GPU Memory
Pattern
Top-1
Top-5
200
(I)
[TM] 12 (Ours)
83.7
96.5
FPS
150
ITI 12 (M] 12
80.6
92.8
[T] [TMM] 6
81.5
94.0
100
[TTMM] 6
82.8
95.5
25
50
75
100
125
25
50
100
125
Frame
Frame
video encoding is essential for effective processing. (2) The [TTMM]6 pattern performs worse than [TM]12, suggesting that a more balanced integration of both components in the adapter structure is more beneficial.
Figure 4. Comparison speed performance. Compared with full attention method UMT and spatial temporal separate method AIM, our method, powered by the SeqMod operation, achieves faster processing with more gradual increase in memory usage.
4.5. Speed Comparison
We compare the speed performance of MoMa with two clas- sic video sequence processing approaches: full-attention and spatial-temporal separation. Specifically, we select rep- resentative methods from each category. UMT (Li et al., 2023) uses full-attention, treating the entire video sequence as a single input, leading to quadratic complexity and high computational cost. AIM (Yang et al., 2023a), on the other hand, applies spatial and temporal attention separately, first the spatial level and then the temporal level.
Impact Statement
This paper presents work aimed at advancing video under- standing by improving the ciency of pre-trained image foundation models. Our method has the potential to enhance various applications, such as video analyties in healthcare, security, and autonomous sys- tems. While there are broad societal implications, including privacy concerns and ethical considerations in rideo-based AL, we do not believe any specific issues need to be high- lighted beyond the general ethical discussions in the field of machine learning.
modeling effi-
As shown in Figure 4, the GPU memory usage of UMT increases rapidly with the number of input frames and even- tually runs out of memory at 32 frames. Its inference FPS also drops significantly as the frame count rises. In contrast, both AIM and our method show stable memory growth and a gradual decrease inference speed as the number of frames increases. However, thanks to our powerful SeqMod opera- tion, our method achieves faster processing and experiences more gradual increase in memory consumption. Addition- ally, the SSM module is more parameter-efficient than the attention module, further enhancing the overall efficiency of our approach.
A Acknowledgement
This work is supported by the National Key R&D Program of China (No. 2022ZD0160703), National Natural Seience Foundation of China (No. 62306178) and STCSM (No. 22DZ2229005), 111 plan (No. BP0719010).
References
Amnab, A., Dehghani, M.. Heigold, G., Sun, C., Lučié, M., and Schmid, C. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 6836-6846, 2021.
5. Conclusion
In this work, we proposed a novel method to adapt IFMs for video understanding. We first introduces a modulation operation SeqMod. It injects spatial -ertialltempoal information into the features of IFMs without interfering with their pre- trained parameters, thereby maintaining computational effi- ciency. We further proposed our architecture MoMa with Shivide-and-Modullld
Bao, H., Dong, L., Piao, S., and Wei, F. BEÎT: BERT Pre-Training of Image Transformers. arXiv preprint 2021.
strategy. It applies local attention within each video frame, followed by modulation step to capture the full spatial-temporal dynamics. Through exten- sive experiments on multiple video understanding bench- marks, we demonstrate that our approach MoMa improves both performance and efficiency compared to existing meth- ods, providing balance between accuracy and scalability.
Bertasius, G., Wang, H., and Torresani, L. Is space-time attention all you need for video understanding? In Pro- ceedings of the International Conference Machine Learning (ICML). July 2021.
Byeon, M., Park, B., Kim, H., Lee, S., Back, W., and Kim, S. 700m-700m: Image-text pair dataset, 2022.
9

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Carreira, J. and Zisserman, A. Quo vadis, action recogni- tion? model and the kinetics dataset. In proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition, Pp. 6299-6999962 6308, 2017.
Han, B., Yao, J., Liu, T., Li, B., Koyejo, S., Liu, E,, al. Trustworthy machine learning: From data to models. Foundations and Trends@ in Privacy and Security, 7(2-3): 74-246, 2025.
Chen, G., Huang, Y., Xu, J., Pei, B., Chen, Z., Li. Z., Wang, J., Li, K., Lu, T., and Wang, L. Video mamba suite: State space model as a versatile alternative for video understanding. arXiv preprint arXiv: 2403.09626, 2024.
Han, T., Xie, W. and Zisserman, A. Turbo training with token dropout. British Machine Vision Conference, 2022.
Hara, K., Kataoka, H., and Satoh, Y. Learning spatio- temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE international conference on computer vision workshops, pp. 3154- 3160, 2017.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations, 2020.
Dumoulin, V., Perez, E., Schucher, N., Strub, E., Vries, H. d., Courville, A., and Bengio, Y. Feature-wise transforma- tions. Distill, 2018.
Hatamizadeh, A. and Kautz, J. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083, 2024.
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6824-6835, 2021.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. Clipscore: reference-free evaluation metric for im- age captioning. Conference Empirical Methods in Natural Language Processing. 2021.
Feichtenhofer, C. X3d: Expanding architectures for effi- cient video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 203-213, 2020.
Huang. and Belongie, S. Arbitrary style transfer in real- time with adaptive instance normalization. In Proceed- ings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Feichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast networks for video recognition. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 6202-6211, 2019.
Hussein, N., Gavves, E., and Smeulders, A. W. Timeception for complex action recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019a.
Feichtenhofer, C., Fan, H., Li, Y., and He, K. Masked autoencoders as spatiotemporal learners. arXiv preprint arXiv:2205.091 13, 2022.
Hussein, N., Gavves, E., and Smeulders. A. W. M. Video- graph: Recognizing minutes-long human activitics in videos. arXiv preprint arXiv: /905.05143, 2019b.
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueiler-Freitag, M., et The" something something" video database for learning and evaluating visual com- sense. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5842-5850, 2017.
Islam, M. M. and Bertasius, G. Long movie clip classifica- tion with state-space video models. Computer Vision ECCV 2022, pp. 87-104. Springer Nature Switzerland, 2022.
Jia, C., Yang, Xia, Y., Chen, Y.-.., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, 7., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning, pp. 4904-4911 PMLR, 2021.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprini arXiv:2312.00752, 2023.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 2021a.
Karras, T., Laine, S., and Aila, T. A style-based generator ar- chitecture for generative adversarial networks. Computer Vision and Pattern Recognition, 2018.
Gu, X., Lin. T.-Y., Kuo, W., and Cui, Y. Open-vocabulary object detection via vision and language knowledge dis- tillation. International Conference on Learning Repre- sentations, 2021b.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv:/70s. 1705006500 2017.
06950,
10

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. Hmdb: A large video database for human motion recog- nition. In 2011 International Conference Computer Vision, pp. 2556-2563 2263, 2011.
Lin, X., Petroni, F., Bertasius, G., Rohrbach, M., Chang, S.-F., and Torresani, L.. Learning recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13853-13863, June 2022a.
Kuehne, H., Arslan, A., and Serre, T. The language of actions: Recovering the syntax and semanties of goal- directed human activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
Lin, Z., Geng, S., Zhang, R., Gao, P., de Melo, G., Wang, X., Dai, J., Qiao, Y., and Li, H. Frozen clip models are efficient video learners, arXiv preprint arXiv:2208 20:::2000003000 2022b.
Kuo, W., Cui, Y., Gu, X., Piergiovanni, A., and Angelova, A. F-vim: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv: 2209.15639, 2022.
Liu, J., Wang, Y., Ju, C., Ma, C., Zhang, Y., and Xie, W. Annotation-free audio-visual segmentation. arXiv preprint arXiv: 2305.11019 2023.
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L.,, Wang, Y., Ye, Q.. and Liu, Y. Vmamba: Visual state space model. arXiv preprint 2411:1240..10166, 2024.
Li, J., Shi, Y., Cao, J., Ni, B., Zhang, W., Zhang, K., and Gool, L. V. Mipmap-gs: gs: Let gaussians deform with scale-specific mipmap for anti-aliasing rendering. arXiv preprint arXiv:
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., and Hu, H. Video swin transformer. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 3202-3211, 2022.
3408.06286, 2024a.
Li, K., Wang, Y., Peng, G., Song, G., Liu, Y., Li, H., and Qiao, Y. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Conference on Learning Representations, 2021.
Ma, C., Yang. Y., Wang, Y., Zhang. Y., and Xie, W. Open- vocabulary semantic segmentation with frozen vision- language models. In British Machine Vision Conference, 2022.
Li, K., Wang, Y., Li. Y., Wang. Y., Hc. Y., Wang. L., and Qiao, Y. Unmasked teacher: Towards training- efficient video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19948-19960. 2023.
Ma, C., Yang, Y., Ju, C., Zhang, F., Liu, J., Wang, Y., Zhang, Y., and Wang, Y. Diffusionseg: Adapting diffusion to- wards unsupervised object discovery. arXiv preprint arXiv: 000300913,,.. 2023a.
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., and Qiao, Y. Videomamba: State space model for efficient video understanding, 2024b.
Ma, C., Yuhuan, Y., Ju, C., Zhang, F., Zhang, Y., and Wang, Y. Attrseg: open- vocabuacayylary semantic segmentation via attribute decomppsittion aggrregaton......in Advances in neural information processing systems, 16:1088--0070, 2023b.
Li, Y., Wu, C.-Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., and Feichtenhofer, C. Mvitv2: Improved multiscale vision transformers for classification and detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4794 4804, 2022.
Ma, C., Yang, Y., Ju, C., Shi, Y., Zhang, Y., and Wang, Y. Freesegdiff: Annotation saliency segmentation with diffusion models. In International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1-5, 2025.
Li, Y., Dou, Y., Shi, Y., Lei, Y., Chen, X., Zhang, Y., Zhou, P.. and Ni, B. Focaldreamer: Text-driven 3d editing via focal-fusion assembly. In Proceedings of the AAAI con- ference artificial intelligence, pp. 3279-3287, 2024c.
Minderer, M., Gritsenko, A., and Houlsby, N. Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems, 36:72983-73007, 2023.
Liang, D., Zhou, X., Xu, W., Zhu, X., Zou, 7.,, Ye, X., Tan, X., and Bai, X. Pointmamba: A simple state space model for point cloud analysis. In Advances in Neural Information Processing Systems, 2024.
Mokady, R., Hertz, A., and Bermano, A. H. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv: 2111.09734, 2021.
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., et al. Jamba: A hybrid transformer mamba language model. arXiv preprint arXiv:2403. 19887, 2024.
Ni, B., Peng. H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., and Ling, H. Expanding language-image pretrained models for general video recognition. arXiv preprint arXiv:2208.022 028/6, 2022.

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Nukrai, D., Mokady, R., and Globerson, A. Text-only train- ing for image captioning using noise-injected clip. arXiv 2022.
C., Wortsman, M., Schramowski, P., Kundurthy, S., Crow- son, K., Schmidt, L.., Kaczmarczyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv: 22 2210.08402, 2022.
preprint arviv:
Oquab, M., Darcet, T., Moutakanni, T. Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.-Y., Xu, H., Sharma, V., Li, S.-W., Galuba, W., Rabbat, M., As- sran, M., Ballas, N., Synnaeve, G., Misra, L., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski. P. Dinov2: Learning robust visual features without supervi- sion, 2023.
Sevilla-Lara, L.,, Zha, S., Yan, Z., Goswami, V., Feiszli, M., and Torresani, L. Only time can tell: Discovering temporal data for temporal modeling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 535-544, 2021.
Shi, Y., Rong, D., Chen, C., Ma, C., Ni. B., and Zhang, W. Darf: Depth-aware generalizable neural radiance field. Displays, 88: 102996, 2025.
Pan, J., Lin, Z., Zhu, X., Shao, J., and Li, H. St-adapter: Parameter-efficient image-to-vided transfer learning for action recognition. arXiv preprint arXiv:2206.135555 2022.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12 12.0422, 2012.
Pan, Z., Zeng, H., Cao, J., Chen, Y., Zhang, K., and Xu, Y. Mambasci: Efficient mamba-unet for quad-bayer patterned video snapshot compressive imaging. arXiv preprint arviv: +410.14214, 2024.
Tang, Y., Ding, D., Rao, Y., Zheng, Y., Zhang, D., Zhao,
L., Lu, J., and Zhou, J. Coin: large-scale dataset for comprehensive instructional video analysis. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Park, J., Lee, J., and Sohn, K. Dual-path adaptation from image to video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2203-2213, 2023.
Teng, Y., Wu, Y., Shi, H., Ning, X., Dai, G., Wang, Y.. Li, Z., and Liu, X. Dim: Diffusion mamba for efficient high-resolution image synthesis, 2024.
Peebles, W. S. and Xie, S. Scalable diffusion models with transformers. IEEE International Conference on Com- puter Vision, 2022.
Tong. Z., Song, Y., Wang, J., and Wang, L. Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203. 12002, 2022.
Perez, E., Strub, F., Vries, H. D., Dumoulin, V., and Courville, A. C. Film: Visual reasoning with a gen- eral conditioning layer. AAAI Conference on Artificial Intelligence, 2017.
Waleffe, R., Byeon, W., Riach, D., Norick, B., Kor- thikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., al. An empirical study of mamba- based language models. arXiv preprint arXiv:2406.0788 2024.
Qing, Z., Zhang, S., Huang, Z., Zhang, Y., Gao, C., Zhao, D., and Sang. N. Disentangling spatial and temporal learning for efficient image-to-video transfer learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 13934-13944, 2023.
Wan, Z., Zhang, P., Wang, Y., Yong, S., Stepputtis, S., Sycara, K., and Xie, Y. Sigma: Siamese mamba network for multi-modal semantic segmentation. arXiv preprint arXiv: 2404.04255 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-87878 8763. PMLR, 2021.
Wang, M., Xing, J., and Liu, Actionclip: new paradigm for video action recognition. arXiv preprint arXiv: (tfi::/099.044:4 2021.
Ryoo, M. S., Piergiovanni, A., Arnab, A., Dehghani, M., and Angelova, A. Tokenlearner: Adaptive space-time tokenization for videos. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
Wang, W. et al. Image as a Foreign Language: BEiT Pre- training for All Vision and Vision- LanguaLe Tasks. arXiv preprint arXiv:2208. 10442, 2022.
Wang, Z., Chen, Z., Wu, Y., Zhao, Z., Zhou, L., and Xu, D. Pointramba: A hybrid waasformer-mante framework for point cloud analysis. arXiv preprint arXiv: 2405.15463. 2024.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis,
12

---

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
Xiong, Y., Shi, Y., Dou, Y., and Ni, B. Dualnerf: Text-driven 3d scene editing via dual-field representation. arXiv preprint arXiv: 2502.16302. 2025.
Zhong, Y., Yang. J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, I.., Li, Y., et al. Regionclip: Region-based language-image pretraining. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16793-16803, 2022.
Xue, H., Sun, Y., Liu, B., Fu, J., Song. R., Li, H., and Luo, Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint arXiv:2209.064333 2022.
Zhou, J., Lin, K.-Y., Li, H., and Zheng, W.-. Graph- based high-order relation modeling for long-term action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 8984 8994,999,, June 2021.
Yan, S., Xiong, X., Arna,, A., Lu, Z.,, Zhang, M., Sun, C., and Schmid, C. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3333- 3343, 2022.
Zhu, L.,, Liao, B., Zhang. Q. Wang, X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learn- ing with bidirectional state space model. arXiv preprint arXiv:2401.09417 2024.
Yang. S., Wang, Y., and Chen, H. Mambamil: Enhancing long sequence modeling with sequence reordering in com- putational pathology. arXiv preprint arXiv:2403.0680 2024a.
Yang, T., Zhu, Y., Xie, Y., Zhang, A., Chen, C., and Li, M. AIM: Adapting image models for efficient video action recognition. In The Eleventh International Conference on Learning Representations, 2023a.
Yang, Y., Ma, C., Ju, C., Zhang, F., Yao, J., Zhang, Y., and Wang. Y. Multi-modal prototypes for open- world se- mantic segmentation. International Journal of Computer Vision, (32(12)):004-602) 2024b.
Yang. Y., Ma, C., Yao, J., Zhong, Z., Zhang, Y., and Wang. Y. Remamber: Referring image segmentation with mamba twister. European Conference on Computer Vision (ECCV), 2024c.
Yang, Z., Xu, Q., Bao, S., Cao, X., and Huang, Q. Learning with multiclass auc: Theory and algorithms. IEEE Trans- actions Pattern Analysis and Machine Intelligence, 44 (11):7747-7763, 2021.
Yang. Z., Xu, Q. Bao, S., Hc, Y., Cao, X., and Huang, Q. Optimizing two-way partial auc with an end-to-end framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1528):1022 2023b.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. arXiv preprint arXiv: 2303, 1 15343, 2023.
Zhang, T., Li, X., Yuan, H., Ji, S., and Yan, S. Point cloud mamba: Point cloud learning via state space model. arXiv preprint arXiv:2403. 00762, 2024.
Zhang, Y., Li, X., Liu, C., Shuai, B., Zhu, Y., Brattoli, B., Chen, H., Marsic, I., and Tighe, J. Vidtr: Video transformer without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13577-13587, 2021.
13


Reviews:
[{"review_id": "jk3GSIw9HT", "reviewer": "Reviewer_5nFH", "summary": "This paper presents MoMa, a video foundation model that is built on top of the image foundation model by leveraging Mamba as an efficient adapter. Specifically, Mamba block is used to capture spatial-temporal dynamics without interfering with the pre-trained IFMs. Besides, to avoid excessive computational overhead, the authors also use window attention to reduce the attention module’s computation cost. The proposed MoMa shows competitive results on video classification benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">## 1. Marginal improvement on K400 and SSv2 datasets\n\nWe focus on striking a balance between performance and efficiency instead of solely pursuing the highest performance. Besides achieving state-of-the-art performance, we managed to cut nearly 30% calculation FLOPs off with minimum training parameters compared with other PEFT methods. Furthermore, our speed advantage becomes increasingly pronounced with more frames while sustains its performance superiority, as demonstrated in Figure 4.\n\nWe also conduct another experiment to demonstrate our model's potential by equipping MoMa with MLLMs and answering complicated video understanding tasks. Please see our response to ``R4(17be) #2.1 Equipping MoMa into MLLMs`` for detail.\n\n>## 2. Further analysis on SeqMod operation.\n\nWe apologize for the omission of a comparison with the raw AdaN method and some explanation in our initial submission. Here we supplement more detailed comparative experiments.\n\n| Number | Methods  | Operation| Top-1 | Top-5 |\n| --| -------- | -| ----- | ----- |\n| 1 | Skip     | $x$ | 72.4  | 90.8  |\n| 2 | Add      | $x + y$ | 69.3  | 88.7  |\n| 3 | Max      | $max(x, y)$ | 70.2  | 89.6  |\n| 4 | Concat   | $Linear([x, y])$  | 75.5  | 92.5  |\n| 5 | Raw-AdaN | $\\alpha_y\\cdot x+x+\\beta_y$ | 78.9  | 94.1  |\n| 6 | SeqMod   | $\\mathbf{y_1} \\odot x + x + \\mathbf{y_2}$ | 84.8  | 96.5  |\n\nThe Raw-AdaN method learns scalar parameters $\\alpha_y$ and $\\beta_y$ for adaptation. The table shows that the Raw-AdaN already outperforms other methods like Add, Max, etc. Inspired by its promising results, we further design SeqMod in our paper. We here give some detailed discuss.\n\n\n### 2.1. Why Raw-AdaN superior Add/Max/Concat, etc.\n\nThis is because the feature spaces of Transformer and Mamba do not match or are not perfectly aligned, as observed in [1]. Direct Add/Max/Concat operations interfere with the feature of the original Transformer model.\n\n[1]: ReMamber: Referring Image Segmentation with Mamba Twister\n\n* Add/Max: Adding the output sequence of SSM $y$ to the original features $x$ can lead to information “confusion”, especially when the two features are misaligned or have inconsistent distributions. These two operations can cause information overlap, leading to the loss of key details, especially in multi-scale spatial and temporal features.\n\n* Concat: While it preserves more information by adding learnable linear layer, it does not guarantee that the modification of the sequence is orthogonal. \n\nDifferent from above, Raw-AdaN uses a global scalar modulation. The scalar acts as a “soft gating” mechanism, where the changes to the sequence’s features are **orthogonal** to the original CLIP feature outputs. This helps avoid altering the feature distribution of the original CLIP outputs when learning global spatiotemporal representations, reducing interference with pre-trained knowledge.\n\n### 2.2 Why SeqMod significantly outperform Raw-AdaN\n\n**Fine-grained spatiotemporal awareness for video tasks.**\n\nSpatiotemporal modeling is the core to video understanding tasks. Though powerful, Raw-AdaN’s global scalar adjustment (with the same scaling and bias parameters shared across all positions) cannot distinguish dynamic changes in different regions, which is essential for video tasks.\n\nBy using fine-grained modulation (retaining independent spatiotemporal modulation parameters at each position), we can capture more complex spatiotemporal dynamics, thus preventing information loss. \n\n\n**Experimental results**\n\nThe table above (lines 5 and 6) shows that extending Raw-AdaN’s global scalar modulation to SeqMod’s fine-grained spatiotemporal modulation significantly improves model performance (both $y_1$ and $y_2$ are vectors generated by SSM).\n\n**To sum up:**\n\n1. AdaN-like methods bring **orthogonal** modulation to the original CLIP, making it more suitable for adaptation tasks and avoid altering the feature distribution too much.\n2. SeqMod further extends the AdaN-like methods to **fine-grained** spatiotemporal modulation, which is more suitable for video tasks.\n\nWe will add the ablation and the according discussion in our final version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, using a sub-quadratic attention module for spatial-temporal modeling makes sense.", "theoretical_claims": "This paper does not provide any theoretical proof.", "experimental_designs_or_analyses": "The experimental design in this paper looks reasonable and makes sense.", "relation_to_broader_scientific_literature": "The design of MoMa and SeqMod are both highly related to the efficient video modeling field.", "essential_references_not_discussed": "I do not identify any important but missing references in this paper.", "other_strengths_and_weaknesses": "**Weaknesses**\n\n1. The improvement over previous methods appears to be quite marginal on both the K400 and SSv2 datasets.\n2. The results in Table 5 do not seem entirely reasonable, as SeqMod shows an improvement of more than 10 points compared to each alternative design, despite their mathematical structures being quite similar. I suggest the authors conduct further analysis and provide a theoretical explanation for this discrepancy.", "other_comments_or_suggestions": "Please refer to weaknesses.", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U4OgKX3fg2", "reviewer": "Reviewer_jWpG", "summary": "The paper proposes using Mamba layers as adapters to apply CLIP pre-trained transformer-based image models for video tasks. For each transformer block, the proposed method first divides each frame into multiple windows, applies self-attention within each window, then the tokens of all windows of all frames are concatenated into a flat sequence, over which Mamba layer is applied in multiple scanning orders. The output of Mamba is linearly projected into a 2x dimensional space and split into two to produce a scale and bias terms that are used to modulate the self-attention (image) features, taking inspiration from Adaptive Normalization works. Experiments are carried out on standard video classification datasets (Kinetics400, SSv2) and long-term video recognition datasets (COIN, Breakfast), and zero-shot transfer to smaller scale datasets (HMDB51, UCF101). Ablation experiments are presented to justify architectural choices, mainly the way the modulation of the image features is performed and the size of the window in the divide stage. The proposed method outperforms existing baselines on all datasets, while having reduced latency and memory footprint compared to spatio-temporal attention architectures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">## 1. Divide part is not justified, using higher resolution inputs.\n\nThanks for advice! Our original $224 * 224$ comparison aimed to align with other baselines, but we’ve now conducted additional ablation studies at $640*480$ resolution (SD video standard) to address your concern.\n\n**Experiment details**\n- Input video resolution $640 * 480 * 16$\n- CLIP ViT-B/16 backbone\n  - After image embedding layer, the feature shapes $40 * 30 * 16$ (patches).\n- Window size is defined on patch level.\n\n|Window Size|Speed(frame/s)|Attention(%)|SSM(%)|FFN(%)|Top-1(Acc)|\n|-|-|-|-|-|-|\n|Full| 8.2 |39.7|1.7|41|69.2|\n|16| 10.4|31.7|1.6|52|70.3|\n|8(Ours)| 14.0|23.9|2.1|70|70.1|\n|4| 15.1|18.8|2.0|75|67.5|\n\n(* Experiment conducted on SSv2. Speed tested on A100.)\n\nFrom the table we observe that the divide operation brings both **speed up** and **performance gain**.\n- **Speed**: Divide the window from full to $8*8$ brings significant increase in speed (175% fps) since attention operation dominates the computation. However, further dividing brings marginal improvement when FFN becomes the bottleneck.\n- **Performance**: Conducting full attention on the whole image performs worse than window dividing. That's may because CLIP originally trained on $16*16$ attention sequence. Understanding long sequence is out of its training scope.\n\n>## 2. Ablation of the Mamba state size\n\nI'm not sure whether you're asking for Mamba's state size or the model’s hidden dimension.\n\nFor state size, we’ve set to 16 as default. Increasing the state size does not enhance the model’s capacity. Even large-scale models like falcon-mamba-7b ([1]) and Jamba (52B MoE) ([2]) use a state size of 16.\n\n[1] huggingface:falcon-mamba-7b\n[2] huggingface:Jamba-v0.1\n\nRegarding the model’s hidden dimension, we’ve set to 1024, which aligns with the hidden size used in CLIP. Further increasing the hidden dimension would result in a denser Mamba component, making training more challenging, as it would primarily increase parameters related to MLPs.\n\nWe also conducted a comparison to assess the impact of increasing Mamba’s capacity by expanding the hidden dimension. With nearly double the number of parameters, the improvement was marginal.\n\n|Hidden Dim|Top-1|Top-5|\n|-|-|-|\n|1024|83.7|96.5|\n|2048|83.9|96.5|\n\nAdditionally, we’ve provided a new ablation on SeqMod operation. Please refer to our response to ``R1(5nFH) #2: Further analysis on SeqMod operation``.\n\n>## 3. Adding a VLM task\n\nYes. See our response to ``R4(17be) #2.1 Equipping MoMa into MLLMs`` for detail.\n\n>## 4. Experimental Designs\n### 4.1 Fine-tuning results on K400 and SSv2\nMoMa is an adapter method that conducts parameter-efficient fine-tuning (PEFT). Which means, it always fine-tunes a pre-trained image encoder (eg., CLIP) with a small number of trainable parameters. Therefore, all the experiments in Tab.1-3 are finetuning experiments:\n- Tab.1: Fine-tune CLIP on K400\n- Tab.2: Fine-tune CLIP on SSv2\n- Tab.3: Fine-tune Tab.1 checkpoint on COIN and Breakfast\n\n### 4.2 Long-video understanding experiment\nSorry for term mis-use. Our MoMa should be clarified as non-e2e method since it conduct PEFT. We will correct this.\n\nFor implementation details, both standard and long-video training follows the details in Sec.4. Batch size varies according to the input frame. When input 32 frames, the batch size is 8/gpu.\n\n### 4.3 Fix misleading table\nThanks for kind reminder!\n\nWe used dividing lines to distinguish between non-PEFT and PEFT methods. All PEFT methods are based on CLIP and can be fairly compared with ours. For non-PEFT methods, there are both large-scale data based methods (eg., ActionCLIP) and those based solely on ImageNet (eg., ViViT).\n\nTo avoid misleading, we'll update Tab.1-3 to include pre-training datasets, and we'll take CLIP pre-training dataset into account.\n\n### 4.4 12-Hour training time\nAlthough the CLIP parameters do not require training, they still participate in gradient propagation since the adapter layers are interspersed between the CLIP transformer layers. In fact, for video understanding, 12 hours of training time is already quite fast. For comparison, AIM takes over 16 hours for training with the same number of parameters.\n\n>## 5. Literature review\n\nThank you for the suggestion. Indeed, there are several excellent linear-complexity sequence models besides Mamba, such as S4, TRecViT and RWKV. We will make sure to include them in the related work section.\n\n>## 6. Typos and unclear notions\n\nWe apologies for any confusion caused by the typos and unclear notions. Here are some clarifications:\n\n- The $16*16$ windows are expressed in terms of patches.\n- The frame number 16 refers to extracting 16 frames from a video clip for downstream tasks (randomly sampling for training, average sample for inference).\n- $y$ or $y_1,y_2$: We modify the SSM module to output two sequences $(y_1, y_2)$ during SeqMod operation. For the baseline operations, the SSM module only outputs a single sequence $(y)$.", "claims_and_evidence": "The use of the Divide part (self-attention applied on sub-windows of frames) is not justified through compelling experiments. When using window (local) attention, the performance decreases (Table 6) for a small increase in throughput. Using higher resolution inputs with window attention could have revealed some interesting interplay between mixing of information through self-attention (larger windows) vs mixing directly over space and time through Mamba (smaller windows). With the current experiments, Divide only seems to complicate the story without adding convincing benefits. \n\nFor the SeqMod part, an ablation of the Mamba state size would be needed to see if you can get better performance by increasing the capacity of Mamba together or instead of using the proposed modulation.", "methods_and_evaluation_criteria": "The evaluation focuses on video classification alone; it should be improved by adding a VLM task, given that CLIP has strong semantic features.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments on long-term action recognition (COIN, Breakfast) are confusing. The authors perform end-to-end finetuning (Table 3), but line 368 mentions PEFT? There are no implementation details about the long-term fine-tuning experiments.\nWhat batch size is used in experiments? 12h to converge on K400 seems quite long, given that only the Mamba parameters need to be learnt. \n\nThe comparisons in Tables 1,2 could be misleading as they don’t mention pre-training data. It is not fair to directly compare the proposed method which uses CLIP weights trained on a very large dataset to e.g. ViViT that uses only Imagenet pretrained weights. This is included in Table 3, but still a bit misleading: the CLIP pre-training dataset should be included under Dataset. \n\nSince the authors include fine-tuning results for long-term action recognition, the fine-tuning results for k400 and ssv2 should also be included.", "relation_to_broader_scientific_literature": "The discussion about SSM for vision should be more general and include hybrid SSM works prior to or concurrent to Mamba. ViS4mer (using S4 SSM) is included in experiments comparison, but it’s not discussed in related work. TranS4mer should be mentioned as well. They both rely on standard image encoders. More recently, TRecViT used pre-trained weights with linear recurrent units, which are very similar to Mamba layers.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The writing needs improvement. \n\nThere are multiple typos, some that hinder comprehension (included in the section below). The notations are not all well explained or used consistently. Image dimensions H,W seem to represent pixels at L154, but the model probably operates on embedded patches. When using local attention, 16x16 windows are expressed in terms of pixels or patches? \nL379 “Key frame number 16” – how are key frames extracted? There is a typo in this sentence as in L428 “frame number 16” ?\nIn Fig3, the notations are not clear to me, how do y1 and y2 relate to x,y in the figure?\n\nFirst part of 3.5 talks about Related work again, would be better placed in Section 2.", "other_comments_or_suggestions": "typos: \nL100: without interfering the pre-trained IMF → with the \nL55: significantly improvements → significant improvements?\nL70: “While we aim to capture …” sentence needs rephrasing \nL84: “be well generalized” – needs rephrasing\nL154: Given a image → an image\nL173: “While we aim to have a more precise…” sentence needs rephrasing\nL319: rage → rate\nL325: we fine-tune … and employs → employ \nL428: we first introduces → introduce", "questions_for_authors": "1. What is the performance when fine-tuning on K400 and SSv2?\n2. Did the authors try to ablate the SSM state size and hidden size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uGlzKpwTCg", "reviewer": "Reviewer_XKgu", "summary": "This paper proposed a framework \"modulated Mamba\" to adapt image foundation models for video understanding tasks by PEFT. There are two stages within the frame work. The first stage is \"divide\", which runs CLIP feature extraction on the pacthes of each frame. The second stage is \"SeqMod\" which draws intuiation from AdaN and merge the output from SSM layer to integrate both spatial and temporal informtion. The proposed method achieved state-of-the-art results on both short and long video understanding benchmarks.\n\n\n## update after rebuttal\nThe authors addressed my concern on the ablation analysis and related works, I increased score to accept and suggest adding these analysis in the final version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">## 1. Whether CLIP is the best model\n\nNote that our MoMa is an adapter method and is thus backbone-agnostic. We choose CLIP as our backbone following previous methods DiST and AIM for fair comparison.\n\n| Method    | Accuracy |\n| --------- | -------- |\n| MoMa-CLIP | 83.7     |\n| MoMa-MAE  | 81.2     |\n\nHere we replace the CLIP backbone with ViT-MAE from facebook, considering that MAE-style pre-training may be more focused on image local feature. The performance drops since the MAE is pre-trained on ImageNet-1K, which is not as large as CLIP.\nBesides, discussing which image pre-trained model better suits video understanding is out of the scope of this paper.\n\n>## 2. Why different merging method matters so much. Explain Tab. 5 more.\n\nWe’ve provided a new ablation study on our SeqMod operation. For more details, please refer to our response to ``R1(5nFH) #2: Further analysis on SeqMod operation``. We hope this clarifies and provides a more comprehensive understanding of SeqMod.\n\n>## 3. Temporal grounding & LVBench\n\nThanks for the suggestion!\n\nWe've add a new experiment by equipping MoMa with MLLMs and answer three complicated video understanding tasks from MVBench: \n- `action_antonym`: distinguish the correct action from two inversely ordered actions. \n- `action_localization`: determine the time period when a certain action occurs.\n- `action_sequence`: retrieve the events occurring before or after a specific action.\n\nWe achieve comparable performance on these tasks compared with LLaVA and Video LLaMA while operating with merely 5% of their training data and 21% of their parameter budget (1.5B vs. 7B parameters), demonstrating our efficiency while showing our scaling potential across a broader range of wider application. Please see our response to ``R4(17be) #2.1 Equipping MoMa into MLLMs`` for implementation detail. \n\n\n>## 4. Related works\n\nWe here supplement a detailed literature review.\n\nThe concept of learning scale and bias for feature adaptation was first introduced in FiLM [1] (**F**eature-w**i**se **L**ayer **M**odulation) in 2017. At the same time, [2] implemented this adaptation before the network normalization layer, which led to the term ‘adaptive instance normalization’ (AdaIN). A 2018 survey [3] summarized these concepts as ‘feature-wise transformations,’ which have since been widely applied across a variety of tasks such as image recognition and generative modeling. Thus, AdaN refers to a family of methods that share this underlying principle, including, but not limited to, FiLM, AdaIN, and DiT.\n\nWe will clarify the definition of AdaN in the final version for better understanding. Thank you for the suggestion, and please let us know if there are any additional references or concepts we should include.\n\n[1] FiLM: Visual Reasoning with a General Conditioning Layer\n\n[2] Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization\n\n[3] https://distill.pub/2018/feature-wise-transformations", "claims_and_evidence": "yes. the rationale is clear for the proposed method.", "methods_and_evaluation_criteria": "I'm not sure if CLIP is the best model to use for extract the patch features, becasue, as the author also mentioned in the paper, CLIP is trained for full images. Tab. 6 shows how you extact patch features matters a lot, the authors might try different ways for get patch wise features, e.g. the pretrained tokenizer in ViT or even traditional methods that focus more on local features.\n\nThe methods and eval metrics are valid.", "theoretical_claims": "yes", "experimental_designs_or_analyses": "The experimental design is comprehensive. Though I would like to see more analysis on the ablation studies. For example, why does different merging method matters so much in Tab. 5? \"concat\" and \"SeqMod\" have the same input without extra parameters but there are alost 10% difference. I also have a question below.\n\nDoes the method applies to differen video understanding tasks like temporal grounding? For long video understanding, does it applies to longer videos like LVBench?", "relation_to_broader_scientific_literature": "The paper dives deep into the topic of using PEFT to integrate temporal information into a pretrained image foundatio model, and cited related works related to video understanding, image foundation models, PEFT and SSM.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "As I said in other sections, the method is novel and the results are good, but more analysis are needed.", "other_comments_or_suggestions": "no", "questions_for_authors": "I don't thin adaptive normalization (AdaN)  is proposed in Perez et al., 2017 (L218), wrong citation or you refers to FiLM? I think you may be referring to a group of method following similar formula, the authors should make it clear in the text and add to related works, since it's the core idea behind he scene.\n\nIn Tab.5, it’s counterintuitive that \"add\" and \"max\" perform worse than \"skip\", since the trained SSM layer should learn to minimize y and achieve the same results with skip?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "no"}}, {"review_id": "EWfgxMxAIC", "reviewer": "Reviewer_17be", "summary": "The method in the paper attempts to adapt image foundation models for video understanding tasks. The authors introduce MoMa, an efficient adapter framework that integrates Mamba’s selective state space modeling into image foundation models. They propose a novel SeqMod operation designed to inject spatial-temporal information into pre-trained image foundation models without disrupting their original features. MoMa is incorporated into a Divide-and-Modulate architecture, enhancing video understanding while maintaining computational efficiency. Extensive experiments on multiple video benchmarks demonstrate that MoMa achieves superior performance compared to existing models, with reduced computational costs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">## 1. Ablation studies focusing on the layer design\n\nThanks for your advice!\n\nAs discussed in Section 3.5, unlike Jamba, which focuses on fine-grained architectural design, our architecture builds upon CLIP and cannot undergo drastic changes. Instead, we focus on how to maximize the advantages of Mamba without disrupting the original pre-trained weights.\n\nBelow, we supplement an ablation on architectural patterns.\n\n- ``[TM]12``: Alternating sequence of Transformer and SSM layers, each repeated 12 times (our current design).\n- ``[T]12[M]12``: 12 consecutive Transformer layers followed by 12 SSM layers, with the SSMs functioning as the decoder rather than being inserted as an adapter in the middle of the encoder.\n- ``[T]6[TM]6``: No modulation for the first half of the Transformer layers.\n- ``[TTM]6``: One modulation layer is inserted between every two transformer layers.\n\n| Pattern           | Top-1 | Top-5 |\n| ----------------- | ----- | ----- |\n| ``[TM]12`` (Ours) | 83.7  | 96.5  |\n| ``[T]12[M]12``    | 80.6  | 92.8  |\n| ``[T]6[TMM]6``    | 81.5  | 94.0  |\n| ``[TTMM]6``       | 82.8  | 95.5  |\n\nFrom the table we find that:\n\n- Using MoMa directly as the decoder performs the worst. Under this pattern, the encoder is entirely non-trainable, preventing it from adapting to understand video inputs. As a result, all temporal information must be learned within the lightweight SSM layer, which makes it difficult to capture temporal dependencies effectively.\n- Adapting only the latter half of the backbone is sub-optimal, Since learn to recognizing temporal information in the early stages of video encoding is essential for effective processing.\n- The ``[TTMM]6`` pattern performs worse than ``[TM]12``, suggesting that a more balanced integration of both components in the adapter structure is more beneficial.\n\nWe will add this table into our final version.\n\n>## 2. The benchmark might be outdated\n\nHere we supplement a new experiment by equipping MoMa with MLLMs and answer complicated video understanding tasks.\n\n### 2.1 Equipping MoMa into MLLMs\n\n**Experiment setting**\n\nWe first train MoMa with CLIP-L backbone on Kinetics-700 for video understanding. Then we apply a VideoLLaMA2-style VLM architecture and train the only projection part on video instruction tuning dataset. We use Qwen2.5-1.5B LLM decoder and ``LLaVA-Video-178k`` for training. Finally, we evaluate on a subset (``[action_antonym, action_localization,action_sequence]``) of MVBench.\n\n- `action_antonym`: distinguish the correct action from two inversely ordered actions. \n- `action_localization`: determine the time period when a certain action occurs.\n- `action_sequence`: retrieve the events occurring before or after a specific action.\n\n\nNotably, our training configuration involves significantly fewer resources compared to contemporary VLMs. We use only 0.5M video data for pre-training and 178k video instruction tuning data for fine-tuning, only **5%** of the 12M+ video-text data used in VideoLLaMA. This resource-constrained setup inevitably limits the model's generalizability across comprehensive video understanding tasks. Therefore, we choose some action-related subtasks from MVBench since these are most similar with our training resource.\n\n| Model         | LLM          | AA   | AL   | AS   |\n| ------------- | ------------ | ---- | ---- | ---- |\n| LLaMA-Adapter | LLaMA-7B     | 51.0 | 21.5 | 23.0 |\n| LLaVA         | Vicuna-7B    | 63.0 | 20.5 | 28.0 |\n| VideoLLaMA    | Vicuna-7B    | 51.0 | 22.5 | 27.5 |\n| Ours          | Qwen2.5-1.5B | 52.1 | 22.7 | 23.4 |\n| GPT-4V        | GPT-4        | 72.0 | 40.5 | 55.5 |\n\nWe achieve comparable performance with LLaVA and Video LLaMA while operating with merely **5%** of their training data and **21%** of their parameter budget (1.5B vs. 7B parameters), demonstrating our efficiency while showing our scaling potential across a broader range of wider application.\n\n\n>## 3. Include image size in Table 1 & 2\n\nThanks for the suggestion. For a fair comparison, all input resolution in our Table 1 and Table 2 are 224, unless otherwise specified. (For example, the MViTv2-L (312$\\uparrow$) in the sixth row (L284) of Table 1 indicates an input size of 312, which is larger than the default size.) We will clarify this in the final version.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence from the experimental results. The authors demonstrate that the MoMa framework, which integrates Mamba’s selective state space modeling into image foundation models.", "methods_and_evaluation_criteria": "Yes, the proposed efficient method has the impact on application side.", "theoretical_claims": "No, there is quite a little theoretical part in this paper. While the paper introduces a novel method for improving video recognition, it lacks a thorough theoretical explanation of the underlying mechanisms and the potential limitations of the approach.", "experimental_designs_or_analyses": "Yes, the experimental settings are clear and well-explained. However, the benchmark might be somewhat outdated or overly saturated for the current video understanding research.", "relation_to_broader_scientific_literature": "The paper is an extension from the mamba and other SSM model used in NLP tasks.\n\n\nThe paper builds upon existing work in the domain of selective state space modeling (SSM), particularly in the context of natural language processing (NLP) tasks. It extends the Mamba architecture and similar SSM models to address video understanding challenges. While the approach is an extension, it would be helpful for the authors to more explicitly position this work within the broader literature and discuss how it advances beyond prior methods, particularly in video processing and multimodal tasks.", "essential_references_not_discussed": "No, the paper makes a good review of the literature. It discusses relevant background and builds upon existing work in the domain of state space models (SSM) and their application to video understanding.", "other_strengths_and_weaknesses": "Strength\n1. This paper effectively adapts an image model to handle video tasks, a crucial step for video understanding research, especially since image models are often trained on much larger datasets.\n2. The use of Mamba (or other SSM models) is promising, as it has demonstrated strong performance across various NLP and computer vision tasks. It is exciting to see this approach applied in the video understanding domain as well.\n\nWeaknesses: See Qusetion section", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. The divide and modulate stages are actually the combination of window-attn and SSM hybird architecture.  Are there any ablation studies focusing on the layer design? (e.g., adding the SSM layer only in certain parts of the architecture like in Jamba)?\n2. Video understanding as a field has evolved rapidly. The benchmarks used in the paper seem to be outdated, particularly with datasets like K400 and SSv2, which are now saturated (with performance rates over 95%). Would it be possible to include more recent datasets such as ActivityNet, or VideoMME for a more challenging comparison?\n3. For a fair comparison, it would be helpful if the authors could also include the input image size used in Table 1 and Table 2 like the table in VideoMamba, as it may influence the performance result.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes, MoMa, to adapt an Image Foundational Model (such as CLIP, ViT-MAE) for video classification & understanding by integrating Mamba’s selective State Space Modeling into image foundational models. Experiments are done on video classification standard benchmarks for both short and long video classification. During rebuttal, the authors also further provide additional experiments on video QA with MLLMs. After rebuttal, all reviewers recommend to accept the paper. AC reads all reviews, rebuttal and discussions between authors and reviewers. AC agrees with the recommendations from the reviewers, thus recommends to accept this paper.

Note to authors: please incorporate the additional experiments and ablations into the final camera-ready version of the paper if it is accepted.
Decision: Accept (poster)