State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding
Authors: Jiahuan Zhou
Kai Zhu
Zhenyu Cui
Zichen Liu
Xu Zou
Gang Hua
Venue: NeurIPS 2025 poster ()
Keywords: Computer Vision and Pattern Recognition
Primary area: deep_learning

Abstract: Recently, pre-trained state space models have shown great potential for video classification, which sequentially compresses visual tokens in videos with linear complexity, thereby improving the processing efficiency of video data while maintaining high performance. To apply powerful pre-trained models to downstream tasks, prompt learning is proposed to achieve efficient downstream task adaptation with only a small number of fine-tuned parameters. However, the sequentially compressed visual prompt tokens fail to capture the spatial and temporal contextual information in the video, thus limiting the effective propagation of spatial information within a video frame and temporal information between frames in the state compression model and the extraction of discriminative information. To tackle the above issue, we proposed a State Space Prompting (SSP) method for video understanding, which combines intra-frame and inter-frame prompts to aggregate and propagate key spatiotemporal information in the video. Specifically, an Intra-Frame Gathering (IFG) module is designed to aggregate spatial key information within each frame. Besides, an Inter-Frame Spreading (IFS) module is designed to spread discriminative spatio-temporal information across different frames. By adaptively balancing and compressing key spatio-temporal information within and between frames, our SSP effectively propagates discriminative information in videos in a complementary manner. Extensive experiments on four video benchmark datasets verify that our SSP significantly outperforms existing SOTA methods by 2.76\% on average while reducing the overhead of fine-tuning parameters.

State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding
Jiahuan Zhou', Kai Zhu', Zhenyu Cui', Zichen Liu', Xu Gang Hua
Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China
Huazhong University of Science and Technology. Wuhan 430074,China
3 AAmazon.com Inc, Bellevue, WA 98004, USA
jiahuanzhou@pku. edu. en, zhukai2022@ruc eeterruccceu..... cn
(euizhenyu,11c22088887
pku, edu. en, zxλzoux. me, ganghua@gmail.com com
Abstract
Recently, pre-trained state space models have shown great potential for video classification, which sequentially compresses visual tokens in videos with linear complexity, thereby improving the processing efficiency of video data while main- taining high performance. To apply powerful pre-trained models to downstream tasks, prompt learning is proposed to achieve efficient downstream task adaptation with only small number of fine-tuned parameters. However, the sequentially compressed visual prompt tokens fail to capture the spatial and temporal contextual information in the video, thus limiting the effective propagation of spatial informa- tion within a video frame and temporal information between frames in the state compression model and the extraction of discrininative information. To tackle the above issue, proposed State Space Prompting (SSP) method for video under- standing, which combines intra-frame and inter-frant frame prompts to aggregate and propagate key spatiotemporal information in the video. Specifically, Gathering (IFG) module is designed to aggregate spatial key information within each frame. Besides, AiterFrrame Spreading (IFS) module is designed spread discriminative spatio-temporal information across different frames. By adaptively balancing and compressing key spatio-temporal information within and between frames, our SSP effectively propagates discriminative information in videos in a complementary manner. Extensive experiments four video benchmark datasets verify that our SSP significantly outperforms cxisting SOTA methods by 2.76% on average while reducing the overhead of fine-tunine parameters. The code is available at http://iihub..com//oojjjjaa 199 htt.:/piiaaaaan191///////2/000 INeuu1PP22225-SSSP
Intra-Frame
Introduction
In recent years, the Vision Transformer (ViT) has demonstrated its promising performance in video understanding due to its powerful attention-basee context modelling capabilities [ 2 5 g| [0]. However, the computational cost of the attention mechanism, which increases quadratically with the length of the input data, incurs huge computational and memory costs, especially when processing long video sequences. To achieve efficient video processing, a state space modelling method, called VideoMamba, is introduced achieve comparable performance to the VIT while maintaining linear computational complexity [2]. Despite some progress in model pre- pre-training [3] [3] it still suffers from the heavy overhead of downstream task adaptation through parameter fine- tuning. Therefore, Parameter-Efficien Fine-Tuning (PEFT) has aroused extensive attention to achieve
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

comparable even higher performance than Full Fine- Tininnnnng (FFT) and reduce the adaptation costs by optimizing only few set of parameters [T6]
18.
Early PEFT methods mainly focused on parameter-level efficient fine-tuning, e-g. Adapter 19
21 22
LoRA [23 24], but still suffered from inefficiency with the additional introduction of model parameters. To this end, visual prompt learning technology aims to embed a small set of learnable prompting tokens the imput level, enabling efficient downstream task adaptation without extending internal model parameters 22
26
27
28
29
30
31
32
33 [34] However, as shown
in Figure [ existing video prompting methods are typically ViT which exploit global attention mechanisms to propagate key information in intra- and inter-frame prompts [35] 36 37
[8). ignoring the requirement to balance efficiency and effectiveness in video understanding for VideoMamba. Specifically, ignoring the sequential compression state space [39 40
41
42 43
existing video prompting methods fail to gather spatial information in long video token sequences for Mamba. In addition, the global in VideoMamba, leading to the dissemination of key spatial and temporal information the long video token sequence. Consequently, as shown in Figure [b), the embedded prompts mitigate the aggregation of discriminative spatio-temporal information due to the information decay after long-term compression
prompting method mitigates the high efficiency
Information Decay
Intra-Frame Prompt
Inter- Inter-Framm Frame Aimoon
Mamba Layer
Learnable
Prompt Token
/-2
[-1
f+1
#+2
(b) Visualization of Existing Video Prompting
Information Enhancement
Video Token
Input Video
Frame
(a) Existing Video Prompting on Mamba
(e) Visualization Our SSP
Figure Existing video prompting methods on Mamba and its visualization results of updating gate compared to our SSP. Existing methods directly concatenate learnable prompts to video tokens, resulting in the information decay problem after the long-term state space compression. However, our SSP achieves information enhancement through spatial gathering and temporal spreading.
To address the above challenges, we propose a State Space Prompting (SSP) framework for video understanding, which gathers and spreads apatio-temporal information in an efficient and effective manner, as shown in Figure [c). Specifically, an Intra aura-Framee Gathering (IFG) module is designed to exploit low-rank local convolution to aggregate spatial information within each video frame. Sequentially, a low-rank iieettio--oriente Inter-Frame Spreading (IFS) module is further proposed spread key information at the temporal level, which develops low-rank attention module to refine the temporal information that is gradually aggregated between frames, where long-term context information is effectively spread within global temporal prompts. Among them, the information entropy of each frame adjusts the attention given to each frame when refining the temporal information, and the frame-specific spatial variance is employed to gate the influence strength from the spreading information to each frame.
In summary, contributions three-fold: (1) We proposed State Space Prompting method for Video Understanding, called SSP, which gathers and spreads discriminative spatio-temporal information compressed by the state space model achieve high effectiveness while maintaining its computational efficiency. (2) We design an Intra-Frame Gathering module and an Inter-Frame Spreading module to facilitate
contextual information interaction by spreading gathered local spatial information a temporal manner. (3) Extensive experiments on multiple video understanding benchmarks demonstrate that method achieves superior performance against. existing methods with only ~3% of tunable parameters compared to full tuning.

---

2 Related Work
2.1 State Space Model (SSM)
In recent years, State Space Models (SSMs) have emerged as a promising approach for sequence modeling, offering the ability to capture long-range dependencies with linear computational complex- ity 39
41 42 43). Building this foundation, Mamba and Mamba2 introduced input-dependent update and forget gates to address limitations in architectures, this allows SSM parameters be dynamically modulated by input, significantly im- proving expressiveness in discrete modalities. Coupled with hardware- friendll parallelization, these advances lead notable gains in computational efficiency and performance in language modeling.
reasoning [45] 46]. Unlike Transformer
Building on this progress, Vision Mamba extended the Mamba framework to 2D image modeling via bidirectional spatial scanning, achieving strong results in image understanding tasks [47 48). More recently, VideoMamba further generalized this approach to video by introducing apatio-tempore scanning, enabling efficient modeling of global dependencies across both spatial and temporal dimensions with linear complexity [
This architecture rivals or surpasses traditional CNN 52 establishing itself competitive backbone for video understanding. However, applying pre-trained models like Video.Mamba to downstream tasks using full fine-tuning typically requires large data and high computational cost. Thus, developing parameter- efficient fine-tuning strategies for VideoMamba remains an important and urgent challenge.
49
and Transformer models [12 50
51
2.2 Parameter-Efficient Fine-Tuning for SSM
Parameter-efficient fine-tuning techniques aim to reduce learnable parameters while maintaining model performance, thereby reducing storage and computational costs when adapting pre-trained models to downstream tasks [16]. Several studies have attempted to apply parameter-efficient fine- tuning methods to Mamba architecture models [53] 54 [55]. These methods can be categorized into partial-based, addition-based, and prompt-based approaches.
Partial-based methods typically fine-tune only subset of parameters in the pre-trained Mamba model, such as projection layers, convolution layers, or forget gates [54] These methods are straightforward and easy to implement. However, partial-based approaches are constrained by the model's 's inherent parameter space, limiting their adaptability to downstream tasks across different domains. Addition-based methods generally freeze the original model parameters and incorporate learnable components, such as adapter modules [9 21 While these plug-and-play modules can be readily transferred to the Mamba model architecture, they simply apply transformations to the input data without considering the sequential progression characteristics inherent to Mamba's 's architecture. The Additional-Seaa approach attempts to learn downstream knowledge by increasing the state dimensions of the SSM (55]. However, when applied to video domains, merely increasing state dimensions proves insufficient for effectively extracting critical spatio-temporal information. Prompt- based methods add a small number of learnable prompt vectors, optimizing only these parameters during training. Existing work such as SVP has transferred prompt learning methods to Mamba models by generating prompts for each token, effectively activating Mamba's update and forget gates during fine-tuning [53]. However, such methods are designed exclusively for static 2D images. When transferred video tasks, they similarly struggle with the challenge of modeling long-context spatio-temporal relationships.
2.3
Video Prompting
Prompt learning methods were first introduced in natural language processing (NLP) to transfer
pre-trained models to various downstream tasks (56] 57 58 59). Inspired by the success of prompt
learning in NLP, these methods have been extended to the visual domain 600 61] VPT and VFPT fine-
tune models by concatenating noken-based prompts with input data to capture image features 25 26).
More recently, methods like DGL and STOP have applied prompt learning to video tasks, using intra-frame and inter-frame prompt modules to capture temporal information 35 36
38 For
instance, DGL employs prompt vectors as query, key and value vectors to model both local features and global features in videos [36]. MPT utilizes prompts as query vectors, leveraging the Q-Former mechanism to extract spatial, temporal, and global features [37] STOP generates inter-framc prompts
3

---

ce
Intra-Frame
Pronep
Fasch trebeddin
Spatial
Cooniication
Eimaay
Imrganal Proeage
Sumpled Tokent
Inter-Erame
Concolational Laye:
Pauegred Tollo
Spreatling (IF5)
Figure 2: The pipeline of our SSP. embed videos into image tokens spatial prompts. After the initial Mamba layer, our complementary IFG IFS modules operate IFG aggregates spatial information while IFS spreads temporal information. Information entropy and frame-specific spatial variance bridge these modules. The prompted tokens and CLS token then pass through subsequent Mamba layers for vidco classification.
from all tokens while using intra-frame prompts to highlight the varying importance of frames, dynamically inserting prompts between frames [38].
However, these methods were originally designed for the Transformer architecture, leveraging its unique structure by connecting prompts to query and key-value vectors to store task information. Transformer-bast video prompting methods use a global attention mechanism [62]. allowing tokens within the sequence interact at any position.
contrast, the Mamba architecture propagates causing adjacent tokens to contain more overlapping information. As result, when applying existing video prompting methods Mamba, feeding all tokens into the inter-frame prompt module introduces redundant contextual information. This issue is more pronounced in video data with high spatiotemporal redundancy, making difficult for the prompt module to effectively capture and propagate discriminative spatiotemporal context, ultimately limiting model's performance.
tokens sequentially (63 (4 65
Methodology
In this section, we illustrate the proposed SSP comprehensively, and the overall pipeline depicted in Figure
3.1 Preliminary of Mamba
The SSM-based models, Mamba, Vision Mamba (ViM), and VideoMamba, are inspired by continuous systems that map one-dimensional equations sequences æ<() E IR y(t) R through D-
dimensional hidden state (tt) E DR These hidden states evolve over time via parameter matrices A, B. and C, following linear ordinary differential equation:
H'(t) = Ah(t) + Br(t), s(t) CC((t)
(I)
where parameter A E IR RDD represents the forgetting gate matrix, B E RDx1 denotes the update gate matrix, and E R serves the output projection matrix.
To facilitate application in deep learning, SSMs are discretized into discrete-time systems using the zero-order hold technique. The continuous parameters A and B transformed into their discrete counterparts A DDDD and B R Dx1, employing sampling time interval A E E:
= exp(AA), B = (AA) -1 (exp(AA) - 7) AB.
(2)
Consequently, the discretized SSM be expressed follows:
h; = Ah,-1 + Bz, Vi = Cha
(3)
where h4 IR Dxd and F4, W I is the dimension of the input sequences.

---

3.2 State Space Prompting
The backbone of our method is VideoMamba 49, the input a video V RT*OxHXW where T represents the number of frames, C represents the number of channels, and H x W is spatial size. HxW W fixed-size patches of size h and these patches hxw
Each video frame {F, {Fr) split into
are flattened into sequence of vectors 1; = i
where denotes the frame index E R [ Jj=1
while j denotes the patch index. These vectors are then projected into input tokens æ; ( where Zij E IR4, and dl the hidden dimension of the input sequence. For video classification tasks, the class token Bete is prepended to the sequence of input tokens, which is [Ttsss, 11 22,...... ær). Then the input tokens are fed into the VideoMamba backbone, which consists of I. layers Mamba block. The class token Eets from the last layer is used for classification tasks. The final output of the VideoMamba backbone is obtained by applying linear classifier head on Vele-
Our SSP consists of two complementary modules: an intra-frame gathering module (IFG) and an inter-frame spreading module (IFS). These modules interact complementarily to aggregate spatial information and spread discriminative long-term context information at the temporal level during fine-tuning.
3.2.1 Intra-Frame Gathering Module
The IFG p processes each frame F; to generate intra-frame prompts p R information entropy weights w, E R xd and spatial variance measurements E R 1x The intra-frame prompts then overlaid to the imput tokens produce spatial prompted tokens æf E Rdx
P wi, (*.),, x x; P
The IFG takes the tokens from each frame a, = { input. These tokens are pro- cessed through a downsampling layer Ldown followed by a 2D convolutional layer No to generate low-rank feature maps L E Nxd for each frame, where d' represents the internal dimension of the intra-frame gathering module.
Videe
ST-Adapter
The low-rank feature maps L; are subsequently upsampled via linear layer LTP to match the dimensionality the input tokens, resulting in intra-frame prompts p.. Concurrently, the low- rank feature maps undergo additional 2D con- volution and upsampling operations to produce prompts
spatial variance v, The o then fed into entropy calculation mod- ule & to calculate information entropy weights w;:
Figure 3: Visualization of our intra-framm prompts, which capture discriminative local features.
L, Ni (C (zx)),
p (L,) w; Elp)).
0. -2 (L)),
The entropy calculation module & evaluates the informational significance of intra-frame prompts and generates frame-level weights. This module first transforms the intra-frame prompts pt into probability distributions, then calculates the information entropy, which subsequently scaled by learnable factor A small positive constant is incorporated to prevent taking the logarithm of zero:
H,
P; = softmax(p?). H. -- -
+ +e,, E, 1.0 max H.)
(6)
w; = a (F)t where the mean of E, across tokens per frame.
The intra-frame prompts p gather the model's attention to local features during downstream fine- tuning, as shown in Figure Information entropy weights adjust the attention given to each frame during inter-frame prompt generation based on the certainty information distribution, while spatial variance gates the influence strength from the long-term context information local features.
5

---

3.2.2 Inter-Frame Spreading Module
After processing through the first Mamba layer, the inter-frame spreading module (IFS) activates. This module first samples the last token 8; E R from frame F; the Mamba forward scanning sequence. This token is Hadamard multiplied with the information entropy weights generated by intra-frame gathering module and fed into the inter-frame spreading module pt, The module's output is then Hadamard multiplied with the spatial variance v; to produce inter-frame prompts p Rixd,
8; N:N; P P'(s: ④ w:) ④ U/-
The IFS processes the input through sequence of operations including downsampling linear transformation Liown, an attention computation A, and an upsampling linear transformation L333 The resulting output is then computed Hadamard product with the spatial variance v;, and subsequently scaled by learnable factor to produce the inter-frame prompts p This architecture facilitates the spreading of long-term context information across video frames while preserving local context information:
p - 3 C3 (A(L2 (8:))) Uz-
The last token of each frame aggregates contextual information from both the current frame and preceding frames during forward scanning, as well as subsequent frames during backward scanning. As shown in Figure the generated inter-frame prompts p represent temporal inductive biases that spread the gathered temporal information from all frames to the current frame.
Ko Imper Inaniiees lendl
1.0
Video
STOP
SSP
Figure 4: Visualization of our frtmr- prompts. The action "catch" can be parsed into two distinct phases: preparation for the catch and the actual reception the ball. Our inter-frame prompts effectively locate the key transitional moments between these critical phases.
Subsequently, the inter-frame prompts p are concatenated with the spatial prompted tokens a; Mamba layers (B, / extract
along with the class token Tels, are fed into the following spatio-temporal features. Inspired by VP'T-deep [25], we overlay the intra-frame prompt on the input of each layer and embed the fitar-frame prompt the input of all layers after the first layer. To obtain
the final predicted probability distribution y. apply a linear classification head H. on Weis-
3.2.3 Overall Optimization
As mentioned above, our SSP introduces only a few additional parameters:
M {P* P'}.
Following prior works, we keep the pre-trained model frozen during training, allowing only the classification head H and the newly added modules M to be trainable. The optimization objective is defined as follows:
arg min Lee Cye Mgt): M.H
(10)
where Lee the cross-entropy loss, and Mgt the ground truth video label.
4 Experiments
4.1 Datasets
HMDB51 contains 6849 clips across action categories, with an average duration 3.15 seconds and 91.49 frames per video. It was collected from various sources, mostly from movies, and a small proportion from public databases such as the Prelinger archive, YouTube and Google videos, featuring diverse real-world actions with variations in background and camera angles.

---

Table 1; The comparison results K400 pretrained VideoMamba-S (Parameters 25, 25.42M).
Method
Venue
Param
HMDB51
SSV2
UCFI01
Breakfast
Full 40
ECCV'4
25.42M
67.58
58.57
92.96
94.27
Adapter
19
NeurlPS/22
2,40M
73.79
36,45
94.18
84.89
VideoMamba-S
ST-Adapter
21
NeurlPS'22
2.69M
70.52
30.94
94.87
77.60
VPT
ECCV'22
1,50M
72.74
30.68
95.16
81.25
VFPT 26
NewrlPS'24
1.50M
72.41
30.37
95.08
79.68
SVP S
AAA125
2.76M
69.93
38.01
95.58
80.72
Additional-Sean
55
ICLR'25
0.66M
73.20
33.71
95.63
78.64
STOP 38
CVPR'25
1,49M
70.06
21.22
93.44
65.62
SSP(Ours)
This Puper
0.98M
74.38
38.68
95.69
85.41
Table 2: The comparison results on CLIP-400M pretrained and K400 pretrained VideoMMmmb-M (Parameters 74.00M).
[66]
(Parameters 88.00M)
Method
Venue
Param
HMDB51
SSV2
UCFI01
Breakfast
CLIP
DGL-Linear
36
AAAI'44
0.83M
67.20
18.30
92.50
Tran-Transormmrr
36
AAAT'24
9.57M
69.80
18.10
93.60
STOP
38
CVPR'25
7.53M
72.00
21.40
95,30
Full 49
ECCV24
74.00M
76.30
67.30
96.00
95.31
Adapter
19
NeurlPS'22
2.40M
73.59
46.98
96.24
89.06
VideoManba-M
ST-Adapter 21
NeurlPSS22
2.69M
71.96
31.91
94.81
67.70
VPT 25
ECCV'2
1.50M
73.59
40.58
95.45
81.77
VFPT 20
NeurlPS'24
1.50M
71.89
39.68
95.71
86.97
SVP
AAAI22
2.76M
73.66
49.08
96.77
90.10
Additional -can
55
ICLR'25
1.33M
73.52
44.65
96.32
86.97
STOP 38
CVPR'25
1.49M
71.96
23.44
94.60
71.35
SSP(Ours)
This Paper
2.41M
76.66
53.72
97.03
93.23
UCF101 [68) is action recognition data set of realistic action videos, collected from YouTube, having 13320 video clips across 101 action categories. The average video length is 7.21 seconds with 186.5 frames per video.
Something-Somethingg V2 (SSV2) is a large-scale dataset for action recognition, containing 220,847 videos across 74 action categories. The dataset is designed to capture fine-grained actions and interactions between objects, with an average video length of 3.82 seconds and 45.84 frames.
Breakfast 77 is a long video understanding dataset containing 1989 video clips divided into 10 categories related breakfast preparation. The dataset has an average video length of 137.53 seconds, with an average of 2062.89 frames per video.
4.2 Comparison Methods
We compare our SSP with both adapter- based and prompt-based parameter-efficient finetuning meth- ods. We also report the fully tuning results baseline, i.e., VideoMamba [49]. For adapter- based methods, we compare with the following methods: Adapter [19] and ST-Adapter (2]. For prompt-
based methods, we compare with the following methods: DGL--Linear 36 DGI.-Transfon Transformrr [36].
VPT 25
VFPT 26 SVP
and STOP 38)
4.3 Implementation Details
Following 49
all video frames are resized to 224 x 224 and split into 14 14 patches. For HMDB51, UCF101 and SSV2 datasets, each video is uniformly sampled to frames, while for Breakfast, we sample 32 frames. We set the learning rates to 3e-3, 5e-3, 2c-4, and 1e-3 for HMDB51, UCF101,

---

SSV2, and Breakfast respectively. Meanwhile, we set the batch size to 32 for HMDB51 and UCFI01, 64 for Breakfast and 512 for SSV2. All the dataset splits are consistent with the official annotation files. The model is fine-tuned with the AdamW optimizer on 4 NVIDIA 4090-24G GPUs, with cosine decay scheduler. Additionally, we adopt a warm-up strategy within the first 5 training epochs.
4.4 Comparison with State-of-the-arts
We evaluated our SSP method on four popular video datasets: HMDB51, UCF101, SSV2, and Breakfast. For fair comparison, we used CLIP-400M pretrained CLIP-ViT-B/32 as the backbone for ViT-oriented methods, while for methods fine-tuned on the Mamba architecture, we employed Kinetics-400 [71] pretrained VideoMamba-M and VideoMamba-S backbones. Our method achieves state-of-the-art performance across datasets, as shown Table When fine-tuned on VideoMamba-M, our ap- proach attains top-1 accuracies of 76. 76.66%, 53. 33.72%, 97.03%, and 93.23% on HMDB51, Something Sorenthing V2, UCF101, and Breakfast respectively, representing improve- ments of 3.00 % 4.64%. %, 0.26%, and 3.13 %
Accuracy Breakfast (%)
90-
93.1%)
Addi-Soul
SVP (2.76, 98.15))
(IJJMM KE.0%))
85-
Adapter (2-4MM, 99.1%)
existing parameter-efficient video fine- tuning methods. As demonstrated in Table [] even with the parameter constraints of smaller- scale models, our method still delivers supe- rior results when fine-tuned on VimeoMamba-S. This effectiveness stems from our tailored de- sign for the Video architecture, which enables efficient gathering and spreading of dis- criminative ppatio-temporal information within state space models, through the complemen- tary IFG and IFS modules. Notably, our ap- proach demonstrates particularly significant improvements on the challenging large-scale Komething-Somettie V2 dataset and the long- video Breakfast dataset, as shown in Figure 5 method exceeds others when tuning only 1.30M parameters (using only one IFS module as shown in Figure 7). This is because our SSP can gather local information on key regions in complex video data and spread critical global information in long contexts.
VPT 81.5%)
75-
70-
ST. Adggdde (7.7%)
STOP (1,RM,
Parameters (M)
Figure 5: Comparison on Breakfast. SSP outper- forms existing methods while tuning minimal pa- rameters.
4.5 Ablation
4.5.1 Influence of Different Components
To verify the effectiveness of the intra gathering (IFG) module and inter-frame spreading (IFS) module, conducted ablation experiments three datasets: HMDB51, UCF101, and Breakfast, as shown in Table 3 As demonstrated, when neither component is used, SSP achieves the lowest accu- racy all datasets. When the intra-frame gathering module is used alone, the model's performance improves by 9.78% on average. This is because the intra-frame spatial prompts effectively capture local features and enhance the model's ability to focus on discriminative information within each frame. When the -nter-frame spreading module is used alone, the model's performance improves by 9.20% on average. This can be attributed to that the inter-frame temporal prompts effectively aggregate and spread global contextual information across frames. When both components are used together, the model achieves the best performance by an average improvement of 13.46% across all datasets, as they complement each other in gathering and spreading both local and global information. When the spatial variance gate or the entropy gate (denoted as V, w. in Equation 7 is removed, the model's 's performance drops by 2.33% and 2.27% respectively on average across all datasets. This indicates that both gates play crucial role facilitating the inter-frame prompts to propagate spatial information in complementary manner based on the key local features of each frame.

---

Table 3: Ablation different prompting modules and gates.
Different Prompting Modules
Different Gates
IFG
HMDB51
UCFI01
Breakfast
Entropy
Spatial
HMDB51
UCFI01
Breakfast
59.34
90.64
76.56
74.70
96.56
89.58
74.44
96.56
84.89
75.09
96.29
88.54
v
72.61
96.14
85.41
V
74.96
96.61
88.54
76.66
97.03
93.23
V
76.66
97.03
93.23
4.5.2
The Visualization Results of Update Gate
Video
SVP
SSa (Ours)
Figure 6: Visualization the update gate. In SSP, the key region is activated effectively.
To further cxplore the impact of our intra-frame gathering module and inter-frame spreading module, visualize the normalized update gate values in VideoMamba on the last layer, which contribute to the final classification results di- rectly. As shown in Figure existing prompting methods designed for Mamba like SVP can't effectively gather and spread the discriminative spatio-temports information. This caused the pre-trained VideoMamba model focus on irrelevant regions in each frame. As result, the model struggles to accurately understand dynamic key features in the video. In contrast, our SSP method highlight the key regions with dynamic changes in the video, leading to a more accurate understanding of the video content.
HMDBSI
Breakfast
95.0
07.0
76.66%
93.23%
76.8
98.0
(90.10))
Accuracy Accuracy(9))
Accuracy
76.
SVP((3.68%)
STOP (71.96%)
72.0
(7135%)
7L.
Number
Number
Figure Ablation of the number of IFSs in our SSP.
4.5.3 Influence of Hyperparameters
The number of the inter-frame spreading modules (Pt) one of important hyperparameters in our method. To assess its impact, we conduct extensive ablation experiments. As shown in Figure the model's 's performance initially improves but then fluctuates as the number of inter-frame spreading modules increases.

---

To balance performance and tunable parameters, we set the number of inter-frame spreading modules when using VideoMamba-M. as backbone, and when using VideoMamba-S as backbone, we set the number to 1.
Conclusion
In this paper, propose novel State Space Prompting (SSP) approach for efficient adaptation of pre-trained state space models to video understanding tasks. SSP combines complementary intra-frame gathering module and inter-frame spreading module to aggregate key spatial information within frames and spread discriminative temporal information between frames, enabling effective propagation of crucial spatio-temporal features state space models. By employing the two modules, we can adaptively balance and compress discriminative information in videos. The effectiveness of our proposed SSP has been validated on four video benchmarks compared other parameter-efficien fine-tuning methods.
6
Acknowledgements
This work supported by the National Natural Science Foundation of China (62376011) and the National Key R&D Program China (2024YF11410000;
References
[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /929, 2020.
[2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836-6846, 2021.
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021.
[4] Jiahuan Zhou, Kunlun Xu, Fan Zhuo, Xu Zou, and Yuxin Peng. Distribution-awarc knowledge aligning and prototyping for non-exemplar lifelong person re-identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[5] Kunlun Xu. Fan Zhuo, Jiangmeng Li, Xu Zou, and Jiahuan Zhou. Self-reinforeing prototype evolution with dual-knowledge cooperation for semi-supervised lifelong person re-identification. ICCV, 2025.
[6] Kunlun Xu, Zichen Liu, Xu Zou, Yuxin Peng, and Jiahuan Zhou. Long short-term knowledge decomposition and consolidation for lifelong person re-identification. TPAMI, 2025.
[7] Kunlun Xu, Chenghao Jiang, Peixi Xiong, Yuxin Peng, and Jiahuan Zhou. Dask: Distribution rehearsing via adaptive style kernel learning for exemplar-free lifelong person re-identification. In AAAI, volume 39, pages 8915-8923, 2025.
[8] Kunlun Xu, Haozhuo Zhang, Yu Li, Yuxin Peng, and Jiahuan Zhou. Mitigate catastrophic remembering via continual knowledge purification for noisy lifelong person re-identification. In ACM MM, pages 5790-5799, 2024.
[9] Kunlun Xu, Xu Zou, and Jiahuan Zhou. Lstke: Long short-term knowledge consolidation for lifelong person re-identification. AAAI, volume 38, pages 16202-16210, 2024.
[10] Kunlun Xu, Xu Zou, Yuxin Peng, and Jiahuan Zhou. Distribution-awanl knowledge prototyping for non-exemplar lifelong person re-identification. In CVPR, pages 16604-16613, 2024.
10

---

[11] Jinyoung Park, Hee-Seon Kim, Kangwook Ko, Minbeom Kim, and Changick Kim. Video- mamba: Spatio-temporal selective state space model. European Conference on Computer Vision, pages 1-18. Springer, 2024.
[12] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3202-3211, 2022.
[13] Zhan Tong, Yibing Song, Jue Wang. and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078- 2022.
[14] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 14549-14560, 2023.
[15] Sucheng Ren, Xianhang Li, Haoqin Tu, Feng Wang, Fangxun Shu, Lei Zhang, Jieru Mei, Linjie Yang, Peng Wang. Heng Wang, Alan Yuille, and Cihang Xie. Autoregressive pretraining with mamba in vision, 2024.
[16] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 12799-12809 2023.
[17] Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al. Visual tuning. ACM Computing Surveys, 56(12):1-38, 2024.
[18] Yi Xin, Jianjiang Yang, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. -ffficien fine-tuning for pre- pre-trained vision models: A survey. arXiv preprint arXiv:2402.000 02242, 2024.
[19] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 35:16664-16678 2022.
[20] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-lame language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[21] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter- efficient image-to-vided transfer learning. Advances in Neural Information Processing Systems, 35:26462 26477, 2022.
[22] Jan-Martin O Steitz and Stefan Roth. Adapters strike back. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[23] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[24] Ahmed Agiza, Marina Neseem, and Sherief Reda. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16196-16205, 2024.
(25) Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 709-727. Springer, 2022.
[26] Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huangg, Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning. Advances in Neural Information Processing Systems, 37:5552-5558 2024.

---

[27] Yuzhu Wang, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Manni Duan, and Meng Wang. Revisiting the power of prompt for visual tuning. arXiv preprint arXiv:2402.02382, 2024.
[28] Han Cheng, Wang Qifan, Cui Yiming, Cao Zhiwen, Wang Wenguan, Qi Siyuan, and Liu Dongfang. E2vpt: An effective and efficient approach for visual prompt tuning. In International Conference on Computer Vision (ICCV), 2023.
[29] Hsi-Ai Tsao, Lei Hsiung, Pin-Yu Yu Chen, Sijia Liu, and Tsung-Yt Yi Ho. AutoVP: An Automated Visual Prompting Framework and Benchmark. In The Twelfth International Conference on Learning Representations, 2024.
[30] Zichen Liu, Yuxin Peng, and Jiahuan Zhou. Insvp: Efficient instance visual prompting from image itself. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6443-6452, 2024.
[31] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009))2337-2348, 2022.
[32] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for rision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816-16825, 2022.
[33] Kunlun Xu, Xu Zou, Gang Hua, and Jiahuan Zhou, Componential prompt-knowledge alignment for domain incremental learning. In ICML, 2025.
[34] Chenyu Zhang, Kunlun Xu, Zichen Liu, Yuxin Peng, and Jiahuan Zhou. Scap: Transductive test- time adaptation via supportive clique-based attribute prompting. In CVPR, pages 30032-30041, 2025.
[35] Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. Vop: Text-video co-operative prompt tuning for cross-modal retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6565-6574, 2023.
[36] Xiangpeng Yang, Linchao Zhu, Xiaohan Wang, and Yi Yang. Dgl: Dynamic global-local prompt uning for text-video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6540 6548, 2024.
[37] Haonan Zhang. Pengpeng Zeng, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Mpt: Multi- grained prompt tuning for text-video retrieval. In Proceedings the 32nd ACM International Conference Multimedia, pages 1206-1214 1214, 2024.
[38] Zichen Liu, Kunlun Xu, Bing Su, Xu Zou, Yuxin Peng, and Jiahuan Zhou. Stop: Integrated spatial-temporal dynamic prompting for video understanding. arXiv preprint arX11:2503.15973. 2025.
[39] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022.
[40] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances neural information processing systems, 34:572-585, 2021.
[41] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-22994, 2022.
[42] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1487, 2020.
[43] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971-35985 2022.
[44] Lamg Authors LongMamba. Longmamba: Enhancing mamba's 's long-context capabilities via training-free free receptive field enlargement. Openreview ICLR, 2025.
12

---

[45] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[46] Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML., 2024.
[47] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In Forty-first International Conference on Machine Learning, 2024.
[48] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances neural information processing systems, 37:103031-10306, 2024.
[49] Kunchang Li, Xinhao Li, Yi Wang. Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding. In European Conference on Computer Vision, pages 237-255. Springer, 2024.
[50] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211, 2019.
[51] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2020.
[52] Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint 2rXiv:2201.04670 2022.
[53] Yifeng Yao, Zichen Liu, Zhenyu Cui, Yuxin Peng. and Jiahuan Zhou. Selective visual prompting in vision mamba. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22083-22091, 2025.
[54] Seokil Ham, Hee-Seon Kim, Sangmin Woo, and Changick Kim. Parameter efficient mamba tuning via arXiv:2411.15224, 2024.
diagonal-centric linear transformation. arXiv preprint
[55] Masakazu Yoshimura, Teruaki Hayashi, and Yota Maeda. Mambapeft: Exploring parameter- efficient fine-tuning for mamba. arXiv preprint arXiv:2411.03855. 2024.
[56] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Lan- guage Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[57] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini arXiv:2101.00190 2021.
[58] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 10..0:21 2021.
[59] Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng. Lifu Huang, Xiaojun Quan, Zenglin Xu, al. Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 9147-9160, 2023.
[60] Qiwei Li, Kunlun Xu, Yuxin Peng, and Jiahuan Zhou. Exemplar-free lifelong person re- identification via prodpt-guided adaptive knowledge consolidation. International Journal of Computer Vision, (32(11)4850-4466 2024.
13

---

[61] Zichen Liu, Yuxin Peng, and Jiahuan Zhou. Compositional prompting for anti- morgeftrgg in domain incremental learning. International Journal of Computer Vision, 132(12):5733-5800, 2024.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[63] Dongchen Han, Ziyi Wang. Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song. Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. In NeurlPS, 2024.
[64] Farnoush Rezaci Jafari, Grégoire Montavon, Robbrt Müller, and Oliver Eberle. Mambalrp: Explaining selective state space sequence models. Advances in Neural Information Processing Systems, 37:1 18540-118811000 2024.
[65] Itamar Zimerman, Ameen Ali Ali, and Lior Wolf. Explaining modern gated-linear RNNs via unified implicit attention formulation. In The Thirteenth International Conference Learning Representations, 2025.
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PmLR, 2021.
[67] Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pages 2556-2563, IEEE, 2011.
[68] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11):1-7, 2012.
[69] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitte et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842- 5850, 2017.
[70] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 780-787, 2014.
[71] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arviv preprint arXiv: I 1705.0005555 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose SSP, state space prompting method for video understanding. The main contributions of SSP (i.e., the fine tuning effectiveness and the parameter efficiency) are claimed in both the abstract and introduction accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification:
All theorems used in the paper are properly referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We claim reproducibility in section after acceptance.
4 Our code will be publicly available
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We claim reproducibility in S $ All the datasets included in our study are publicly available. Our code will be publicly available after acceptance. The publicly available code should be adequate to replicate the primary experimental results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
publicc/guides//odeSSttiii sot.eeeoabbmiiioooooooooc.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/mipss.cc/pullc/giidde htt///CooSSSeiiaaeeeeeeee sisssssssiniyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify experimental and implementation details in $
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Considering the large size of the datasets, the experiments are too expensive repeat many times. Additionally, by fixing random seeds and releasing our code after publication, our main experimental results are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are provided in
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper complies with the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
The social impact of our research is discussed in appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We show related asset license and consent to our work in appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
21

---

A Discussion and Analysis
In this section, we discuss and analyze the effectiveness of our SSP method for parameter-efficient fine-tuning the VideoMamba architecture in downstream task adaptation.
For each Mamba layer's input, denote the i-th input token along the scanning order as z;. The hidden state computation for the subsequent Mamba layer is formulated as:
h. A + B x x; + P
(11)
where h; is the i-th compressed hidden state token of the Mamba layer. In the Mamba architecture, each layer's parameter matrices B RPx1 C R IxD and A I are derived from the input token i; through functions SB, Sc, and Sa. respectively:
A ex,( B: Sz(a?),
C, = Sc()),
A, = Sa(x))
(12)
In our proposed SSP method, the intra-frame prompt P enables direct fine-tuning of the parameter matrices generated by each token:
A exp(Ss(a, P)),A), = + p.)S + př);
(13)
where A E RR DxD and Be E RDx represent the forget and update gates directly controlled by the intra-frame prompt pi. facilitating the extraction of locally discriminative information, thereby gathering the spatial information effectively when fine-tuning on downstream tasks.
For the inter-framm prompt p) the j-th frame, this prompt vector aggregates global spatio-temporal information and is gated through Hadamard multiplication with the spatial variance vy. It is inserted between the j-th and + 1)-th frames, directly influencing the hidden state that position:
h5+1,0 hoN + BJ+1,PP
(14)
where denotes the number of tokens per frame. Tokens following the j-th frame can access global discriminative information through the hidden state 1,+ .oo of inter-frame prompt p¹:, overcoming the sequential
information transfer limitation of the original Video.Mamba model and achieving efficient gathering and spreading of spatio mentio-emmporl information.
Next, we further analyze the impact of inter-frame prompts on long-range spatio-temporal information transmission. By expanding Equation I1
we obtain:
hj = A h
B
B
A
A h
h
B)-1
BB,3j
II
Ar
Σ
(1
Σ
A k=t+1
II
Bi
(15)
where i < The coefficient of the first term in the above equation, Ii A represents the influence strength from the i-th hidden state token to the j-th hidden state token along the scanning sequence. We denote this as the transmission matrix T Tz
T T
II A k=++1
- exp
Σ W=++1
A
(16)
where A is negative matrix. Examining the equation above, observe that the mutual influence strength between tokens at different positions in the sequence exponentially decays to zero as the distance j increases. Without the insertion of inter-frame prompts. the maximum information transmission path length the sequence equals the sequence length OTTN), where 7 is the number of video frames and N is the number of tokens per frame. However, after inserting inter-frame prompts, information from tokens at different positions can be transmitted through the inter-frame prompts inserted after each frame, reducing the maximum information transmission path length to O(N). This significant reduction in information transmission path length facilitates the aggregation and propagation of global spatio-temporal information.
22

---

B Visualization of More Cases
B.1 Update Gate Value Visualization
Vidloo
SWP
SSP (Ours)
Video
SVP
SSP (Ours)
Video
SVP
SSP (Ouns)
Figure Visualization of the values of the update gate over the last layer VideoMamba.
The update gate value of each token, denoted as B Equation 13 represents the degree of influence each token exerts the hidden states of VideoMamba. By visualizing the update gate values across tokens, we can observe which specific tokens receive greater attention from the VideoMamba architecture during processing.
23

---

As illustrated in Figure 8 existing prompting approaches for Mamba architecture (e.g.. SVP) employ static prompts for each frame. This limitation prevents the pre-trained VideoMamba model from effectively integrating and modeling global video contextual information, thereby hindering accurate interpretation of human actions within the video sequence. contrast, our SSP method, through the complementary application of intra-frame gathering and inter-frame spreading modules, dynam- ically emphasizes regions exhibiting significant temporal variations in the video. This approach enables more precise comprehension of video content by capturing the most relevant spatio-temporal information.
B.2 Intra-Frame Prompts Visualization
In our SSP method, the intra-frame prompts, which are overlaid on the input tokens fed into each Mamha layer, employed to capture the discriminative local features and gather the spatial information of each frame. To visualize the intra-frame prompts, we plot the values intra-frame prompts of all frames in video over the last Mamba layer heatmaps, and overlay them the original video frames.
Video
Video
ST-Acapter
ST-Adapter
SSP(Ouur))
SSP(Ours)
Video
Video
ST-Adapter
ST-Adapte
SSP(Ours)
SSP(Ours)
Video
Video
ST-Adapler
ST-Acapter
SSPOOurs)
SSP(Ours)
Figure Visualization of our intra-frame prompts over the last layer VideoMamba.
As shown in Figure 9 when existing parameter-effiient fine-tuning methods (e.g., ST-AAapter) are applied to VideoMamba, they fail to effectively capture the local feature maps of individual frames,

---

preventing the model from attending to local features that undergo temporal variations. In contrast, our approach, through efficient spreading of global temporal information, enables intra-frame prompts to more accurately capture discriminative local features, thereby achieving effective gathering of spatially relevant information.
B.3 Inter-Frame Prompts Visualization
The inter-frame prompts in our SSP method facilitate the propagation of temporal information across sequential frames. Through visualization of these inter-frame prompts, identify which specific frames receive heightened attention from the model during global context integration, providing insights into the temporal dynamics of information processing within the architecture.
L.
Iraperal
Video
STOP
SSP
0.0
Ka Intprral Irantiere the
1.0
Video
STOP
SSP
0.0
1.0
Video
STOP
SSP
0.0
1.0
Video
STOP
SSP
0.0
Le tesprd Ireeisiit
1.0
Video
STOP
SSP
0.0
Le Inapral tnroiimii
1.0
Video
STOP
SSP
0.0
Figure 10: Visualization of our inter- fnter-frame prompts over the last layer in VideoMamba.
As demonstrated in Figure TO compared to existing video prompting methods (e.g., STOP), the frem-frame prompts of our SSP method effectively identify key frames exhibiting temporal variations
25

---

within the video sequence. This enhanced capability stems from our approach being specifically engineered for the Mamba architecture, enabling efficient refinement and propagation of temporal information from compressed hidden states.
C Discussion on Different Method Designs
In our methodology design, we generate inter-frame prompts by sampling the last token from each frame in the forward scanning sequence, illustrated in Figure Ia). This section explores how different sampling strategies affect the generation of inter-frame prompts, with supplemen- tary experimental results presented in Ta-
Table 4: Ablation of Different Sample Methods.
Method
HMDB51
UCF101
Breakfast
Middle
74.44
95.87
91.14
Bidirection
75,81
96.00
89.06
Bi- Ii-endepeedent
75.16
96.06
90.10
SSP(Ours)
76.66
97.03
93.23
bleA We discovered that sampling tokens
from the middle of each frame (Figure (T(b) leads average accuracy decrease of 1.82%. We
attribute this decline to the fragmentation of semantic information within each frame during prompt generation. When sampling the last tokens separately from the forward and backward scanning sequences to generate prompts (Figure [[))), the accuracy decreases by an average of 2.01% %. We posit that sampling before the superposition bidirectional sequences causes the inter-frame prompts to overlook complementary information from both directions, resulting in separation of forward and backward contextual cues. Additionally, employing independent inter-frame spreading modules to generate prompts for forward and backward directions (Figure IT)) still results in an accuracy drop of 1.86%. Although this approach introduces more learnable parameters to separately model contextual relationships in forward and backward scanning, it fails to address the fundamental issue of separated forward and backward contextual cues.
Token Seqeence
Token Sequence
Tollen Sequence
Toklen Segarnce
12 DAI HEIT
IES
IFS
Sompled
Token
(a) Sampling Tokens from the End of Frames
(b) Sampling Tokens from the Middle of Frames
Sampled Tolecn
Boolovard Stageence
Sequence
Backwaard Seggeence
P
Sequence
IFS
IFS
loken Soqeerce
Ioken Segeznce
Weight Sharing
e
Porvand Saquence
Forward Saquznce
Mumha Lazze
Sumpled Token
Manraa Leyzr
Sampled Takee
(<) Bidirectional Token Sampling
(d) Bidirectional Token Sampling with Independent Processing
Figure 11: Different methods for inter-frame prompting. For simplicity, irrelevant computational
processes in the Mamba block and intra-frame gathering modules have been omitted.
D More Ablation Studies on Hyperparameters
The internal dimensions of our intra-frame frame gathering module (IFG) and inter-frame spreading module (IFS) are also hyper-parameters of our method. To balance efficiency and effecttiveness, we set the internal dimension IFG to 384 and the internal dimension of IFS to 256. We conducted extensive experiments on HMDB51 and Breakfast datasets to investigate the impact of different dimension settings on aee-tuning performance.
As shown in Figure 12 naively increasing the internal dimension of the IFG module does not improve model performance. When the dimension too large, model performance actually decreases due to increased optimization difficulty. Regarding the internal dimension of the IFS module, the long-video
26

---

Breakfast dataset is more sensitive to this setting. From the perspective of reducing training time and parameter costs, setting the internal dimension of the IFS to 32 is also an acceptable option, which demonstrates the robustness of our method to the choices of hyperparameters.
HMDB51
Breakfast
HNDBS1
Breakfast
Accuracy (%)
Accurcs (%)
Accuracy (56)
Accurayy (%)
T8.95
2001
15.%%
76.77
115
142
384
916
5%
64
126
254
513
Dimension
Figure 12: Ablation on the Internal Dimensions of IFG and IFS.
E
More Clarifications
We first clarify the role of spatial variance in our approach. Given that different video frames exhibit varying degrees of association with global information, we learn the spatial variance through the spatial features aggregated by IFG. The spatial variance gates the weight of each inter-frame prompt, thereby controlling the degree of influence each video frame has on global information and achieving more refined temporal information propagation.
To further elaborate on the spreading mechanism, we describe how IFG and IFS work collaboratively. The IFG module aggregates spatial information from each frame into intra-frame prompts and superimposes the intra-frame prompts onto tokens. While the IFS module performs global attention computation through sampled tokens, enabling global interaction of the aggregated spatial information to generate inter-frame prompts. The inter-frame prompt corresponding to each frame thus contains spatial information from other frames, thereby propagating the aggregated local spatial information in temporal manner.
F
Asset License and Consent
DGL VPT and VFPT are licensed under CC-BY-NC 4.0. Adapter, ST-Adapter and CLIP-ViT are licensed under MIT. STOP Additional-Scan and the VideoMamba are licensed under Apache 2.0.
All the datasets included our study are publicly available, and all the models are publicly available. We would like to state that the contents in the dataset do NOT represent our views or opinions.
G Broader Impacts
This study presents SSP, which improves the performance pre-trained Mamba models when fine-tuned on downstream video tasks. Thanks to the reduced parameter count, our research enables deployment of video foundation models on resource-constraneed devices, reduced environmental impact of AI training, and rapid adaptation to specialized domains.
H Limitations
For potential limitations, our method introduces Intra-Frame Gathering (IFG) module and Inter-Frame Spreading (IFS) module to facilitate spatio-temporal contextual information, which brings additional hyperparameters, such as the number of IFSs. However, according to extensive experiments in $ $4..333 we observe that most cases, simply setting this parameter to sufficient. Regarding the internal dimensions of IFG and IFS, as shown in the extensive experiments in appendix D we recommend setting the internal dimension of IFG to 384 and the internal dimension of IFS to 256.
27


Reviews:
[{"review_id": "fhkB6p0EWE", "reviewer": "Reviewer_TNYB", "summary": "In this paper, a state space prompting (SSP), which is the prompt learning algorithm based on VideoMamba, proposed for video understanding. The proposed SSP consists of two core modules: an intra-frame gathering (IFG) module and an inter-frame spreading (IFS) module. IFG module generates the intra-frame prompts, which gathers the models’ attention to local features,  information entropy weights, and spatial variance measurements. IFS module generates the inter-frame prompts representing temporal inductive biases which spread the gathered temporal information across the frames. SSP only optimizes a few additional parameters and classification head during training. The extensive experimental results on various video understanding benchmarks demonstrate that the proposed SSP achieves better performances than conventional methods with small amounts of trainable parameters.", "questions": "I have carefully read the paper, but as I mentioned in the weaknesses section, I don’t think the proposed algorithm has any significant shortcomings. I believe this is a good paper that can make a meaningful contribution to the community. Nevertheless, I would like to offer a few suggestions below.\n\n---\n\n* In Tables 1 and 2, STOP shows significantly lower performances compared to other methods. It would be helpful to include additional explanation for these results. Although STOP is not specifically designed for VideoMamba, it is similar to SSP in that it is based on intra-frame and inter-frame prompts. Providing more insight into the reasons behind the significant performance gap would strengthen the discussion.\n\n* This paper provides experimental results on four popular datasets, including the large-scale SSv2. Nevertheless, including performance comparisons on additional benchmarks such as Kinetics-400 or LVU would further strengthen the effectiveness of the proposed SSP.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "> **Strength**\n- This paper is well-written and easy to follow. \n- The proposed SSP appears to be technically sound. It also provides a mathematical analysis of the proposed algorithm’s long-range information transmission capability.\n- The proposed algorithm outperforms conventional algorithms on various benchmark tests. \n\n> **Weakness**\n- I don’t think there are any significant weaknesses in the proposed algorithm.", "was_revised": true, "final_justification": "During the rebuttal, the authors addressed my concerns well, and I appreciate all their efforts. I have not identified any additional weaknesses in the proposed algorithm, so I have decided to maintain my original rating of “accept.”", "rebuttal": "### Q1 Response: STOP shows significantly lower performances compared to other methods\n\nThank you for your valuable suggestion. As discussed in Section 2.3 of this paper, the significant performance gap relies on two reasons:\n\n(1) Vision prompting models with Transformer architecture leverage global attention mechanisms that allow tokens at arbitrary positions in the sequence to interact, whereas Mamba-based models process tokens sequentially, leading to adjacent tokens tending to contain more overlapping information. **STOP inputs all tokens into the inter-frame prompting module for computation, introducing excessive redundant information when modeling contextual relationships**, a phenomenon that is more pronounced in video data. In contrast, other fine-tuning methods do not simultaneously model all tokens, making them relatively more capable of extracting key discriminative information from tokens that have been highly compressed by Mamba compared to STOP.\n\n(2) As shown in Table 1, when based on Transformer models, this method exhibits similarly low performance on the large-scale dataset SSv2 as when based on Mamba models, **indicating that in the efficient fine-tuning domain with limited trainable parameter space, greater computational overhead does not necessarily lead to better performance**. Reasonable inductive biases in method design, such as the complementary information aggregation and propagation mechanisms in SSP, can yield superior results.\n\nCompared with STOP, our method utilizes the IFS module to generate inter-frame prompts through token sampling while aggregating local spatial features via IFG, and precisely regulates the influence of local features on global temporal features through entropy weight and spatial variance, thereby learning discriminative spatiotemporal information during fine-tuning. Therefore, as shown in Table 1, our method achieves a **30.28%** improvement on the large-scale dataset SSV2, which verifies that our approach more effectively adapts to the Mamba architecture during fine-tuning.\n\nTable 1. Comparison with STOP\n\n| Method | Backbone | HMDB51 | SSV2 | UCF101 | Avg |\n|--------|----------|--------|------|--------|-----|\n| STOP | VisionTransformer-B | *72.00* | 21.40 | *95.30* | 62.90 |\n| STOP | VideoMamba-M | 71.96 | *23.44* | 94.60 | *63.33* |\n| **SSP** *(Ours)* | VideoMamba-M | **76.66(+2.66)** | **53.72(+30.28)** | **97.03(+1.73)** | **75.80(+12.47)** |\n\n### Q2 Response: Comparisons on more benchmarks\n\nThanks for your valuable suggestion. **We have conducted additional experiments as follows:**\n\n(1) For the Kinetics-400 benchmark, since our VideoMamba is pre-trained on Kinetics-400, where high zero-shot performance of **83.3%** has already been achieved, and existing fine-tuning work typically does not test on pre-training datasets [1][2], we did not conduct further fine-tuning experiments on Kinetics-400 to avoid redundancy.\n\n(2) For the LVU benchmark, we conduct experiments on the LVU benchmark, as shown in Table 2 below, SSP achieves an average performance improvement of **3.64%** over existing state-of-the-art methods on LVU through efficient aggregation and propagation of discriminative spatiotemporal information, which further confirms the effectiveness of our approach.\n\n\nTable 2. Comparisons on LVU\n\n| Methods | Backbone | Params | Rel. | Speak. | Genre | Wtr | Avg |\n|---------|----------|--------|------|--------|-------|-----|-----|\n| Adapter | VideoMamba-Ti | 0.59M | 53.65 | *32.44* | *52.32* | 29.76 | 42.04 |\n| VFPT | VideoMamba-Ti | 0.27M | 56.09 | 31.38 | 51.97 | 26.19 | 41.40 |\n| SVP | VideoMamba-Ti | 0.63M | *56.81* | *32.44* | 52.22 | *33.33* | *43.70* |\n| **SSP** *(Ours)* | VideoMamba-Ti | 0.21M | **58.53(+1.72)** | **34.89(+2.45)** | **53.70(+1.38)** | **42.26(+8.93)** | **47.34(+3.64)** |\n\n[1]: Pan, Junting, et al. \"St-adapter: Parameter-efficient image-to-video transfer learning.\" Advances in Neural Information Processing Systems 35 (2022): 26462-26477.\n\n[2]: Liu, Zichen, et al. \"STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "06uU0L8BFk", "reviewer": "Reviewer_Br1d", "summary": "SSP is a PEFT method for Mamba-based video understanding models. It tackles the information decay in state space models (SSMs) through a \"gathering and spreading\" framework. This framework comprises Intra-Frame Gathering (IFG), which extracts salient spatial features and quantifies their importance via information entropy and spatial variance, and Inter-Frame Spreading (IFS), which uses this data to create global temporal prompts and gates temporal information based on spatial variance. SSP achieves competitive performance with minimal trainable parameters.", "questions": "In Table 2, the number of trainable parameters for SSP increases significantly with model size (e.g., from 0.98M in VideoMamba-S to 2.41M in -M), while STOP remains relatively stable despite also using attention and convolution layers. It would be helpful if the authors could provide a breakdown of where these additional parameters come from, and explain why the proposed method scales more heavily with model size compared to STOP.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "(+) The visualization of information decay in state space models is insightful and helps motivate the need for the proposed SSP framework.\n(+) The proposed method demonstrates strong performance across multiple video understanding benchmarks, showing its practical effectiveness.\n\n(-) While the paper states that the proposed modules are applied after the first Mamba layer, it remains unclear why this particular depth was chosen, and how the performance or parameter efficiency would change if the modules were applied at deeper layers. This is especially relevant for the IFG module, which computes entropy and spatial variance from intermediate representations — quantities that are likely to be sensitive to the semantic level of features. An ablation or discussion on the layer-wise placement of IFG/IFS would help clarify the design rationale and the robustness of the proposed method.\n(-) The comparison with STOP could be misleading, as the two methods are evaluated under different objectives—STOP adopts a text-video matching paradigm, whereas SSP is evaluated using a classification loss. To more fairly assess the effectiveness of the proposed method, it would be helpful to report results under the same matching-based setup as STOP.\n\nMinor\nFigure 2 shows the proposed modules being inserted after the i-th layer, while the main text states that they are applied after the first Mamba layer. For clarity and consistency, it would be better if the figure also reflected the first-layer placement.", "was_revised": true, "final_justification": "The rebuttal clarifies most of the points I raised. Although there are still some limitations, as noted by other reviewers, I acknowledge the strengths of the work and will retain my initial evaluation.", "rebuttal": "### W1 Response: The Placement Depth of IFG/IFS\n\nThank you for your valuable suggestion. We conducted ablation experiments on the placement depth of IFG/IFS on the HMDB51 and Breakfast datasets, with results shown in the Table 1, which implies the following conclusions:\n\n(1) **Applying the modules to deeper layers does not yield performance improvements**. When our module is applied after the 8th layer, our method experiences slight accuracy decreases of **0.19%** and **0.01%** on HMDB51 and Breakfast datasets, respectively. When applied to deeper layers, the accuracy gradually declines. When our module is applied after the 24th layer, the accuracy significantly drops by **5.21%** on the Breakfast dataset and by **3.00%** on the HMDB51 dataset. *It is because that after sequential compression by multiple Mamba layers, some discriminative information in the token sequence becomes more difficult for IFG/IFS to extract.* While our method achieves optimal performance through appropriate hyperparameter selection, obtaining the highest accuracy of **76.66%** and **93.23%** on the HMDB51 and Breakfast datasets, respectively\n\n(2) **Applying the modules to deeper layers **does not** change the parameter efficiency**, as our IFG/IFS weights are shared across layers. As shown in Table 1, when SSP modules are applied at different depths, the trainable parameters are fixed at **2.41M**. Our method achieves effective aggregation and propagation of discriminative spatiotemporal information using only a small number of trainable parameters, surpassing existing state-of-the-art fine-tuning methods by an average of **2.76%** as demonstrated in Table 2 in the main text of our paper, *with the number of trainable parameters remaining constant regardless of placement depth, which further verifies our robustness.*\n\nTable 1. Ablation Experiments on the Placement Depth of IFG/IFS\n\n| Methods \\ Layers | 1     | 8     | 16    | 24    |\n|------------------|-------|-------|-------|-------|\n| HMDB51          | **76.66** | 76.47(-0.19) | 75.55(-1.11) | 73.66(-3.00) |\n| Breakfast       | **93.23** | 93.22(-0.01) | 92.70(-0.53)  | 88.02(-5.21) |\n\n### W2 Response: Comparison with STOP\n\nThanks for your valuable suggestion. **We have reported results under the same matching-based setup with STOP.**\n\nIn the original STOP paper, the authors performed evaluations in two paradigms: classification and matching.\n\n(1) As for the classification task, our method achieves accuracy improvements of **4.66%**, **1.73%**, and **32.32%** over STOP on the HMDB51, UCF101, and SS-V2 datasets respectively, as shown in Table 1 of the original STOP paper. Our method efficiently aggregates and propagates key information through fine-tuning designs tailored for the Mamba architecture, demonstrating superior performance on classification tasks, which further verifies the effectiveness of our SSP method in parameter-efficient fine-tuning on the Mamba architecture.\n\n(2) As for matching task, we have conducted more experiments on text-video matching tasks using MSVD dataset. As shown in Table 2, our method achieves an average recall rate that exceeds STOP by **9.06** in text-to-video retrieval and by **12.33** in video-to-text retrieval. As a prompt tuning method that similarly focuses on the Visual Encoder, **SSP demonstrates stronger text-video matching capability through efficient aggregation and propagation of discriminative spatiotemporal features**, which further showcases the effectiveness of SSP in parameter-efficient fine-tuning on the Mamba architecture.\n\nTable 2. Comparison results with STOP under the text-video matching paradigm\n\n| Methods | Visual Backbone | Pretrain Pairs | Params | Text→Video R@1 | Text→Video R@5 | Text→Video R@10 | Text→Video R@Avg | Video→Text R@1 | Video→Text R@5 | Video→Text R@10 | Video→Text R@Avg |\n|---------|-----------------|----------------|--------|----------------|----------------|------------------|------------------|----------------|----------------|------------------|------------------|\n| STOP    | VideoMamba      | 25M            | 1.5M   | 36.11          | 65.19          | 75.86            | 59.05            | 52.84          | 71.04          | 76.87            | 66.92            |\n| **SSP** (*Ours*)    | VideoMamba      | 25M            | 2.4M   | **45.62(+9.51)**          | **75.06(+9.87)**          | **83.66(+7.80)**            | **68.11(+9.06)**            | **66.12(+13.28)**          | **83.73(+12.69)**          | **87.91(+11.04)**            | **79.25(+12.33)**            |\n\n### W3 Response: About Figure 2\n\nThank you for your comments. As you said, our proposed modules are applied after the first layer and every subsequent Mamba layer. Therefore, we have supplemented Figure 2 with the insertion of our modules after the 1st layer to further clarify our method, which will be reflected in the camera-ready version. Thank you again for your valuable suggestions.\n\n### Q1 Response: Where the additional parameters come from?\n\n**It should be clarified that the variation in the number of trainable parameters in our SSP stems from changes in the number of stacked IFS modules, rather than changes in model size.**\n\n*Firstly*, when applied to the smaller VideoMamba-S model, we only need to use **1** IFS module to surpass other fine-tuning methods, with a parameter count of **0.98M**. Stacking more modules does not yield performance improvements.\n\n*Secondly*, when applied to the larger VideoMamba-M model, as shown in the ablation experiment in Figure 7, the best performance is achieved with **3** stacked IFS modules, corresponding to a parameter count of **2.41M**. Following the original STOP paper, we did not stack the attention layers or convolutional layers of the STOP method, thus the parameters remained stable at **1.49M**. \n\n*In summary*, regardless of whether under the setting of fewer trainable parameters (**0.98M**) or more trainable parameters (**2.41M**), our method achieves average improvements of **1.21%** and **2.76%** compared to existing state-of-the-art fine-tuning methods in Table 1 and Table 2 in the main text, respectively, through the extraction of discriminative features from the hidden states of the Mamba model. **Under the same parameter efficiency, this demonstrates the effectiveness of our approach.**"}, {"review_id": "GD0NkukEPI", "reviewer": "Reviewer_VAWP", "summary": "This paper proposes State Space Prompting (SSP), a parameter-efficient fine-tuning method for VideoMamba, a state space model for video understanding. SSP introduces two lightweight modules—Intra-Frame Gathering (IFG) and Inter-Frame Spreading (IFS)—to capture spatial cues within frames and propagate temporal information across frames, mitigating information decay in state space compression. SSP enables efficient adaptation to downstream video tasks with minimal tunable parameters and achieves state-of-the-art results across multiple video benchmarks.", "questions": "1. Have you evaluated the proposed method on more complex video understanding tasks beyond classification, such as temporal action segmentation or video object segmentation?\nWithout such evaluations, it is unclear whether the IFG and IFS modules truly benefit spatio-temporal reasoning as claimed.\n\nHow do you mitigate the risk of losing fine-grained spatial or temporal details due to the heavy aggregation performed by IFG and IFS?\nHave any ablation or visualization studies been conducted to confirm that key local information is preserved?\n\nGiven that the method is only applied to video classification, would you consider narrowing the paper’s scope or revising the title to better reflect its actual contributions?\nThis would help manage reader expectations and align the paper's claims with its experiments.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths\nThe paper is technically sound and proposes a novel architecture that integrates state space modeling with spatio-temporal feature manipulation via Information Gathering (IFG) and Information Spreading (IFS) modules. The implementation appears well-executed, and the model achieves solid results on several standard video classification benchmarks. The paper is clearly written and well-structured. The roles of IFG and IFS are well explained, and diagrams help convey the pipeline effectively.\n\nWeaknesses\n1. Despite the title claiming a contribution to \"video understanding,\" the paper only evaluates on video classification, which is relatively simple and insufficient to validate the claimed benefits of IFG and IFS for spatio-temporal reasoning. It would be more convincing to test on video segmentation (e.g., instance/object/panoptic) or temporal tasks like action segmentation, detection, or localization.\n\n2. The IFG and IFS modules perform heavy information aggregation, which may lead to a loss of fine-grained spatial or temporal details. This could limit the method's effectiveness on tasks that require precise local reasoning, especially outside of classification.\n\n3. If the proposed approach is designed specifically for video classification, the title should reflect that scope more accurately. A clearer title would be:\n\"State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Classification.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed most of my concerns. I am currently considering raising my score and will make a final decision during the discussion phase.", "rebuttal": "### W1 & Q1 Response: Evaluation on more tasks beyond video classification\n\nThank you for your valuable suggestion. **We have conducted more experiments on video understanding tasks beyond video classification.**\n\nFollowing the parameter-efficient video learning community, we conducted fair comparisons with existing fine-tuning methods under the same video classification paradigm using four mainstream datasets in the main text. **Additionally, we also compared with existing methods on the THUMOS dataset under the temporal action localization paradigm and on the 50salads and GTEA datasets under the temporal action segmentation paradigm.**\n\n(1) For the temporal action localization paradigm, as shown in Table 1, the average mAP score of our method exceeds existing state-of-the-art fine-tuning methods by **1.83**. Our proposed SSP demonstrates stronger temporal action localization capability through efficient aggregation and propagation of discriminative spatiotemporal features, which further showcases the effectiveness of SSP in challenging spatial-temporal reasoning tasks.\n\n(2) For the temporal action segmentation paradigm, as shown in Table 2, on the 50salads dataset, SSP achieves an accuracy that exceeds the existing best fine-tuning method by **2.90%**, with the F1@50 score obtaining an improvement of **2.27**. As shown in Table 3, on the GTEA dataset, SSP achieves an accuracy that surpasses existing state-of-the-art methods by **1.47%**, with the F1@50 score also obtaining an improvement of **1.73**. This demonstrates that our method achieves efficient extraction of discriminative spatiotemporal information from Mamba hidden states through effective aggregation of local information and efficient propagation of temporal information, **making improvements in spatio-temporal reasoning on tasks beyond classification as well**, which further verifies the effectiveness of our SSP method.\n\n**In summary, all the above experimental results demonstrate our effectiveness and generalization ability on various video tasks.**\n\nTable 1. Comparison on the THUMOS datasets under the temporal action localization paradigm\n\n| Methods | Backbone     | Params | mAP@0.3 | mAP@0.4 | mAP@0.5 | mAP@0.6 | mAP@0.7 | mAP@Avg |\n|---------|--------------|--------|---------|---------|---------|---------|---------|---------|\n| Adapter | ActionMamba  | 1.05M  | *81.87*   | *77.37*   | *69.67*   | *56.82*   | *41.36*   | *65.42*   |\n| SVP     | ActionMamba  | 1.31M  | 81.56   | 77.33   | 68.78   | 56.65   | 38.21   | 64.51   |\n| **SSP** *(Ours)*    | ActionMamba  | 1.05M  | **83.40(+1.53)**   | **78.88(+1.51)**   | **71.40(+1.73)**   | **59.55(+2.73)**   | **43.00(+1.64)**   | **67.25(+1.83)**   |\n\nTable 2. Comparison on the 50salads datasets under the temporal action segmentation paradigm\n\n| Methods | Backbone | Params | Acc   | Edit  | F1@10 | F1@25 | F1@50 |\n|---------|----------|--------|-------|-------|-------|-------|-------|\n| Adapter | ASMamba  | 0.82M  | 70.45 | 51.25 | 57.62 | 53.38 | 43.64 |\n| VPT     | ASMamba  | 0.49M  | *72.75* | 50.92 | 59.70 | 57.14 | *45.20* |\n| VFPT    | ASMamba  | 0.49M  | 69.74 | **57.98** | *61.02* | *58.35* | 44.09 |\n| SVP     | ASMamba  | 0.88M  | 68.63 | 51.68 | 56.75 | 53.08 | 42.56 |\n| **SSP** *(Ours)* | ASMamba  | 0.82M  | **75.65(+2.90)** | *53.58* | **64.61(+3.59)** | **60.65(+2.30)** | **47.47(+2.27)** |\n\nTable 3. Comparison on the GTEA datasets under the temporal action segmentation paradigm\n\n| Methods | Backbone | Params | Acc   | Edit  | F1@10 | F1@25 | F1@50 |\n|---------|----------|--------|-------|-------|-------|-------|-------|\n| Adapter | ASMamba  | 0.81M  | 66.71 | 64.85 | 65.88 | 58.82 | 40.00 |\n| VPT     | ASMamba  | 0.48M  | 66.69 | *66.54* | *67.20* | 60.80 | *44.80* |\n| VFPT    | ASMamba  | 0.48M  | 65.87 | 65.01 | 66.12 | *61.22* | 39.18 |\n| SVP     | ASMamba  | 0.87M  | *67.39* | 66.49 | 65.30 | 60.40 | 40.81 |\n| **SSP** *(Ours)* | ASMamba  | 0.81M  | **68.86(+1.47)** | **73.91(+7.37)** | **71.83(+4.63)** | **66.12(+4.90)** | **46.53(+1.73)** |\n\n### W2 Response: The potential loss of fine-grained spatial or temporal details\n\n**Compared to existing fine-tuning methods, our SSP method achieves better preservation of fine-grained spatial and temporal details.**\n\nTherefore, we performed further experiments on temporal action localization and temporal action segmentation tasks to validate the effectiveness of our approach in extracting fine-grained video spatiotemporal features.\n\n(1) Regarding the temporal action localization paradigm, the average mAP score of our method exceeds existing state-of-the-art fine-tuning methods by **1.83**, as shown in Table 1, demonstrating better fine-grained spatiotemporal understanding capability during fine-tuning. This is because our IFG and IFS modules both *perform low-rank projection during convolution and attention computation*, enabling the modules to effectively aggregate and propagate discriminative spatiotemporal information while preserving discriminative fine-grained spatial or temporal details.\n\n(2) Regarding the temporal action segmentation paradigm, our SSP surpasses existing fine-tuning methods with accuracy gains of **2.90%** and **1.47%** demonstrated in Table 2 and Table 3, respectively. It illustrates that *our approach better preserves fine-grained spatial and temporal details through the 2D local convolution of the IFG module and the inter-frame information propagation of the IFS module,* which further proves the effectiveness of our fine-tuning method on the Mamba architecture.\n\nIn summary, these experimental results above indicate that our SSP method does not suffer from the potential loss of fine-grained spatial or temporal details. Instead, our approach successfully preserves and leverages fine-grained spatial and temporal details, validating the effectiveness of our proposed fine-tuning method.\n\n### Q2 Response: How do you mitigate the risk of losing fine-grained spatial or temporal details due to the heavy aggregation performed by IFG and IFS?\n\n**In order to effectively aggregate and propagate fine-grained discriminative spatiotemporal information, our IFG and IFS modules both perform low-rank projection during convolution and attention computation.**\n\n(1) Regarding the temporal action localization paradigm, the average mAP score of our method exceeds the SOTA fine-tuning methods by **1.83**, as shown in Table 1, demonstrating better fine-grained spatiotemporal understanding capability. Regarding the temporal action segmentation paradigm, our SSP surpasses SOTA methods with accuracy gains of **2.90%** and **1.47%** demonstrated in Table 2 and Table 3, respectively. This is because our low rank IFG and IFS modules effectively aggregate and propagate discriminative information while preserving discriminative fine-grained spatial or temporal details, which further proves the effectiveness of our fine-tuning method.\n\n(2) We conducted ablation experiments to demonstrate the effectiveness of our approach. As shown in Table 4 and Table 5, on the HMDB51 and Breakfast datasets, when the internal dimension of IFG is increased to **576**, the accuracy decreases by **0.65%** and **4.69%** compared to the optimal configuration, respectfully. When the internal dimension of IFS is increased to **512**, the accuracy decreases by **2.35%** and **4.69%**, respectfully. Therefore, **we achieve a balance between heavy aggregation and fine-grained modeling** by selecting appropriate intrinsic low-rank dimensions for IFG and IFS to ensure effective aggregation of spatiotemporal details.\n\n(3) As shown by the visualization study in Figure 9 in the appendix, even after heavy aggregation, **our intra-frame prompts can still preserve fine-grained spatial details compared to other fine-tuning methods.** As demonstrated by the visualization study in Figure 10 in the appendix, **our intra-frame prompts accurately focus on key temporal transitions in the video, preserving fine-grained temporal information,** which further verifies our effectiveness.\n\nTable 4. Ablation on the Internal Dimensions of IFG\n\n| Methods \\ Dimensions | 48    | 96    | 192   | 384   | 576   |\n|---------|-------|-------|-------|-------|-------|\n| HMDB51  | 74.77(-1.89) | 75.03(-1.63) | 75.42(-1.24) | **76.66** | 76.01(-0.65) |\n| Breakfast | 91.66(-1.57) | 91.14(-2.09) | 89.58(-3.65) | **93.23** | 88.54(-4.69) |\n\nTable 5. Ablation on the Internal Dimensions of IFS\n\n| Methods \\ Dimensions | 32    | 64    | 128   | 256   | 512   |\n|---------|-------|-------|-------|-------|-------|\n| HMDB51  | **77.18** | 76.01(-1.17) | 75.75(-1.43) | 76.66(-0.52) | 74.83(-2.35) |\n| Breakfast | 91.14(-2.09) | 88.02(-5.21) | 92.18(-1.05) | **93.23** | 88.54(-4.69) |\n\n### W3 & Q3 Response: About the title\n\nThank you for your suggestion. We have already supplemented experiments on temporal action localization and temporal action segmentation in previous responses and validated the effectiveness of our method in fine-grained spatiotemporal feature extraction.\n\nAs shown in Table 1, the average mAP score of our method exceeds existing state-of-the-art fine-tuning methods by **1.83** on the temporal action localization task. As shown in Table 2 and Table 3, our method achieves accuracy gains of **2.90%** and **1.47%** on the temporal action segmentation task, respectively. These results indicate that our SSP method effectively aggregates and propagates discriminative spatiotemporal information, proving our effectiveness and generalization ability on various video understanding tasks."}, {"review_id": "89Ff1aW9Ex", "reviewer": "Reviewer_Ab59", "summary": "Pre-trained state space models compress visual tokens efficiently for video classification but struggle to capture spatial and temporal context. To address this, the authors propose State Space Prompting (SSP), which uses intra-frame and inter-frame prompts to better gather and spread key spatiotemporal information. The Intra-Frame Gathering module captures spatial details within frames, while the Inter-Frame Spreading module shares important cues across frames. SSP improves video understanding by effectively balancing and compressing this information. Experiments conducted on four video benchmark datasets demonstrate the contribution of the approach.", "questions": "1) What is the specific technical novelty of the proposed approach beyond incremental improvements, and how does it effectively learn discriminative spatio-temporal information? \n\n2) How exactly do the Intra-Frame Gathering and Inter-Frame Spreading modules spread gathered local spatial information in a temporal manner? \n\n3) Can the authors clearly explain how the proposed spatial variant works and how it enhances the representation of discriminative action features?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The motivation of the proposed approach makes sense, as it enhances the understanding of spatiotemporal information through intra-frame and inter-frame prompts. It is easy to read and follow. The proposed approach achieves state-of-the-art results, as shown in Table 1 and Table 2.\n\nWhile the proposed approach shows some technical improvement, its novelty is limited and largely incremental. It remains unclear how the method effectively learns discriminative spatio-temporal features. Additionally, the explanation in lines 60–65 regarding the “Intra-Frame Gathering module and Inter-Frame Spreading module facilitating spatio-temporal contextual interaction by spreading gathered local spatial information temporally” is vague and lacks clarity on the actual mechanism of temporal spreading.\n\nMoreover, the presentation must clearly explain how the proposed spatial variant operates and how it enhances the representation of discriminative action features. Without this, the contribution to improving spatiotemporal feature learning is not convincingly demonstrated.\n\nIt is not clear how the parameters are calculated. Figure 5 shows 1.30M parameters, but the numbers in Table 1 and Table 2 are different.", "was_revised": true, "final_justification": "Considering the clarifications provided by the authors during the rebuttal period, some of my concerns and questions have been addressed. I am inclined to increase my score to borderline accept", "rebuttal": "### W1 & Q1 Response: What is the specific technical novelty of the proposed approach beyond incremental improvements, and how does it effectively learn discriminative spatio-temporal information? What is the actual mechanism of \"spreading gathered local spatial information temporally\"?\n\nThank you for your valuable review. **We highlight the following aspects to illustrate the novelty and operational mechanism of our approach:**\n\n(1) **Firstly, unlike existing fine-tuning methods, our approach is specifically designed for Mamba architecture models.**\n\n- Through the low-rank convolution of the IFG module, **we learn discriminative spatial features and superimpose the aggregated local spatial information onto tokens**. The IFG module effectively introduces two-dimensional spatial inductive bias into sequential Mamba models while requiring only **~3%**(**2.41M/74.00M**) of tunable parameters compared to full fine-tuning, as shown in Table 2, enabling effective spatial feature aggregation and learning.\n\n- Leveraging the inherent characteristic of Mamba models to sequentially compress tokens, our IFS generates inter-frame prompts through global attention using sampled tokens from each frame, **achieving temporal propagation of local spatial information in this process**. As discussed in Sec 6.1 of the appendix, the IFS module significantly reduces the path length for information transmission, facilitating the flow of temporal information.\n\n(2) **Secondly, the proposed IFG and IFS work in a complementary manner to learn discriminative spatio-temporal information.**\n\n- As mentioned above, the intra-frame prompts generated by IFG are superimposed onto the tokens, enabling intra-frame tokens to acquire discriminative local spatial information.\n\n- IFS conducts low-rank attention computation on sampled tokens, allowing the aggregated spatial information from each frame to interact and generate inter-frame prompts.\n\n- The inter-frame prompts are inserted at the boundaries between frames and fed into the next Mamba layer, enabling the Mamba layer to simultaneously model the local spatial information contained in intra-frame tokens and the global temporal information contained in inter-frame prompts when processing tokens.\n\n(3) **We clarify the spreading mechanism of gathered local spatial information through the IFG and IFS modules as follows:**\n\n- **Firstly**, the IFG module **aggregates spatial information from each frame** into intra-frame prompts and superimposes the intra-frame prompts onto tokens. **Secondly**, the IFS module performs global attention computation through sampled tokens, **enabling global interaction of the aggregated spatial information** to generate inter-frame prompts. **Thirdly**, the inter-frame prompt corresponding to each frame thus contains spatial information from other frames, **thereby propagating the aggregated local spatial information in a temporal manner**.\n\n- As shown by the visualization study in Figure 9 in the appendix, the intra-frame prompts generated by the IFG module **successfully aggregate key local spatial information.** The intra-frame prompts are superimposed onto tokens, further facilitating the IFS in propagating local spatial information.\n\n- As demonstrated by the visualization study in Figure 10 in the appendix, after the propagation of discriminative information in a temporal manner, **our method successfully focuses on key temporal transitions in the video,** which further verifies our effectiveness.\n\n- As shown by the ablation experiments in Table 3, when the IFG module is removed, the accuracy decreases by an average of **4.25%**, and when the IFS module is removed, the accuracy decreases by an average of **3.67%**. **This indicates that the IFG and IFS modules play important roles in the aggregation and propagation of discriminative spatiotemporal information.**\n\n(4) As shown in Table 1 and Table 2, our method achieves average improvements of **1.21%** and **2.76%** compared to existing state-of-the-art fine-tuning methods, realizing effective learning of discriminative spatiotemporal features, which further verifies our effectiveness.\n\nTable 1. The comparison results on VideoMamba-S (Parameters **25.42M**)\n\n| Method | Param | HMDB51 | SSV2 | UCF101 | Breakfast | Avg |\n|--------|-------|--------|------|--------|-----------|-----------|\n| Full | 25.42M | 67.58 | 58.57 | 92.96 | 94.27 | 78.34 |\n| Adapter | 2.40M | *73.79* | 36.45 | 94.18 | *84.89* | *72.33* |\n| ST-Adapter | 2.69M | 70.52 | 30.94 | 94.87 | 77.60 | 68.48 |\n| VPT | 1.50M | 72.74 | 30.68 | 95.16 | 81.25 | 69.95 |\n| VFPT | 1.50M | 72.41 | 30.37 | 95.08 | 79.68 | 69.38 |\n| SVP | 2.76M | 69.93 | *38.01* | 95.58 | 80.72 | 71.06 |\n| Additional-Scan | 0.66M | 73.20 | 33.71 | *95.63* | 78.64 | 70.29 |\n| STOP | 1.49M | 70.06 | 21.22 | 93.44 | 65.62 | 62.58 |\n| **SSP** *(Ours)* | 0.98M | **74.38(+0.59)** | **38.68(+0.67)** | **95.69(+0.06)** | **85.41(+0.52)** | **73.54(+1.21)** |\n\nTable 2. The comparison results on VideoMamba-M (Parameters **74.00M**)\n\n| Method | Param | HMDB51 | SSV2 | UCF101 | Breakfast | Avg |\n|--------|-------|--------|------|--------|-----------|-----|\n| Full | 74.00M | 76.30 | 67.30 | 96.00 | 95.31 | 83.73 |\n| Adapter | 2.40M | 73.59 | 46.98 | 96.24 | 89.06 | 76.47 |\n| ST-Adapter | 2.69M | 71.96 | 31.91 | 94.81 | 67.70 | 66.60 |\n| VPT | 1.50M | 73.59 | 40.58 | 95.45 | 81.77 | 72.85 |\n| VFPT | 1.50M | 71.89 | 39.68 | 95.71 | 86.97 | 73.56 |\n| SVP | 2.76M | *73.66* | *49.08* | *96.77* | *90.10* | *77.40* |\n| Additional-Scan | 1.33M | 73.52 | 44.65 | 96.32 | 86.97 | 75.37 |\n| STOP | 1.49M | 71.96 | 23.44 | 94.60 | 71.35 | 65.34 |\n| **SSP** *(Ours)* | 2.41M | **76.66(+3.00)** | **53.72(+4.64)** | **97.03(+0.26)** | **93.23(+3.13)** | **80.16(+2.76)** |\n\nTable 3. Ablation of different prompting modules\n\n| IFG | IFS | HMDB51 | UCF101 | Breakfast | Avg |\n|-----|-----|--------|--------|-----------| -------- |\n| - | - | 59.34(-17.32) | 90.64(-6.39) | 76.56(-16.67) | 75.51(-13.46) |\n| √ | - | 74.44(-2.22) | 96.56(-0.47) | 84.89(-8.34) | 85.30(-3.67) |\n| - | √ | 72.61(-4.05) | 96.14(-0.89) | 85.41(-7.82) | 84.72(-4.25) |\n| √ | √ | **76.66** | **97.03** | **93.23** | **88.97** |\n\n### W2 & Q3 Response: Can the authors clearly explain how the proposed spatial variant works and how it enhances the representation of discriminative action features?\n\nThanks for your valuable question. We elucidate the working mechanism of spatial variance and its approach to enhancing discriminative action feature representation through the following three points:\n\n(1) As shown in Figure 2 in the main text, the inter-frame prompts generated by IFS correspond one-to-one with the video frames from which the sampled tokens originate, and they are inserted at the boundaries between video frames, **enabling the next Mamba layer to simultaneously model the global temporal information brought by inter-frame prompts when processing adjacent frames.**\n\n(2) However, different video frames have varying degrees of association with global information. We aim to further learn a feature called \"spatial variant\" through the spatial features aggregated by IFG, where spatial variant gates the weight of each inter-frame prompt, **thereby controlling the degree of influence each video frame has on global information and achieving more refined temporal information propagation.**\n\n(3) As shown in the ablation experiment results in Table 4, when the spatial variant mechanism is removed, the classification accuracy drops by an average of **2.33%**, indicating that **the spatial variant mechanism is beneficial for enhancing the modeling capability of our method.**\n\nIn summary, spatial variant can achieve more refined temporal information propagation and enhance the representation of discriminative action features **by effectively controlling the influence of local information on global information**.\n\nTable 4. Ablation of spatial variant\n\n| Entropy | Spatial | HMDB51 | UCF101 | Breakfast | Avg |\n|---------|---------|--------|--------|-----------| -------- |\n| - | - | 74.70(-1.96) | 96.56(-0.47) | 89.58(-3.65) | 86.95(-2.02) |\n| √ | - | 75.09(-1.57) | 96.29(-0.74) | 88.54(-4.69) | 86.64(-2.33) |\n| - | √ | 74.96(-1.70) | 96.61(-0.42) | 88.54(-4.69) | 86.70(-2.27) |\n| √ | √ | **76.66** | **97.03** | **93.23** | **88.97** |\n\n### W3 Response: How the parameters are calculated? Figure 5 shows 1.30M parameters, but the numbers in Table 1 and Table 2 are different.\n\nThe parameter variation in our method stems from changes in the number of IFS modules, **which is demonstrated by the ablation experiment in Figure 7 in the main text.**\n\nThe **1.30M** parameters shown in Figure 5 come from a configuration using only one IFS module, intended to demonstrate that our method surpasses the performance of existing methods while using the minimum number of parameters. This is explained in both the caption of Figure 5 and lines **273-274** of the main text. **Our method achieves better performance under the same parameter scale, which further verifies our effectiveness**.\n\n\n### Q2 Response: How exactly do the Intra-Frame Gathering and Inter-Frame Spreading modules spread gathered local spatial information in a temporal manner?\n\nThanks for your valuable question. **We have clarified the spreading mechanism of gathered local spatial information in the W1 & Q1 section above.**\n\n**Firstly**, the IFG module aggregates spatial information from each frame. **Secondly**, the IFS module enables global interaction of the aggregated spatial information. **Thirdly**, the inter-frame prompt corresponding to each frame thus contains spatial information from other frames, propagating the aggregated information in a temporal manner.\n\nAs shown in Table 4, when the IFG or IFS module is removed, the accuracy decreases by an average of **4.25%** or **3.67%**. **This indicates the important roles of IFG and IFS modules in the aggregation and propagation of spatiotemporal information.**"}]

Author rebuttal:

Meta-review: All reviewers commend the intuitive motivation behind the proposed state-space prompting method for enhancing inter- and intra-frame interactions in video understanding tasks. The visualization of information decay in state space models provides insightful justification for the proposed SSP framework and effectively motivates its design. The authors' rebuttal successfully addressed initial concerns regarding the limitation to video classification tasks, unfair comparisons to STOP, missing ablations on the placement of IFG/IFS modules, and questions about the work's novelty. All reviewers expressed satisfaction with the responses to their points and indicated willingness to increase their ratings, with several following through on this commitment. The area chairs concur with the reviewers' assessment that this submission makes a valuable contribution to the field of video understanding, presenting a well-motivated approach with solid experimental validation and thorough analysis of the proposed components.
Decision: Accept (poster)