Title: State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Computer Vision and Pattern Recognition
Abstract: Recently, pre-trained state space models have shown great potential for video classification, which sequentially compresses visual tokens in videos with linear complexity, thereby improving the processing efficiency of video data while maintaining high performance. To apply powerful pre-trained models to downstream tasks, prompt learning is proposed to achieve efficient downstream task adaptation with only a small number of fine-tuned parameters. However, the sequentially compressed visual prompt tokens fail to capture the spatial and temporal contextual information in the video, thus limiting the effective propagation of spatial information within a video frame and temporal information between frames in the state compression model and the extraction of discriminative information. To tackle the above issue, we proposed a State Space Prompting (SSP) method for video understanding, which combines intra-frame and inter-frame prompts to aggregate and propagate key spatiotemporal information in the video. Specifically, an Intra-Frame Gathering (IFG) module is designed to aggregate spatial key information within each frame. Besides, an Inter-Frame Spreading (IFS) module is designed to spread discriminative spatio-temporal information across different frames. By adaptively balancing and compressing key spatio-temporal information within and between frames, our SSP effectively propagates discriminative information in videos in a complementary manner. Extensive experiments on four video benchmark datasets verify that our SSP significantly outperforms existing SOTA methods by 2.76\% on average while reducing the overhead of fine-tuning parameters.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPre-trained state space models compress visual tokens efficiently for video classification but struggle to capture spatial and temporal context. To address this, the authors propose State Space Prompting (SSP), which uses intra-frame and inter-frame prompts to better gather and spread key spatiotemporal information. The Intra-Frame Gathering module captures spatial details within frames, while the Inter-Frame Spreading module shares important cues across frames. SSP improves video understanding by effectively balancing and compressing this information. Experiments conducted on four video benchmark datasets demonstrate the contribution of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nThe motivation of the proposed approach makes sense, as it enhances the understanding of spatiotemporal information through intra-frame and inter-frame prompts. It is easy to read and follow. The proposed approach achieves state-of-the-art results, as shown in Table 1 and Table 2.\n\nWhile the proposed approach shows some technical improvement, its novelty is limited and largely incremental. It remains unclear how the method effectively learns discriminative spatio-temporal features. Additionally, the explanation in lines 60–65 regarding the “Intra-Frame Gathering module and Inter-Frame Spreading module facilitating spatio-temporal contextual interaction by spreading gathered local spatial information temporally” is vague and lacks clarity on the actual mechanism of temporal spreading.\n\nMoreover, the presentation must clearly explain how the proposed spatial variant operates and how it enhances the representation of discriminative action features. Without this, the contribution to improving spatiotemporal feature learning is not convincingly demonstrated.\n\nIt is not clear how the parameters are calculated. Figure 5 shows 1.30M parameters, but the numbers in Table 1 and Table 2 are different.\n\nQUESTIONS:\n1) What is the specific technical novelty of the proposed approach beyond incremental improvements, and how does it effectively learn discriminative spatio-temporal information? \n\n2) How exactly do the Intra-Frame Gathering and Inter-Frame Spreading modules spread gathered local spatial information in a temporal manner? \n\n3) Can the authors clearly explain how the proposed spatial variant works and how it enhances the representation of discriminative action features?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes State Space Prompting (SSP), a parameter-efficient fine-tuning method for VideoMamba, a state space model for video understanding. SSP introduces two lightweight modules—Intra-Frame Gathering (IFG) and Inter-Frame Spreading (IFS)—to capture spatial cues within frames and propagate temporal information across frames, mitigating information decay in state space compression. SSP enables efficient adaptation to downstream video tasks with minimal tunable parameters and achieves state-of-the-art results across multiple video benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper is technically sound and proposes a novel architecture that integrates state space modeling with spatio-temporal feature manipulation via Information Gathering (IFG) and Information Spreading (IFS) modules. The implementation appears well-executed, and the model achieves solid results on several standard video classification benchmarks. The paper is clearly written and well-structured. The roles of IFG and IFS are well explained, and diagrams help convey the pipeline effectively.\n\nWeaknesses\n1. Despite the title claiming a contribution to \"video understanding,\" the paper only evaluates on video classification, which is relatively simple and insufficient to validate the claimed benefits of IFG and IFS for spatio-temporal reasoning. It would be more convincing to test on video segmentation (e.g., instance/object/panoptic) or temporal tasks like action segmentation, detection, or localization.\n\n2. The IFG and IFS modules perform heavy information aggregation, which may lead to a loss of fine-grained spatial or temporal details. This could limit the method's effectiveness on tasks that require precise local reasoning, especially outside of classification.\n\n3. If the proposed approach is designed specifically for video classification, the title should reflect that scope more accurately. A clearer title would be:\n\"State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Classification.\n\nQUESTIONS:\n1. Have you evaluated the proposed method on more complex video understanding tasks beyond classification, such as temporal action segmentation or video object segmentation?\nWithout such evaluations, it is unclear whether the IFG and IFS modules truly benefit spatio-temporal reasoning as claimed.\n\nHow do you mitigate the risk of losing fine-grained spatial or temporal details due to the heavy aggregation performed by IFG and IFS?\nHave any ablation or visualization studies been conducted to confirm that key local information is preserved?\n\nGiven that the method is only applied to video classification, would you consider narrowing the paper’s scope or revising the title to better reflect its actual contributions?\nThis would help manage reader expectations and align the paper's claims with its experiments.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSSP is a PEFT method for Mamba-based video understanding models. It tackles the information decay in state space models (SSMs) through a \"gathering and spreading\" framework. This framework comprises Intra-Frame Gathering (IFG), which extracts salient spatial features and quantifies their importance via information entropy and spatial variance, and Inter-Frame Spreading (IFS), which uses this data to create global temporal prompts and gates temporal information based on spatial variance. SSP achieves competitive performance with minimal trainable parameters.\n\nSTRENGTHS AND WEAKNESSES:\n(+) The visualization of information decay in state space models is insightful and helps motivate the need for the proposed SSP framework.\n(+) The proposed method demonstrates strong performance across multiple video understanding benchmarks, showing its practical effectiveness.\n\n(-) While the paper states that the proposed modules are applied after the first Mamba layer, it remains unclear why this particular depth was chosen, and how the performance or parameter efficiency would change if the modules were applied at deeper layers. This is especially relevant for the IFG module, which computes entropy and spatial variance from intermediate representations — quantities that are likely to be sensitive to the semantic level of features. An ablation or discussion on the layer-wise placement of IFG/IFS would help clarify the design rationale and the robustness of the proposed method.\n(-) The comparison with STOP could be misleading, as the two methods are evaluated under different objectives—STOP adopts a text-video matching paradigm, whereas SSP is evaluated using a classification loss. To more fairly assess the effectiveness of the proposed method, it would be helpful to report results under the same matching-based setup as STOP.\n\nMinor\nFigure 2 shows the proposed modules being inserted after the i-th layer, while the main text states that they are applied after the first Mamba layer. For clarity and consistency, it would be better if the figure also reflected the first-layer placement.\n\nQUESTIONS:\nIn Table 2, the number of trainable parameters for SSP increases significantly with model size (e.g., from 0.98M in VideoMamba-S to 2.41M in -M), while STOP remains relatively stable despite also using attention and convolution layers. It would be helpful if the authors could provide a breakdown of where these additional parameters come from, and explain why the proposed method scales more heavily with model size compared to STOP.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a state space prompting (SSP), which is the prompt learning algorithm based on VideoMamba, proposed for video understanding. The proposed SSP consists of two core modules: an intra-frame gathering (IFG) module and an inter-frame spreading (IFS) module. IFG module generates the intra-frame prompts, which gathers the models’ attention to local features,  information entropy weights, and spatial variance measurements. IFS module generates the inter-frame prompts representing temporal inductive biases which spread the gathered temporal information across the frames. SSP only optimizes a few additional parameters and classification head during training. The extensive experimental results on various video understanding benchmarks demonstrate that the proposed SSP achieves better performances than conventional methods with small amounts of trainable parameters.\n\nSTRENGTHS AND WEAKNESSES:\n> **Strength**\n- This paper is well-written and easy to follow. \n- The proposed SSP appears to be technically sound. It also provides a mathematical analysis of the proposed algorithm’s long-range information transmission capability.\n- The proposed algorithm outperforms conventional algorithms on various benchmark tests. \n\n> **Weakness**\n- I don’t think there are any significant weaknesses in the proposed algorithm.\n\nQUESTIONS:\nI have carefully read the paper, but as I mentioned in the weaknesses section, I don’t think the proposed algorithm has any significant shortcomings. I believe this is a good paper that can make a meaningful contribution to the community. Nevertheless, I would like to offer a few suggestions below.\n\n---\n\n* In Tables 1 and 2, STOP shows significantly lower performances compared to other methods. It would be helpful to include additional explanation for these results. Although STOP is not specifically designed for VideoMamba, it is similar to SSP in that it is based on intra-frame and inter-frame prompts. Providing more insight into the reasons behind the significant performance gap would strengthen the discussion.\n\n* This paper provides experimental results on four popular datasets, including the large-scale SSv2. Nevertheless, including performance comparisons on additional benchmarks such as Kinetics-400 or LVU would further strengthen the effectiveness of the proposed SSP.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}