Mamba Only Glances Once (MOGO): A Lightweight Framework for Efficient Video Action Detection
Authors: Yunqing Liu
Nan Zhang
Fangjun Wang
Kengo Murata
Takuma Yamamoto
Osafumi Nakayama
Genta Suzuki
Zhiming Tan
Venue: NeurIPS 2025 poster ()
Keywords: Mamba
Lightweight Architecture
Video Action Detection
Encoder-Decoder
Computational Efficiency
Primary area: applications
TL;DR: We present the first pure Mamba-based architecture for video action detection, achieving Transformer-level performance with significantly reduced computation, inference time and memory costs.

Abstract: Mamba, a lightweight sequence modeling framework offering near-linear complexity, presents a promising alternative to Transformers. In this work, we introduce MOGO (Mamba Only Glances Once), an end-to-end framework for efficient video action detection built entirely on the Mamba architecture. In MOGO, our newly designed Mamba-based decoder can even use just one Mamba layer to effectively perform action detection. It uses neither Transformer structures nor RCNN-like methods for proposal detection. Our framework introduces two key innovations. First, we propose a pure Mamba-based encoder-decoder architecture. The encoder processes cross-frame video information, while the decoder incorporates two novel Mamba-based structures that leverage Mamba’s intrinsic capabilities to detect actions. Theoretical analysis and ablation experiments confirm their synergy and the necessity of each structure. Second, we design a video token construction mechanism to improve the model's performance. The token importance block can ensure that the retained token information is highly relevant to the predicted targets. These two innovations make MOGO both efficient and accurate, as demonstrated on the JHMDB and UCF101-24 benchmark datasets. Compared to SOTA action detection methods, MOGO achieves superior performance in terms of GFLOPs, model parameters, and inference speed (latency) with comparable detection precision. Additionally, it requires significantly less GPU memory than some SOTA token reconstruction methods. Code is available at https://github.com/YunqingLiu-ML/MOGO.

Mamba Only Glances Once (MOGO): A Lightweight Framework for Efficient Video Action Detection
Yunqing Liu', Nan Zhang', Fangjun Wang', Kengo Murata", Takuma Yamamoto", Osafumi Nakayama", Genta Suzuki², Zhiming Tan l+
Fujitsu R&D Center
Fujitsuu Research Japan
Abstract
Mamba, a lightweight sequence modeling framework offering near-linear complex- ity, presents promising alternative to Transformers. In this work, we introduce MOGO (Mamba Only Glances Once), an end-to-end framework for efficient video action detection built entirely on the Mamba architecture. In MOGO, our newly designed Mamba-basee decoder can even use just one Mamba layer to effectively perform action detection. It uses neither Transformer structures nor RCNN-like methods for proposal detection. Our framework introduces two key innovations. First, propose pure Mamba-based encoder-decoder architecture. The encoder processes tross-framd video information, while the decoder incorporates two novel Mamba-based structures that leverage Mamba's intrinsic capabilities detect actions. Theoretical analysis and ablation experiments confirm their synergy and the necessity of each structure. Second, we design a video token construction mechanism to improve the model's performance. The token importance block can ensure that the retained token information highly relevant to the predicted targets. These two innovations make MOGO both efficient and accurate, as demonstrated the JHMDB and UCF101-24 benchmark datasets. Compared to SOTA action detection methods, MOGO achieves superior performance in terms of GFLOPs,, model parameters, and inference speed (latency) with comparable detection pre- cision. Additionally, it requires significantly less GPU memory than some SOTA token reconstruction methods. Code is available at https://github.com hoom/mnqingLLiu MLJMOGO.
Introduction
The goal of video action detection is to localize and classify actions within video sequences, requir- ing models to capture spatiotemporal dependencies. Recently, Sia and Rawat [I] address this by introducing lightweight encoder-only model for open-vocabulary detection. Other advancements in this field, such as DETR [2] and TubeR [3], have relied on Transformer- based architectures. These models leverage the attention mechanism to model rang-range dependencies across video frames, achieving SOTA performance in capturing temporal context and action semantics. However, the self-attention mechanism incurs quadratic computational complexity with respect sequence length, making computationally expensive for long video sequences. For instance, the methods proposed in 4 and (5) achieved significant performance improvements but are reported to have relatively high environments.
GFLOPs. This inefficiency makes such models less practical for resource-constraingg
The Mamba framework [6] emerges promising alternative to Transformers. Unlike the attention- based paradigm, Mamba employs state-space modeling to achieve near-linear complexity, offering a scalable solution for processing spatiotemporal data. Recent adoptions, such VideoMamba [7] for video classification and MS-Temba [8] for temporal action localization, demonstrate Mamba's
Corresponding author; zhmtan fujitsu.com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

efficiency in handling sequential data. However, its potential remains underexplored in the context of video action detection.
To develop an efficient and effective framework for video action detection, we proposed purely Mamba-based framework. The motivation of our design detailed in Figure [ The Trans- former depends on self-attention and cross atten- tion capture the dependency relationships. In contrast, Mamba utilizes linear transformations project queries into target spaces and employs selective copying to efficiently model cross- sequence information. Building on this, we pro- pose MOGO (Mamba Only Glances Once), an end-to-end framework for efficient video action detection. Unlike SOTA methods that incorpo- rate external structures such as RCNN for region proposals 4 or Transformer components like VIT 19. MOGO eliminates the need for these additional modules. Our framework introduces two key innovations:
Output
Quadratic Complexity
Self-
Cross-
Transformer
Input
sequence
sequences
Mamba
Cross-Mamba
Linear Complexity
Output
Our NEWLY designed Mamba
Sequence
(1) Pure Mamba-based Architecture. Our de- signed decoder processes learnable queries and video information tokens through a streamlined pipeline EQ-Mamba, QVI-Mamba, and an FFN as shown in Figure 2 Ablation studies con- firm the synergistic necessity of each module.
Mamba
Sequence
Sequenie
$2. QVI WVI-Mamba
(2) Video Token Construction Mechanism. As shown in Figure 3 this module computes im- portance scores for encoder tokens across spa- tiotemporal frames. Coupled with an target- guided loss function, this mechanism reduces redundancy and enhances token relevance, fur- ther boosting efficiency.
Figure 1: Motivation of MOGO. Comparison between Transformer and Mamba complexity in- spires our framework design. By Mamba-izing the Transformer's (1) attention architecture, our framework enables (2) linear transformation for single-sequence modeling and (3) selective copy- ing mechanism for srose-sequence modeling.
The whole method's main strengths are compu- tational efficiency and a token-importance block, as demonstrated on the JHMDB [T] and UCF101- 24 [I] benchmark datasets. Compared SOTA action detection methods, MOGO achieves superior performance in terms of GFLOPs, model parameters, and inference speed (latency) with comparable detection precision. Additionally, it requires significantly less GPU memory than some SOTA token reconstruction methods.
2
Method
2.1
Overall MOGO Architecture
Our proposed MOGO framework introduces an end-to-end action detection pipeline built entirely on Mamba-based architecture, as illustrated in Figure ? Specifically, it detects actions on designated keyframe within a short video clip while using the remaining frames as temporal context. Because predictions made single keyframe, some non-keyframe tokens may be redundant weakly informative, which motivates selective token retention and importance modeling, as illustrated in Figure 3 Unlike SOTA methods that incorporate external structures such as RCNN for region proposals
4 or Transformer components like ViT (9]. MOGO leverages only Mamba's 's inherent capabilities, with primary goal of minimizing computational overhead while maintaining robust detection performance. The MOGO architecture comprises several key components, and the main elements include: Mamba encoder, Mamba decoder, token importance module, and loss function.
2.2 Pure Mamba-based Architecture
Our proposed method adopts an end-to-end purely Mamba-base architecture shown in Table ]

---

Fixed size (T-224-224)
[3.TH.W 2 [NN.C]
Frames within video clip
conv
Patch tokens
bboxes & labels
Mamba Encoder
Mamba Decoder
PE-processed video tokens
Hidden states
S1.EQ-Mamba
Eye-Queries (N-100)
Layer
ReaDrop
normaiization
$3.FFN ILinear transformatoon
Importance values
S2.QVI-Mamba
Loss
Layer nomalizatoo
Drap
Momory
Target-based token
Action Detection
importance block
Figure 2: Overview the proposed structure. The encoder processes PE-embedded video tokens through Mamba layers, accompanied by a based token importance block that highlights informative regions for action detection. The decoder then operates on two Mamba-based modules followed by linear layer. Illustration corresponds to batch size 1. (EQ-Mamba: Eye Query-basee Mamba, QVI-Mamba: Query- Information-fused Mamba, PE: Positional Embedding, ResDrop: Residual connection with Dropout, Drop: Dropout, K: Key frame.)
Encoder. Leveraging the pretrained models 77, our encoder is designed to extract comprehensive feature maps from input video frames. Following the patch embedding, which transforms the input clip into sequence patches, we incorporate spatial positional embeddings and temporal positional embeddings to model inter-frame dependencies across the video sequence. Then we incorporate bidirectional Mamba setup. The encoder comprises le layers, which evaluated through ablation studies in Section 3,4
Table 1: Tensor shape flow across the MOGO architecture. M contains keyframe tokens and important non keyframme tokens.
Stage
Tensor
Shape / Description
Imput
B,3,T, H, W]
Patch Embedding
Conv3D(x)
B. B.C, T, H' W"] where C = embed_dinn W
Flatten Rearrange
T.N,
Add Positional Embedding
+PE.+TPE
B.T.N,
Token Importance Block
MLP(x)
B.T.N.
Apply Mask
(MLP(())
B,T-N,C C
Mamha Encoder (depth m (_)
Block + ... Blocki,
B,T.N,
Mamba Decoder (depth
Block;
Blocki,
Decoder Queries
Q Embeedinn(N.))
B.
Devoder EQ Mambu
Ny. B.C
QVI-Mamba
Concan
M.B.CY B.
Devoder FFN
F
N B,C.
Prediction Heads
Linear( F*)
Ng. num classes +
Box: MLP(F)
B, No: (normalized)
Our decoder, depicted in Figure 2 uses the Mamba operator to process queries and integrate video information. Given an input sequence æ E RRdd the Mamba block computes its output y E Rxdd via a state-space model:
Decoder.
h = A h4-1 B
(I)
3/ C- hu
where A, B, E dgd are learnable transition and projection matrices, and h, is the hidden state at time step t. Compared with attention-based modules, this formulation enables linear-time complexity in sequence length L.
In the decoder design, Mamba serves two key functions: (1) linear transformation, and (2) feature extraction and sequence modeling based on selective copying (6). EQ-Mamba primarily uses the first function, while QVI-Mamba exploits the second. Additionally, the concatenation of query and video information the decoder draws inspiration from [7]. The decoder, consisting la layers,
evaluated through ablation studies in Section 3.4 and A.4 Specifically, EQ- RQ-Mamba operates solely

---

Input tokens
Importance logits map
Frame
Frome
Frens
Franse niis
Token importance block
Top-k Identification
Mamba-based Encoder
Keyframe tokens (Spatial features)
Non-keyframe tokens (Temporal features)
Mamba-based Decoder
Figure 3: Structure of the video token construction mechanism. This module enables the decoder obtain sufficient and effective information for action detection. Sufficient information ensured by incorporating the complete spatial features of the keyframe (prediction frame), while redundancy reduction is achieved by filtering encoder tokens through token importance block. Importance scores are computed for each token in the encoder, retaining only the top-k highest-scoring tokens to supply temporal information. These tokens are then combined with fixed keyframe memory portion, which preserves semantic context.
on the query embeddings, projecting and refining them within their own latent space. This process resembles self-attention its focus on intra-query relationships but relies on Mamba's space modeling instead of attention mechanisms, enabling efficient linear transformations of the query tokens. QVI-Mamba integrates the refined query embeddings with the encoder's video features (hidden states), concatenating them to create mixed representation that captures both query-specific information and video context. Finally, the FFN processes the output of the QVI-Mamba block, further transforming the fused representations predict bounding boxes and class probabilities.
The key difference between Mamba and Transformer architectures underpins our design choices. While Transformers serve attention and correlation mechanisms between tokens, Mamba relies on its unique mechanism of information storage and linear transformation via state-space dynamics.
2.3 Video Token Construction Mechanism
Prior work 4 shows that employing a token-importance mechanism improves performance over processing all tokens. In this work, optimize token utilization, introduce token importance module. This module selects the most relevant tokens from the encoder, which aggregates tokens from different spatiotemporal frames, enhance the decoder's ability capture salient features. In encoder, it employs an MLP to compute importance logits for each token, followed by a sigmoid activation to derive importance scores. should be noted that this part consists of learnable parameters, which dynamically adjusted based on the token importance loss. For details on the computation of the loss, refer Scction 2.4] The decoder's input tokens are derived by selecting high-scoring tokens based on their importance, resulting in a more informative and compact representation for processing, as shown in Figure
Specifically, we compute importance scores for each token in the encoder, and then guide the token selection process in the decoder. To preserve the features learned from the large-scale pretraining model, we do not reduce the number tokens in the encoder; instead, token importance serves only indicator at this stage. The importance scores directed by learnable mechanism. After calculating the importance scores, we perform token selection by picking the top A percentage of tokens with the highest importance scores for each frame, keeping only the most informative tokens for further processing in the decoder. To further enhance this process, we combine a fixed portion of the memory, which contains all the keyframe tokens, with the selected tokens. The keyframes are crucial as they contain the accurate semantic information for the current frame. Next, we describe this process in tensor notation. For a batch of size B, the encoder processes total T x N, tokens, where T is the frame number, and Ny is the number of tokens per frame. The token importance block assigns an importance score to each token (derived via the learnable MLP with sigmoid activation). Based top-k selection strategy (k is evaluated through ablation studies in Section 3.4) ) we retain []: N2J tokens per frame. These selected tokens, totaling N,- = []: (T Ny)J, are concatenated with the keyframe tokens (fixed at N, per keyframe).

---

2.4 Loss Functions
Since Mamba does not possess an intrinsic attention based importance mechanism as Transformers do, this section seeks to answer the question: what constitutes important relevant tokens? We do so under the premise that the L2 magnitude Our training objective comprises two parts:
token does not directly represent its actual importance.
Detection Loss (C)): We adopt a DtTe-style loss to supervise the action detection, which integrates three components: classification, bounding box position, and overlap, assigned weights of Wels Wbox. and Wovl, respectively.
Hungarian Matcher employed to align predictions with ground-truth targets. These weight values are ablated in our ablation experiments to optimize performance, as we not directly adopt coefficients from prior BETR-based works (e.g., 22, [2), given that our loss function is newly designed to suit the Mamba-based architecture. There's also weight for the no-object class.
Token Importance Loss (Cimportance): To learn token importance scores, we construct binary ground-truth mask y/ for each token in sample b, indicating whether the token's s center coordinates
(b) be the raw logit from the MLP for token in fall inside any ground-truth bounding box. Let a; sample We compute the binary cross-entropy loss:
B
N
C Linguortanee
1
log(o(a( ()))) + (1- wi²") log( o(a;
(2)
BN
b=
where B is the batch size, N is the total number of tokens per sample, and ((-) 1) the logistic sigmoid. Total Loss: The final loss combines the two objectives:
L= Ci X Limportance:
where À balances action detection performance and token importance learning, which is evaluated through ablation studies in Section 3.4
Mamba lacks an attention map, unlike Transformers, which use attention scores to represent token importance. Transformers motivated us to design token importance. The proposed token importance block is not an off fff-the-shelie plug-and-play module, but a learning-basee based, multi-stage system requiring end-to-end training. The key steps are:
(1) Encoder-side importance calculation. The encoder includes trainable MLP that outputs continuous importance value for each token, supervised by our custom loss.
(2) Token selection in the decoder. The learned importance scores used to select important tokens from the non-keyframes, which are concatenated with the keyframe tokens as the decoder input.
(3) Loss design. The overall loss combines standard action detection loss and token importance loss. The optimal loss ratio determined through ablation studies.
This system's 's effectiveness relies on the synergy of all its components and joint training.
3 Experiments
3.1
Experimental Setup
We evaluate our MOGO on three common datasets for video action detection: JHMDB I0 UCF101-
24 [I] and AVA [33. JHMDB contains 928 trimmed videos from 21 action classes. We follow the data annotation and procedure outlined in [14). UCF101-2 -24 comprises 3,207 videos spanning 24 sports classes. Following standard practice, we report performance on split- AVA a large-scale benchmark and contains 299 15-minute videos, divided into 21 1k training clips and 57k validation clips. The results for AVA are shown in Section We evaluate performance with mAP under an loU threshold of 0.5 on NVIDIA A40 GPUs. Other implementation details are shown in Section A.3

---

Pauener DiAibution (Pannn))
Computa Loed Dusiiution (OPLONS)
Table 2: Efficiency analysis of MOGO.
Module
#Param
GFLOPs
Encoder
73.996M
101
Query_embed embed
57.6K
Brooder 197 75)
Decoder
7.228M
2.297
13.271K
1.325e-3
Figure 4: Breakdown of parameter and com- putational distribution across components. Based on the pretrained encoder, only a small amount of extra computation is added while still achieving good performance.
Bhox _eo_e
0.667M
6.659e-2
Overall
81.962M
104
3.2 Efficiency Analysis
To assess the computational efficiency of MOGO, we measure the number of parameters, GFLOPs and inference speed and follow the code procedure in 77].
First, Table 2 details the parameters and GFLOPs for the model and its key modules. The entire MOGO model comprises 81.962 million parameters and incurs around 104 GFLOPs, reflecting a lightweight design compared to SOTA methods (see Section 3.3). The encoder dominates the compu- tational load, accounting for roughly 97.7% of the total FLOPs as shown in Figure as the encoder processes the 8-frame input using the pretrained Mamba-base backbone. In contrast, the decoder is significantly lighter, with 7.228M parameters and 2.297 GFLOPs, while auxiliary components like query query_embed embed (57. .6K parameters), class embed (13.271K parameters). and bbox embed (0.6 (0.667M parameters) contribute minimally to the overall computational cost. This means that, based on the pretrained encoder, we only added a small amount of extra computation but still achieving good precision performance.
Second, benchmark the inference speed using the throughput protocol defined in 4 with video clip length of frames. We run the model with 3 warm-up iterations and compute the average inference time over 10 additional runs. Our results show that MOGO achieves an average latency 3.9 ms/img, corresponding to throughput of 256 img/s. As shown in Table 3 this performance surpasses existing transformer-based methods: EVAD 4 (240 img/s VIT-B), WOO [5] (176 img/s on ViT-B, 147 on SF-R101), and TubeR (3] (64 img/s). While EVAD reports a peak throughput of 334 img/s under certain alternative configurations, our approach remains highly competitive, even though the current version of Mamba's CUDA/C+ kernels still has limitations in parallel processing relative to Transformer implementations.
3.3 Comparison with SOTA Methods
Table Comparison of MOGO with SOTA methods. GFLOPs are reported where available, Art.: Architecture, Thrp.: Throughput. C: CNN, T: Transformer, M: Mamba. dec.: decoder. Cv.: The original work only reported GFLOPs based on image-based RGB input, which has been converted here. J: JHMDB, U: UCF101-24.
(a) Comparison on efficiency.
(b) Comparison on performance.
Model
Art. Backbone
4Param
GELOPs
Thp.
Model
Backbone
[-mAP (J)
SF-RIOI-NL
EESSL 19
C
13D-CNN
64.4
69.9
MOC
DLA34
235.2
200
65.7
EVAD +
MaskFaat
16
T
VIT-L
MVITV2-L
185M (dec.)
218M
737
2828
153
SMT 2T
C
13D-CNN
K400
69.8
73.9
McMVIT 17
MVTTV2
52.6M
620
MOC
DLA34
70.8
AVA
13
C
13D-VGG
73.3
76.3
WOO
VIT-B
(bcad)
378
STAD 22
VIT-B
InternVid
61.4
71.6
WO0
SF-1101
314M (bead)
252
147
W00
SF-R101
K600
80.5
76.7*
EVAD 9
VTT-B
185M (dec.)
243
240
TubeR 3
13D
IGHK400
80.7
81.3
TubeR 8
CSN-152
240
64
(MOGO)
Mamba-B
Ours (MOGO) M
Mamnu-M
82M
102-104
measured by
We evaluate our MOGO model against SOTA methods two datasets, with results summarized in T Our approach achieves an mAP of 76.7 7 on JHMDB while maintaining a low computational cost of 102-1004 104 GFLOPs, leveraging an 8-frame Mamba-based backbone. Although our mAP is

---

slightly lower than lop-performing Transformer- based methods such as WOO (80.5) and TubeR (80.7), these models rely on heavier architectures (c.g.. ViT-B), which may increase computational overhead. In contrast, our model surpasses all CNN-based methods on JHMDB dataset, including I3D-CNN architectures like EESSK (64.4), and ACT (65.7), demonstrating the superior efficiency of Mamba-hased modeling over conventional convolution designs. Compared to WOO (SFR101) and TubeR (CSN-152), our MOGO model reduces GFLOPs by approximately 60% and 57%, respectively, while only sacrificing 3.8-4.0 mAP points (from 80.5 to 76.7 for WOO, and 80.7 to 76.7 for TubeR) on the JHMDB dataset. Compared to TubeR (I3D), our MOGO model only sacrifices 3.1 mAP points on the UCF101-24 dataset. Part of this gap likely reflects the availability of strong Transformer pretraining (e.g., IG+K400) leveraged by TubeR and other Transformer-base methods, whereas comparable large-scale pretrained checkpoints for Mamba are still scarce.
Comparison on GPU Memory Usage with EVAD []) We evaluate GPU memory consumption using the same settings as EVAD (an efficient video action detection method with significant performance improvements using Transformer), employing a sin- gle GPU with batch size of 8. Both methods adopt token reduction strategy, where tokens are removed according to predefined rules. The results in Figure demonstrate that: first, our method consistently con- sumes less GPU memory compared to EVAD. EVAD requires around 1.6 times more memory than our method, when token retention ratio is 0.4. Second, as the number of retained tokens increases, the GPU memory usage of EVAD grows significantly. In the extreme case where all tokens are retained, EVAD requires 2.8 times more memory than our method. Therefore, MOGO maintains lower and stable GPU memory footprints compared to EVAD. That means under the same hardware and experimental conditions, it can employ larger batch sizes to improve performance further.
DMD
14.2 MosD
5
105
Token Roturttr Patto
Figure 5: Comparison of GPU memory us- age between our method and EVAD under different token retention ratios.
3.4 Ablation Studies
Raw Mamba
Feature
Improved Feature
Figure 6: Qualitative ablation: visualization of the impact of the token importance mechanism. Row I shows raw Mamba outputs without the loss guidance. Row 2 shows outputs with our proposed importance modeling. Tokens inside target regions are more emphasized.
Qualitative Ablation, Unlike Transformers, which rely on attention mechanisms to compute token- to-token interactions, Mamba encodes temporal dependencies implicitly through sequential modeling. This makes it challenging to directly interpret token importance. As shown in the first row of Figure 6 raw Mamba outputs tend to produce scattered and less structured token representations. To address this, we propose a heuristic loss function (see Sec. 2.4). which encourages tokens inside the ground truth bounding boxes receive higher importance. Furthermore, avoid introducing significant computational overhead, we insert lightweight MLP directly before the encoder tokens as shown in Table T This MLP outputs token-wise importance score, which is then multiplied element-wise with the encoder tokens. With this, as shown in the second row of Figure 6 the token activations become more focused on semantically meaningful regions. More examples are provided in Figure [S4]
Quantitative Ablation. The following ablation experiments are conducted on the JHMDB dataset. The findings are summarized in Table
Pretrained models. Table 4 (a) presents the results of ablation experiments conducted on pretrained models. All models employ Mamba-based middle-sized model as the backbone, pretrained on
7

---

Table 4: Quantitative ablation: experiments results. (a) Pretrained models. Using different pre- trained models as the encoder. MFT: mask with fine-tuning, MPT: mask with pretraining. (b) Encoder depth, enc.: encoder. (c) Decoder components. The EQ-Mamba and QVI-Mamba components are critical design elements. These components are evaluated by removing each one individually to assess their impact on the model's performance. dec.: decoder, md.: model, rmd.: removed. (d) Decoder input: temporal info. In the decoder, the top k% of tokens selected and integrated with keyframe tokens. (c) Decoder input: keyframe info. Ex.: Exchange token positions of keyframe and temporal info. (f) Query number. (g) Ratio of total loss.
(a) Pretrained models.
(b) Encoder depth.
(c) Decoder components.
Model
Baseline
MFT
MPT
mAP
enc.GFLOPs (6param)
Case
dec dec.GFL.OP.
md.GFLOP:
mAP
51.7
65.0
31.2
24
60.4
76 (55.721M)
EQ mn.
2.098
103
32
66.2
101 (73.996M)
QVI md.
0.466
102
61.1
(92.271M)
Proposed
2.297
104
(d) Decoder input: temporal info.
(c) Decoder input: keyframe info.
k mAP
dec. (e..GFL.O)))Paramm)
md.GFLOPs
Case
Removed
Ex.
Proposed
5 66.4
1.198 (7.228M)
103
mAP
53.7
53.9
69.0
10 66.8
1.357 (7.228M)
103
20
67.1
1.676 (7.228M)
103
( Query number.
(g) Ratio total loss.
40
67.4
2.297 (7.228M)
104
Case
50
100
200
2
0.1
0.5
2
50
66.5
2.616 (7.228M)
104
70
65.6
3,237 (7.228M)
105
mAP
58.5
66.0
61.3
MAP
73,3
75.6 74.4
74.4
the K400 dataset 23 with a resolution of 224 The baseline model achieves mAP of 51.7. Introducing a mask with fine-tuning significantly boosts the mAP to 65.0. In contrast, using a mask with pretraining yields a decrease, achieving an mAP of 31.2. These results suggest that fine-tuning with masking is more effective than pretraining, likely due to better adaptation to the target task. Further discussion about pretrained models can be found in Section 4
Encoder depth. The results in Table (b) indicate that using 32 layers in the encoder achieves the highest mAP of 66.2, with an encoder computational cost of 101 GFLOPs (73.996M parameters) and a model-level cost of 104 GFLOPs. Increasing the depth 40 layers results in performance degradation (mAP 61.1) despite a higher computational cost of 126 GFLOPs (92.271M parameters), likely due to overfitting. Therefore, we set the encoder depth 32 as the default.
Decoder components. Table 4 (c) investigates the decoder structure by ablating key compo- nents, EQ-Mamba (EQ) and QVI-Mamba (QVI). and comparing them against the proposed de- sign. Removing EQ-Mamba or QVI-Mamba reduces the mAP significantly (possible theo- retical reason is that removing EQ is equiva- lent to a random query directly entering Mamba, while removing QVI is equivalent to using in- adequate video information). although decoder GFLOPs drops slightly. This demonstrates that every component of our design is indispensable but lightweight, further validating the correct- ness of our extrapolation from Transformer to Mamba as depicted in Figure
Decoddrr RRLOR
67.0
3.0
86.5
88.0
200
0.5
1.5
12
20
40
50
Figure 7: The trends of mAP and decoder GFLOPs with varying numbers of retained non- keyframe tokens. It demonstrates the importance of video feature tokens from non-keyframes as temporal information. We evaluate the impact of selecting the top 1% of tokens.
Decoder input: temporal information. Table 4 (d) evaluates the impact of selecting the top 1% of non-keyframe tokens in the decoder, which are integrated with keyframe tokens for process- ing. As le increases from 5 to 70, the mAP im- proves from 66.4 to peak of 67.4 at k=40, while decoder GFLOPs rise from 1.198 to 3.237, with the number of parameters remaining constant at 7.228M and model GFLOPs slightly increasing from 103 to 105. Beyond k=40, the mAp declines to 66.5 at 55 and further to 65.6 at k=70, despite higher computational costs. This indicates that 4=40 strikes an optimal balance between performance and efficiency, making it the most effective choice for this strategy. Figure 7 demonstrates the importance
8

---

of video information from frames other than the keyframe temporal information. Intuitively, this drop when k>40 occurs because & controls only the retention of non-keyframe tokens. We keep all keyframe tokens and perform detection on the keyframe, while non-keyframes serve primarily as contextual support. As increases, more low-utility background tokens from non-keyframes are admitted, which can dilute salient cues and slightly reduce performance.
Decoder input: keyframe information. Table (e) compares different configurations of keyframe information usage. The one with keyframe information removed achieves an mAP of 53.7, while exchanging the token positions of the keyframe information and the temporal information (Ex.) slightly improves the mAP to 53.9. In contrast, the initial arrangement outperforms both alternatives with an mAP of 69.0. This performance gap demonstrates that the proposed design, leveraging the full potential of keyframe information, is the most effective approach for video token construction.
Query number. Table 4 (f) evaluates the effect of varying the number of queries model performance. This limitation of using a fixed number of queries is discussed in Section
Ratio of total loss. Due to our redesign of the loss function, the relationships between the different components have also changed, making the final results sensitive to the choice of weights. Therefore, re-evaluation of the weight configurations is necessary to optimize performance. The findings are summarized in Table 4 (g). with parameters defined in Section 2.4, For the total loss ratio, adjusting À reveals that 0.5 is the optimal choice, achieving the highest mAP of 75.6.
Discussion
First, we discuss innovation 1: pure Mamba- based architecture:
Extension to multi-label detection. MOGO originally designed for end-to-end detection, while AVA involves multi-label prediction per bounding box. To further evaluate adaptability, we modified the existing decoder to support multiple labels per bounding box without introducing an explicit classification branch. Corresponding adjustments were made to the loss and training setup. On the AVAv2.2 validation set, the adapted MOGO achieved an mAP of 16.2 (at step 26). This shows that MOGO can extend to multi-label settings with competitive accuracy. Training logs are presented in Figure S6 Another Mamba-based work [24] has reported their evaluations; however, a fair comparison would require additional details (c.g., architectural specifications and FLOPs calculation).
Comparison with FlashAttention. This comparison assesses how an optimized Transformer baseline performs relative to Mamba. We implemented decoder variant that replaces the Mamba blocks with FlashAttention 2.8.2 (non-causal). The modified decoder reduces parameters from 7.228M to 5.321M and GFLOPs from 2.297 to 1.116, indicating higher computational efficiency. Trained on JHMDB with 4 A40 GPUs for 50 epochs and a batch size of 32, it achieves an mAP of 70.1 (Initial loss: 7.7654, grad norm: 14. 1749 - Final loss: 0.4765, grad norm: 1.8073). While FltshAttention
offers higher efficiency, its mAP does not surpass our Namba-based decoder. However, this motivates improving Mamba's 's low-level C++/CUDA kernels.
Performance on longer video sequences and varying pretraining models. We conducted experiments on longer video sequences using the UCF101- 224 dataset. For each setup, we employed a corresponding pre-trained encoder from 7 and adjusted the decoder accordingly.
Table 5: Performance on longer video sequences.
Ex. Frames
Pretrained Model
Batch Size
mAP
1 16
K400
16
69.50
2
Breakfistacctoos--aaaatt
41.90
3. 64
K400
63,43
All experiments were trained for 30 epochs on 2 NVIDIA A40 GPUs. The configurations and results
are summarized in Table 5 These results indicate that our method maintains good performance on longer video sequences. Morcover, we observed that the choice of pre-traince model plays a crucial role: using K400 pretraining yields higher accuracy than using the Breakfast dataset [25) To further investigate this factor, we trained with a new encoder pretrained on the SSV2 dataset (26 (2 A40 GPUs, 50 epochs, batch size 30) and obtained an f-mAP of only 64.1 on JHMDB (Initial loss: 7.9834, grad norm: 67.6360 + Final loss: 0.4847, grad norm: 6.6854). The results confirm that K400 pretraining remains superior to SSV2 in our setting.
Training with long frames and autoregressive tracking. Autoregressive trackers such as Track- Former (27] propagate track queries frame-by-frane to jointly detect and track objects across long videos without fixing global clip length, illustrating an alternative way to scale beyond 64 frames. In contrast, our method currently relies publicly available pretrained encoders [7], whose checkpoints

---

are trained for inputs up to 64 frames. Since action-detectonn detectinnnnnnnn mAP is highly sensitive to the encoder's s pretraining quality and capacity, this limitation arises from data and model availability rather than the framework itself. In principle, our pipeline can process much longer sequences.
Bidirectional scan in the encoder. Our encoder employs a bidirectional scan. In Mamba, this means that at each position forward state summarizes all preceding tokens and a backward state summarizes all following tokens; the token representation is then obtained by fusing these two states. As result, each token can store richer information.
Next, we discuss Innovation 2: video token construction mechanism:
Rethinking token selection: RLT-based and random token selection. Inspired by Run-Length Tokenization (RLT)) [28], we replace our original decoder-side token selection with two key com- ponents: (i) token selection, which prunes temporally redundant tokens while retaining tokens that change semantically across frames; and (ii) run-length embedding, which encodes how long a selected token remains stable and injects this temporal continuity into the token features before decoding. On JHMDB (4 GPUs, 60 epochs, batch size 32), this variant reaches an f-mAP of 72.1 (Initial loss: 8.4603, grad norm: 143.7739 - Final loss: 0,4385, grad norm: 2.4390). While slightly below our best MOGO configuration, these results indicate that combining RLT-style token selection with run-length information is a promising direction for further improvement. The above is heuristic attempt. For a stochastic baseline, we removed importance logits and randomly retained a fixed proportion of tokens per frame (40%), keeping all other settings identical to Figure 7 Trained for 30 epochs, this andom-selection variant achieved an f-mAP of 67.2 (Initial loss: 7.9832 - Final loss: 0.6982). Under the same environment and retention ratio, our pnportance-guider selection attains 67.4 mAP, indicating consistent gain over random sampling.
Generalizing the coken-importannce block to other methods. To broaden applicability beyond our own architecture, we integrate the token-importance module into TubeR (3) and evaluate on JHMDB, comparing the standard full-token setting with variants that retain only a subset of non-keyframe tokens. Using the released JHMDB -pretraiid CSN-152 checkpoint, the detector attains 71.2 mAP after one epoch. We therefore adopt the pretrained setting as the full-token baseline. With our importance scores guiding token reduction, keeping 80% of non-keyframe tokens yields 72.05 mAP, and keeping 40% yields 72.0 mAP. Note that while the Transformer in TubeR inherently possesses an attention-based token importance mechanism, we align its setup with our proposed method by using the outputs of the trainable MLP to represent token importance within the encoder. The intermediate frames are also designated key frames. For the loss, however, we adhere the original TubeR implementation. These results are on par with the full-token baseline in small-scale experiments.
Seyframe-centrie detection without an action-switch head. Our approach does not employ an explicit action-switch head; instead, it follows DETR- style formulation with Hungarian matching over class logits and bounding box predictions. Conceptually, the method can be viewed as image- style action detection: we predict on a designated keyframe (typically the middle frame) while importing contextual tokens from preceding and following frames. This design helps the model sense how a person's action shifts across time, but it does not explicitly model precise action boundaries subtle temporal transitions.
5 Conclusion and Future Work
Our proposed MOGO framework introduces a pure Saaba-basedd architecture for end-to-end video action detection, eliminating reliance on Transformer components. This design addresses limitations noted in prior work, such as the complexity overhead in (4) by leveraging Mamba's state-space modeling for efficient processing of video sequences. This results in significant reduction in computation while maintaining competitive precision. Additionally, our video token construction mechanism can obtain important token information across spatiotemporal frames. In this way, important cues (closely related to the action prediction) can be effectively retained.
MOGO's performance is among the top-tier the very best ones. This limitation stems from two factors: first, GFLOPs reduction may trade off some precision; second, the scarcity of pre-trained Mamba models limits feature representation quality, unlike the extensive pretraining pools available for Transformers. A further limitation the use of fixed query; in its current form, the model cannot reliably handle scenes with more than 100 people. However, this study highlights Mamba's 's promise in this new field, Future work will focus on developing richer pretrained models to help enhance performance.
maansoomer-basedd methods, though it has not surpassed
10

---

Acknowledgments and Disclosure of Funding
Funding:
This work received no thirt-party funding or support. Competing Interests: The authors declare no competing interests.
References
[1] Zhen Hao Sia and Yogesh Singh Rawat. Scaling open-vocabulary action detection. arXiv preprint arXiv:2504.03009, 2025.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[3] Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, Bing Shuai, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, Ivan Marsic, Cees G. M. Snoek, and Joseph Tighe. TubeR: Tubelet transformer for video action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[4] Lei Chen, Zhan Tong, Yibing Song, Gangshan Wu, and Limin Wang. Efficient video ac- tion detection with token dropout and context refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[5] Shoufa Chen, Peize Sun, Enze Xie, Chongjian Ge, Jiannan Wu, Lan Ma, Jiajun Shen, and Ping Luo. Watch only once: An end-to-end video action detection framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[6] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 2312:22312.07722, 2023.
[7] Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding, arXiv preprint arXiv:2403.06977, 2024.
[8] Arkaprava Sinha, Monish Soundar Raj, Pu Wang, Ahmed Helmy, and Srijan Das. MS- Temba: Multi-scale temporal mamba for efficient temporal action detection. arXiv preprint M.Xiv:2501.06/30 2025.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[10] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J. Black. Towards understanding action recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013.
[11] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv:1212.0402,, 2012.
[12] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional DETR for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[13] Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick. Caroline Pantofaru, Yeqing Li, Sudheen- dra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: A video dataset of spatio-te localized atomic visual actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[14) Yixuan Li, Zixu Wang, Limin Wang, and Gangshan Wu. Actions as moving points. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
11

---

[15] Christoph Feichtenhofer, Haoqi Fan. Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[16] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[17] Chao- Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. MeMViT: Memory-augmente multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[18] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems (NeurIPS 2022), 2022.
[19] Akash Kumar and Yogesh Singh Rawat. End-to-end semi-supervised learning for video action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[20] Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari, and Cordelia Schmid. Action tubelet detector for ppatio-temporal action localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
[21] Akash Kumar, Sirshapan Mitra, and Yogesh Singh Rawat. Stable mean teacher for semi- supervised video action detection. arXiv preprint arXiv:2412.07072, 2024.
[22] Tao Wu, Shuqiu Ge, Jie Qin, Gangshan Wu, and Limin Wang. Open-vocabulary spatio-temporal action detection. arxiv preprint arXiv:2405. 10832, 2024.
[23] Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv preprint arXiv: 2017.
[24] Jinyoung Park, Hee-Seon Kim, Kangwook Ko, Minbeom Kim, and Changick Kim. Video- mamba: Spatio-temporal selective state space model. In Proceedings of the European Confer- ence on Computer Vision (ECCV), 2024.
[25] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[26] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEÉE International Conference on Computer Vision (ICCV), 2017.
[27] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[28) Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niiuuma, Kris M. Kitani, and László A. Jeni. Don't look twice: Faster video transformers with run-length tokenization. Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), 2024.
[29] Lei Chen, Zhan Tong, Yibing Song, Gangshan Wu, and Limin Wang. CycleACR: Cycle modeling actor-context relations for video action detection, arXiv preprint 1681 2023.
[30] Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
12

---

[31] Hui Lu, Albert Ali Salah, and Ronald Poppe. Snakes and ladders: Two steps up for videomamba. arXiv preprint 1xXY::2406.19006 2024.
[32] Ali Hatamizadeh and Jan Kautz. Mamba Vision: hybrid mamba-transformerr vision backbone. arXiv preprini arXiv:2407.08083, 2024.
[33] Soumyabrata Chaudhuri and Saumik Bhattacharya. Simba: Mamba augmented u-shiftgen for skeletal action recognition in videos. arXiv preprint arXiv:2404.07645, 2024.
[34] Xiuwei Chen, Wentao Hu, Xiao Dong, Sihao Lin, Zisheng Chen, Meng Cao, Yina Zhuang, Jianhua Han, Hang Xu, and Xiaodan Liang. Transmamba: Fast universal architecture adaption from transformers to mamba. arXiv preprint arXiv:2502.151333 2025.
[35] Yuhui Lin, Jiaxuan Lu, Yue Yong, and Jiahao Zhang. Mv-gmn: State space model for multi-view action recognition, arXiv preprint arXiv:2501. 13829,, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They state the main contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses key limitations in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
This paper provides the full set of assumptions and complete proof in Section[[]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide full implementation details in Section 2 and Section
A.3
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The paper provides open access to its code in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com / cassiicnncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All experimental settings are detailed Section 3 and Section A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For latency evaluation, report the average inference time over multiple runs to account for runtime variability, as described in Section 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the GPU type and batch size used for inference speed evaluation in Section 3 and Section A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As foundational work introducing a lightweight Mamba-based architec- ture for video action detection, the paper's primary positive impact lies in reducing the computational cost and memory usage as described in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of any high-risk models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all datasets and pretrained models used in Section S All datasets are publicly available and used in accordance with their licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: Algorithms and experiments are well documented Section 2 Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve any human participants or data derived from human subjects, thus IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our research does not involve large language models as core component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: httt:paa/eearrrssscc////o/eeeeee for what should or should not be described.
20

---

A Technical Appendices and Supplementary Material
A.1 Why Mamba, not Transformers?
We argue that the proposed Mamba-base framework is well-suited for video action detection due to the following reasons:
I Linear Complexity. Transformers rely on self-attention to model long-range dependencies, resulting in quadratic complexity with respect to sequence length. Specifically, for sequence of length I. and feature dimension d, the computational cost of self-attention
Cransfoasfoomee = d).
(S1)
In contrast, the Mamba block updates hidden states via a linear state space model:
Cost Mamba = (LL)) O(D).
(S2)
This is critical for video sequences in real-time or resource-constrainee settings.
Causal Modeling, Equation I models causal dependencies. Each output depends only on the previous hidden state and current input. This aligns well with the nature of action detection, which relies on the context of preceding and current frames. (However, Mamba may not be well-suited for non-causal tasks such as image restoration, which require access to future tokens.) By contrast, models perform undifferentiated attention over a large number of previous,
general Transformer-basee current and future information tokens, which can introduce redundancy and noise. Mamba, therefore, offers more efficient and focused modeling paradigm for video-based tasks. This is essentially a coarse-to-fine design. While Transformer attention is detail-oriented (attention), Mamba views the input-output process as holistic dynamic system. This allows to design global optimization strategies. From a theoretical perspective, it supports more principled modeling than heuristically stacking attention layers.
However, Mamba lacks explicit attention mechanism. This why we introduce minimalistic linear projection (MLP) layer coupled with purposefully designed loss function to simulate attention- like behavior. This enhancement enables the model to focus on important tokens while preserving the framework performance.
A.2 Related Works
Transformer-Base Action Detection. Recent Transformer- based methods often employ a two-stage pipeline, combining 2D backbones for actor localization with 3D backbones for temporal context extraction. For instance, CycleACR (29) uses Transformer model actor-context relations, enhanc- ing detection through cyclic consistency across frames. Query- based approaches like TubeR (3) and WOO [5] build on DETR (2). predicting action tubes and categories with the quadratic complexity of attention mechanisms. In VAT (30, spatiotemporal features are aggregated around actors. Despite these advances, the computational burden of Transformers remains a challenge, especially for long sequences, motivating exploration of efficient alternatives.
Mamba- Applications. The Mamba architecture (6), with its linear-time state-space model- ing, has emerged as lightweight alternative to Transformers. VideoManba 77 adapts Mamba for video classification, achieving competitive top-1 accuracy on K400 by leveraging selective state-space mechanisms to model temporal dependencies efficiently. VideoMambaPro [31 further enhances
this by addressing historical decay and clement contradiction. MS-Temba [8] extends Mamba to multi-scale temporal localization on datasets like MuliTHUMOS. However, while Mamba excels in classification and localization, its application to video action detection remains underexplored, providing gap that our MOGO framework addresses with pure Mamba-basee end-to-end solution.
Hybrid Approaches. Mamba has also been explored in hybrid models like Mamba Vision B32 work combining Mamba and Transformer for vision tasks, and Simba [3) which augments Mamba with graph networks for skeletal action recognition. Recently, Tamba 34 introduced hybrid Merafformr-Mambhee backbone that adapts attention mechanisms for faster inference while preserving detection accuracy. Similarly, MV-GMN (5] combines rule-based and KNN-based methods with state-space models to enhance robust action recognition.
21

---

A.3 Implementation Details
Data Processing. We decode raw videos using the Decord backend and uniformly sample 8 frames for training and evaluation. The input frames are resized and cropped to 224 resolution. To construct training and testing samples, we rely on the provided split lists from relative datasets, and retain only clips that have valid annotation files. The underrepresented classes were balanced. For data augmentation during training, we include random short-side scale jittering ((256, 320D), random horizontal flipping, and optional color jittering (PCA- lighting augmentation). During testing, center cropping is applied after resizing the shorter side to 256. All frames are normalized using the mean and standard deviation values, respectively. Bounding boxes are clipped to image boundaries to avoid numerical instability. Per-frame annotations are matched using the keyframe index of each sampled clip, and annotations are encoded as bounding boxes and class indices for downstream use.
Training Engine. All experiments are conducted using PyTorch with mixed-precisionn training forrhh.cuda.amy aut.aaaa..sss......mmmm......t.. Gradient clipping is applied to stabilize training. Both the learning rate and weight decay are scheduled using precomputed cosine decay curves and updated at every step. We adopt the Adam W optimizer with betas set to (0.9, 0.999), weight decay of 0.05, and an initial learning rate of le-4. The learning rate is scaled according to the effective batch size and follows a cosine annealing schedule. Training is performed for 50 epochs, with a 5-epoch warmup phase. Since the designed model has an advantage in GPU memory usage, we recommend using a larger batch size for training (230). We use batch size of 30 per GPU and set the update frequency to 1. Training is distributed using forch.distributed. with full synchronization across processes. The total loss consists of standard object detection loss, including classification, bounding box regression, and GloU terms, along with a token importance loss. The latter encourages aligmment between the model-predicted token importance and the ground-trute spatial token relevance derived from bounding box annotations, computed using binary cross-entropy. (Specifically, we first generate normalized spatial coordinates for all tokens in each frame. Given H x W spatial patches and T temporal frames, this produces a tensor of shape [T x HW, 2]. These are compared with all bounding boxes to obtain binary inclusion map.) Evaluation is done using torchmetrics with COCO-style AP. For each sample, we extract predicted logits and bounding boxes, apply softmax to logits, and select the top score per query. Each training batch consists of B video clips, where each clip is represented as a tensor of shape [3, T, H, W). Annotations are processed into lists of dicts with keys boxes and labels, compatible with TorchMetries and detection criteria. We log training loss, learning rate, gradient norm, and validation mAp at each step using custom logger. We use torchrun to launch distributed training with synchronized logging and gradient updates across GPUs.
Model. The core architecture consists of Mamba-base spatiotemporal encoder and a lightweight decoder. The video encoder is designed based the Mamba sequence modeling block. Following standard video modeling practices, we adopt a 3D convolutional layer to embed T-frame video clips into patch tokens of shape [B, T N, DJ. where N is the number of spatial patches per frame. Each patch is then projected using a temporal tubelet embedding (kernel size = 1). Spatial and temporal positional embeddings are added separately. The token features are further enhanced by stack of Le Mamba-baan based encoder blocks, each composed of a selective copy mechanism and RMSNorm. To improve efficiency and enable token reduction, we introduce a simple yet effective module that estimates the importance of cach token via an MLP. This module predicts scalar score for each token, and soft masks are applied before the encoder layers. For the decoder, a learnable query embedding of shape [100, D) is provided input, and an la-layer Mamba decoder processes over the encoded memory tokens. The final outputs are passed through a classification head and a bounding box regression head (MLP with sigmoid activation). We adopt RMSNorm as the default normalization layer. The encoder is initialized from a pretrained checkpoint 7 on a large-scale masked video modeling task.
A.4 Ablation Studies- Supplementary
Table S1: Quantitative ablation (supplementary); experiments results. (a) Decoder depth. (b)
Weights of detection loss. w= Whox: Worl
(a) Decoder depth.
(b) Weights of detection loss.
4
6
11:1:1
5:1:1
1:1:1
1:5:2
2:3:2
mAP
75.6
71.7
32.0
mAp
67.2
68.6
46.3
4.2
35.6
22

---

Decoder depth. Table SS a) shows that -llayer decoder gives the best mAP (75.6), while or 6 layers degrade performance. likely due to overfitting. We adopt single-layer decoder by default. Therefore, MOGO (Mamba Only Glances Once), i.e., our Mamba-ban based decoder that glances only once. For the corresponding training dynamics, please refer to Figure ST
14 6
30-
Figure S1: Training curves and diagnostics. Training loss (right axis) and gradient norm (left axis) for decoder depths la 1.
Weights of detection loss. This set of experiments was conducted with parameter (Table 4 fixed. For the detection loss (Table ST],, emphasizes the classification weight (war) proves beneficial, with ratios of 5:1:1 1 and 11:1: yielding strong mAP scores of 68.6 and 67.2, respectively, compared to lower performance at other settings (c.g., 46.3 at 1:1:1). This suggests that a larger Wets enhances detection accuracy. We note that the 1:5:2 configuration performs poorly; controlled re-runs confirm this trend. When using 1:5:2 weights, observed high loss and unstable grad norms (400 after 20 epochs), while mAP stayed under 0.05. In contrast, the 5:1:1 configuration yields stable optimization dynamics: the loss decreases steadily (epoch[0):8.00 [ooch(29]:0.80 gradient norms consistently drop after 10 epochs, and the mAP rises above 0.6. So now have enough evidence to say that the 1:5:2 setting may overwhelm optimization with box/ovl weights, preventing the model from learning effective action features.
59x - epoch[29|:2.x,
A.5 Video-level Results
Video-Level Detection mAP. We provide the video-mAP results on JHMDB and LCCF101-24 datasets. As shown in Table Table and Table S$4 our method achieves lower video-level mAP compared to WOO while outperforming both WOO and TubeR in terms of model size, GFLOPs, and throughput. Because our method detects actions on a designated keyframe within each clip (e.g., the middle frame in an 8-frame snippet), using the remaining frames as temporal context, frame-level metrics such as f-mAP@0.5 are more appropriate measure. contrast, video-level mAP emphasizes constructing continuous tubes over the entire clip, which may understate the strengths of our keyframe formulation.
Table S2: Per-class AP on JHMDB under video-level detection evaluation. (IoU: 0.2)
Class
brash_hair
cutch
clap
climb stmirsstirss
golf
jump
kilk_ball
pick
pullup
push
AP
0.833
0.553
0,743
0.725
1,000
0.437
0.791
0,813
1.000
0.937
0.976
Class
run
shootballl
sboot_bow
sboot_gun
sit
stand
swing_ baseball
throw
walk
wave
AP
0.486
0.450
1.000
0,870
0.297
0.421
0.667
0.181
0.576
0.465
Table S3: Per-class AP on UCF101-24 under video-level detection evaluation. (loU: 0.2)
Basketalll
BackethullDonk
Biking
ClifDivingg
CrickefBewwing
Diving
Fencing
Floeffymmatiic
AP
0.586
0,519
0.506
0.757
0.404
0.974
0.767
0.981
Class
GoltSwing
HasseMiding
ReDDancnn
Longlump
PoleVault
Rop:Clinhing
SalsaSpin
SkateBoardngg
0.886
0.440
0.227
0,864
0.928
0,884
0.447
31993
Class
Skiing
Skijet
Soeceilaggging
Surfing
TowisSwiin
Trampolinehanping
VolleyhallSSpiking
WWlkingWihhDog
1.000
0:913
0.494
0.189
0.817
Table S4: Comparison of video-level detection performance and model efficiency.
Method
Vidoo-aAPPel (U)
Video-AAPoro (J)
Video-mAPPro.. (U)
Video-mAPieu.S
#Params
GHLOPS
Throoghpur
WOO
74.4
314M
252-378
147-176
TubeR (L3))
Ours
70,8
67.9
39.4
42.0
82M
102-104
256
23

---

Video-Level Classification mAP. We further provide the video-level classification mAP results on the JHMDB and UCF101-24 datasets. We conduct this experiment because the dataset provides video-level category labels. Therefore, we aim to evaluate the model's ability to extend frame- level predictions. Our method follows dldding-widdww inference strategy to compute video-level predictions. Given an input video, we first extract RGB frames and apply standard preprocessing including resizing, normalization, and batching. The video is divided into clips, and each clip is fed into the model to obtain per-query logits. We compute the average of softmaxed logits (excluding the background class) across all queries and clips. Finally, the class probabilities are aggregated over the video to produce the final prediction. The performance is measured by per-class AP and the overall mean AP. On the JHMDB dataset, as shown in Table S5 our model achieves strong results across most categories, particularly excelling in actions such golf. pullup, push, and shoot bow. Overall, our approach attains a video-level mAP of 0.812, demonstrating robust performance on this benchmark.
Table S5: Per-class AP on JHMDB under video-level classification evaluation.
Class
brash_hair
catch
clap
climb sttir_s
golf
jump
kick_ball
pick
pullup
push
AP
0,932
0.711
0,908
0.981
1,000
0.811
0.986
0.870
1.000
1.000
1.000
Class
run
shoot_blll
shoot_bow
shoot_gun
sit
stand
swing_ buseball
throw
walk
wave
AP
0.560
0.555
1.000
0.951
0.378
0.486
0.996
0.560
0.751
0.616
On the UCF101-24 dataset, it achieves an mAP of 0.971. This result demonstrates that our model possesses strong classification capabilities at the video level without considering bounding box predictions.
Table S6: Per-class AP UCF101-24 under video-level classification evaluation.
Class
Baskethall
BaskethallDDank
Biking
ClinDivang
CrickeBBowling
Diving
Fescing
Ploxtiymaatics
AP
0.927
1,000
0.999
0.991
0.902
1.000
1.000
Class
GollSwing
Honseliding
Reelanning
Longlump
PoleVault
RopeClinhing
SalvaSpin
SkateBoardngg
0,929
1.000
1.000
0.972
Skinng
Skijet
Soccecluggling
Surting
TennisSwing
Trampolinc/unping
VoileyballSpikknn
WalkingWWhDDen
AP
0,980
0.974
0.851
1,000
0.981
1.000
0.996
0.996
A.6
Result Analysis and Visualization
Per-Class Metric Visualization. To further investigate the importance of keyframe information in our framework, conduct study comparing three variants: (1) Ours, the full model using both selected tokens and the fixed keyframe segment; (2) Exchange, where the keyframe segment is exchanged with non-keyframe tokens; and (3) Remove, where the keyframe segment is discarded altogether. As shown in Figure S2 our method consistently outperforms the baselines across most action classes, indicating that preserving keyframe information is crucial for accurate action localization. In particular, categories such as golf, shoot boo, and pullup benefit significantly from retaining discriminative keyframe cues.
01
94
04
01
Figure $2: Per-class AP(0.5:0.95) under different keyframe strategies on the JHMDB dataset. Ours uses both the preserved keyframe and selected tokens. Exchange swaps keyframe tokens with other temporal tokens. Remove drops keyframe information completely. Preserving keyframe cues leads to the best performance across most categories.
24

---

Notably, the impact of certain parameters on mAP may vary slightly across datasets. We ablate the matcher's coefficients to understand the role of classification versus localization in action detection on the UCF101-24 -24 dataset. As shown in Figure [3] using balanced weights (1:1:1) generally yields higher AP across a majority of the 24 classes. Emphasizing classification via (3:1:1) slightly improves a few classes (e.g., Skinng).
06
05
04
Figure S3: Per-class AP(0.5:0.95) under different matcher weight ratios on the UCF101-24 dataset. We compare the decoder performance using the default matcher weights (1:1:1) versus biased setting (3:1:1).
Learning Token Importance without Explicit Attention. Unlike Transformers, which rely on explicit attention mechanisms to compute token-to-token interactions, Mamba encodes temporal dependencies implicitly through sequential modeling. This makes it challenging to directly interpret token importance. However, token selection is part of our design. shown in the first rows of Figure (S4] raw Mamba outputs tend to produce scattered and less structured token representations. To address this, we propose heuristic loss function (see Section 2.4), which encourages tokens inside the ground truth bounding boxes to receive higher importance. Furthermore, to avoid introducing significant computational overhead, we insert lightweight MLP directly before the encoder tokens as shown in Table I This MLP outputs token-wise importance score, which is then multiplied element-wise wise with the encoder tokens. Notably, the output dimension the MLP matches the number of encoder tokens, enabling one-to-one correspondence. With this, as shown in the second row, the token activations become more focused on semantically meaningful regions.
Raw Mamba Feature
Improved Feature
Raw Mamba Feature
Improved
Feature
25

---

Raw Mamba Feature
Improved Feature
Raw Mamba
Feature
Improved Feature
Raw Mamba
Feature
Improved Feature
Raw Mamba Feature
Improved Feature
Raw Mamba Feature
Improved Feature
Figure $4: Visualization of token importance. Row I shows Mamba outputs without the loss guidance. Row 2 shows outputs with our proposed importance modeling. Tokens inside target regions emphasized.
26

---

Spatial Distribution of Preserved Tokens and Corresponding Prediction Results. To better understand the effectiveness of our token selection mechanism and the final prediction results, we visualize the spatial distribution of selected tokens alongside the predicted bounding boxes for some classes shown Figure S5 around target subjects and action-relevant regions, showing strong alignment with the predicted bounding boxes. In particular, actions such as clap involve fine-grained upper-body motions, where the preserved tokens closely overlap with the hands and head regions. This confirms that our token importance module effectively captures discriminative cues, helping the decoder focus on relevant spatial areas and improving localization precision.
evident that the retained tokens are highly concentrated
Label: brush_ hair
Label: catch
Label: clap
Label: climb stairs
Label: golf
Label: jump
27

---

Label: kick ball
Label: pick
Label: pullup
Label: push
Label:
Label: shoot ball
Label: sit
2010 MIGAR
28

---

Label: swing_ baseball
Label: throw
Label: walk
Label: wave
Figure S5: Visualization preserved tokens by token importance block on JHMDB dataset. Important cues such as people can be effectively retained and are closely related to the bbox prediction of the decoder.
Extension to Multi-Label Detection. On the AVA v2.2 validation set, the adapted MOGO achieved an mAP of 16.2 (at step 26). This demonstrates that MOGO can flexibly extend to multi-label settings with competitive accuracy. Training logs are presented in Figure S6
valivll mA? asoch
0.7
Run Smaothed Value Step Time
Selashve 0.200 day
6.1559 B 1.302 25
211025 2:7 AM
8:077 day
Figure S6: Training logs the AVA dataset.
A.7 Architecture Algorithm
To further clarify our architectural design, we present step-by-step breakdown of the key modules in the form of pseudocode. Our approach introduces a token importance mechanism into video understanding pipelines, enabling dynamic token weighting and selection for efficient and effective representation learning.
29

---

Algorithm Token Importance-Guuided Frame Encoding
Require: Video input & E XBBxxxXxXXW
Ensure: Feature representation f. importance logits
x f (atchEmbed(zz) [Convert frames patch tokens]
2: Add spatial positional embedding to each frame
3: Add temporal positional embedding across frames
4: + ImportanceMLP((2) (Predict token importance scores)
5: Z r I& (()) /Weight tokens by importance (sigmoid))
6: f + MambaEncoder((z)
7: return f
Algorithm 2 Top-k Token Selection per Frame
Require: Feature f E BBXxD importance & E R BxN
Ensure: Reduced feature faetecte
1: Divide f into I' temporal segments, each with Nr = N/T tokens
2: for each frame 1 to T do
3: s(;t Nr: (t++1) Nr)
Nr:: 4: flat Nr (t + +
5: Select top-k indices by 8t
6: ft Gather ft using top-k indices
7: Append ft to fseleated
8: end for
9: return Sselected
Algorithm 3 Object Detection with Mamba Decoder
Require: Memory fmemory, learnable queries 9 E R
Ensure: Predicted boxes 6, class logits ù
q < Repeat q for each batch and initialize tgt
2: hs + MambaDecoder(tgt) fmemory)
3: Ú + ClassHead hss) (Predict object class}
4: + BoxHead (Predict bounding box (normalized))
5: return ú, b
Together, these components define the core workflow of our action detection frame- work.
30


Reviews:
[{"review_id": "RdGPDBNQfs", "reviewer": "Reviewer_Ambp", "summary": "This paper proposes a Mamba-based framework for video action detection. It includes a pure Mamba-based encoder-decoder with better computational efficiency than Transformers and a video token construction mechanism that selects the most relevant spatio-temporal tokens. The proposed method demonstrates good performance in terms of efficiency and action detection ability on the JHMDB and UCF101-24 datasets.", "questions": "I have two questions about the robustness of token construction mechanism: \n- It applies a fixed top k percentage of tokens. How does this fixed-rate strategy handle videos with high action density that require more and more tokens?\n- According to Table 2(c) and Figure 5, when k is above 40, the performance will decrease. This is very counter-intuitive. Do the authors have some insight about it ?", "limitations": "Yes, the limitations are discussed in Section 4, Conclusion and Future Work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The introduction of EQ-Mamba and QVI-Mamba enables Mamba's capabilities to the task of video action detection.\n- The proposed method demonstrates superior performance in terms of GFLOPs, inference speed, and memory usage compared to Transformer-based methods.\n\nWeaknesses\n- The main concern of this paper is limited contribution to the community. Previous works have shown the effectiveness of Mamba on video tasks, this paper is more like a specific adaption of Mamba to video action detection. \n- The experiments are conducted on relatively short 8-frame clips, quite short video sequences. This experiment setting is not very convincing when considering Mamba's potential advantage on long sequences.\n- According to Table 4, the performance is still behind some Transformer-based models, which means a potential risk of trading peak performance for computation efficiency.", "was_revised": true, "final_justification": "Thank authors for the detailed rebuttal. Besides the trade-off between peak performance and efficiency, most of my concerns have been addressed. I will increase my score to weak accept.", "rebuttal": "Thank you so much for your review. Below, we address each of the Weaknesses and Questions you mentioned.\n\n**[Weaknesses addressed]**\n\n**W1: The main concern of this paper is limited contribution to the community. Previous works have shown the effectiveness of Mamba on video tasks, this paper is more like a specific adaption of Mamba to video action detection.**\n\nR1: MOGO is re-designed from the ground up for Mamba. We provide both motivation (Fig. 1, Sec. 2.2) and step-by-step method evolution (Sec. 2.4), showing how we restructured each part of the detection pipeline based on the capabilities of Mamba (state-space modeling, linear-time sequence processing), rather than directly borrowing prior approaches. We respectfully clarify the following key points:\n\n(1) To the best of our knowledge, our work is the first to present an end-to-end, **pure Mamba-based architecture** specifically tailored for efficient video action detection, without reliance on Transformer or RCNN-like modules (see Sec. 2.2).\n\n(2) **The video token construction mechanism with learnable token importance** enables effective selection of spatio-temporally relevant tokens (Sec. 2.3). This is not present in previous Mamba-based video works.\n\n(3) Our newly designed loss function effectively distinguishes between important and unimportant tokens (**see Figure 4**). This allows the decoder to receive more informative inputs while also compressing the video tokens. \n\n(4) Our approach achieves strong results in terms of **params, FLOPs, memory, and inference speed**, outperforming or matching Transformer-based SOTA methods with significantly lower resource usage (see Table 4, Figure 7).\n\n**W2: The experiments are conducted on relatively short 8-frame clips, quite short video sequences. This experiment setting is not very convincing when considering Mamba's potential advantage on long sequences.**\n\nR2:  **[New experiments added.]** Thank you and indeed, previous experiments were based on 8-frame clips. So we conducted 3 new experiments on longer video sequences and report the results on UCF101-24. For each setup, we used the corresponding pre-trained encoder (matching the number of frames), and adjusted the decoder settings accordingly. The results are summarized below:\n\n|Ex.|#GPUs|Frames|Pretrained Model|Batch Size|Epoch|mAP|\n|--|--|--|--|--|--|--|\n|1|2×A40|16|K400|16|30|69.50|\n|2|2×A40|64|Breakfast-actions-dataset|4|30|41.90|\n|3|2×A40|64|K400|4|30|63.43|\n\nThese results demonstrate that our method performs well on longer sequences. **Due to limited time, we trained on only part of the data and for a relatively small epochs, so we expect mAP could improve further with more training.** We also observed that the choice of pretraining model is important, using K400 outperforms breakfast significantly for the same 64-frame setup. One advantage of our approach is that it supports larger batch sizes than SOTA methods, which is beneficial for efficiency. Other training metrics are shown for your reference:\n\n1. loss: epoch[0] 1.4113  -> epoch[29] 0.3047, grad_norm: epoch[0] 81.2901 -> epoch[29] 4.8639 \n\n2. loss: epoch[0] 1.5368 -> epoch[29] 0.5196, grad_norm: epoch[0] 43.4297 -> epoch[29] 19.2594\n\n3. loss: epoch[0] 1.5457 -> epoch[29] 0.3382 , grad_norm: epoch[0] 38.0396 -> epoch[29] 9.7863\n\n**W3: According to Table 4, the performance is still behind some Transformer-based models, which means a potential risk of trading peak performance for computation efficiency.**\n\nR3: Yes, we acknowledge this observation. However, although our current Mamba implementation is still at an early stage, we are already able to demonstrate **the best efficiency performance** and competitive precision compared to Transformers, and clear advantages over traditional CNN-based approaches on this downstream task.\n\nThe strong performance of Transformer-based models relies on two important factors: (1) the availability of a very large pool of pretrained models, and (2) highly optimized low-level C++/CUDA implementations.\n\nCurrently, due to the limited research on video using Mamba, it does not yet benefit from such extensive pretraining resources or mature hardware/software optimizations, which inevitably impacts peak performance. That said, we think that **Transformers May Not Be The Only Possible Foundation For Future AI**. We sincerely hope our work can motivate more researchers to explore novel low-level architectures and drive further improvements in the AI community. \n\n**[Questions addressed]**\n\n**Q1: I have two questions about the robustness of token construction mechanism: It applies a fixed top k percentage of tokens. How does this fixed-rate strategy handle videos with high action density that require more and more tokens?**\n\nR4: **[New experiments added.]** Thank you for raising this insightful question. We answer this question by 1) theory discussion of our Mamba implementation, and 2) new experimental evidence, in order to show that after processing with Mamba, **each token not only contains information about itself but also incorporates global information from all tokens**. Therefore, the action density in the video and the token keep rate have no longer to be strictly correlated.\n\nFirst, those fixed top-k percentage of tokens are used as decoder input, and the decoder receives tokens from the encoder. In the encoder, we employ a Mamba implementation:\n\nh(t) = A · h(t-1) + B · x(t)\n\ny(t) = C · h(t)\n\nIt tells that the system state describes the **entire** system, so it contains information from all tokens. This means that each individual token (in our system with shape [N, D]) will contain both its own information as well as global information from the whole system. More importantly, we implemented our encoder using **bidirectional Mamba blocks**. Therefore, this design ensures that each token can access information **from its both sides (preceding and following)** from the entire system **through the bidirectional scan**.\n\nSecond, in a new experiment on the JHMDB dataset with a similar setup, we included only tokens with low importance scores. We observed that fmap50 dropped from 76.7 to 72.7 (during training, the loss decreased from 7.98 at epoch [0] to 0.55 at epoch [29], and the gradient norm dropped from 67.6 to around 10). Below is the code:\n\n    for i in range(frame_number):\n        start_idx = i * tokens_per_frame\n        end_idx = (i + 1) * tokens_per_frame\n        frame_memory = memory[:, start_idx:end_idx, :]\n        frame_importance_logits = importance_logits[:, start_idx:end_idx]\n        top_k = int(tokens_per_frame * 0.4)\n        _, indices = torch.topk(frame_importance_logits, top_k, dim=1, largest=False, sorted=False)\n        selected_frame_memory = torch.gather(frame_memory, 1, indices.unsqueeze(-1).expand(-1, -1, D))\n\n        selected_memory.append(selected_frame_memory)\n    memory = torch.cat(selected_memory, dim=1)\n\nThese results show that even background tokens, those originally considered unimportant, can still yield reasonable action detection performance after being processed by the encoder. **This suggests that each token indeed contains global information.** However, the noticeable drop in mAP also demonstrates the importance of proper token selection. \n\nIn summary, Mamba’s intrinsic scan mechanism allows each token to aggregate information across the sequence, including high-density actions. This is a key advantage of our approach.\n\n**Q2: According to Table 2(c) and Figure 5, when k is above 40, the performance will decrease. This is very counter-intuitive. Do the authors have some insight about it?**\n\nR5: **[New experiments added.]** Thank you for bringing up this question. We still answer this question by 1) the evidence from existing works, and 2)  new experimental evidence.\n\nFirst, **similar trends have been observed in prior research**. In Figure 5 (the green line) of Efficient Video Action Detection with Token Dropout and Context Refinement (ICCV), a similar non-monotonic pattern shows up: keeping more tokens doesn’t always mean better results. \n\nSecond, **k is all about non-keyframe tokens**. Our method is to keep **all the key frame tokens** and **the most important tokens from those non-keyframes** (like the main subject or person) but **only detect the key frame**. The non-keyframes are mainly to help provide some extra context, helping the model figure out the order or direction of the action. Therefore, in the following new experiment, we aim to demonstrate that **including 100% of non-keyframe tokens may dilute the representation of keyframe tokens**. \n\n| Case                                              | mAP  |\n|:--------------------------------------------------|:-----|\n| Only non-keyframe                                      | 53.7 |\n| Proposed (key-frame + non-keyframe)        | 69.0 |\n\nWe conducted a set of mini-experiments using a small batch size and similar settings, comparing the use of all non-keyframe tokens with our proposed method. The results show that the approach using 100% of non-keyframe tokens yields lower mAP compared to our proposed method.\n\nIn addition, when we split a video into tokens, not every token is **equally helpful** for action detection. For example, many background tokens, even though including global info, may add noise and distract the model. As shown in Q1's experiment, if we only use low importance tokens (mostly background tokens), **the mAP will drop noticeably (76.7->72.7)**.\n\n**Thank you so much and we hope you will kindly reconsider your evaluation in light of these updates.**"}, {"review_id": "Z2vyHmLoaw", "reviewer": "Reviewer_RyfL", "summary": "The paper proposes MOGO, a fully Mamba-based  framework for spatio-temporal action detection, which consists of a mamba encoder models cross-frame context, and two novel state-space modules: EQ-Mamba (query self-refinement) and QVI-Mamba (query–video fusion). Evaluations on JHMDB and UCF101-24 benchmarks indicate the effectiveness of the proposed models.", "questions": "- **Questions for Figure-5.** In Fig. 5, mAP rises up to k ≈ 40% and then drops sharply as k increases further, even though the decoder has more visual information. Why does retaining a larger fraction of tokens hurt performance? Please give a principled explanation and, if possible, add an ablation that separates key-frame vs. non-key-frame tokens to pinpoint the root cause.\n\n- **Extreme sensitivity to the triplet weight w (cls : box : ovl)**.\nIn Table 3(a) the mAP plummets from 68 → 4 (!) when switching from 5 : 1 : 1 to 1 : 5 : 2, even though the overall loss is still a weighted sum of the same three terms. I wonder what failure mode causes such a catastrophic drop?\n\n- **Non-monotonic behaviour of the total-loss ratio λ.**\nTable 3(b) shows mAP rising from 73.3 → 75.6 at λ = 0.5, then falling to 74.4 for λ ≥ 1. Why does a smaller λ (i.e., weaker auxiliary loss) help, but an even smaller 0.1 hurts? Is there an intuition—e.g., regularisation vs. under-training—that could guide practitioners in picking λ without an exhaustive sweep?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Paper Strengths**\n\n- The paper is overall well-structured and easy to follow. The authors provide clear motivation, detailed methodology, and thorough experimental validation.\n- Large efficiency gains: 2–3× lower GFLOPs, latency (256 img/s) and GPU memory than strong Transformer baselines while keeping competitive mAP\n- The authors provide well-documented ablations (decoder depth, token-retention ratio, loss weights.\n\n**Weaknesses**\n- Limited novelty. Prior work has already demonstrated Mamba or other state-space models on video classification and temporal modeling, so substituting Mamba for the Transformer decoder feels incremental; the contribution lies more in engineering than in fundamentally new theory or tasks.\n- Narrow evaluation scope. Results are reported only on two small datasets (JHMDB 928 clips, UCF101-24 3k clips); the method is untested on large,  benchmarks such as AVA or ActivityNet, where its linear-time claim would matter most.\n- Token-importance mechanism insufficiently validated. Ablations vary k, but there is no comparison against random or heuristic token selection, so the incremental benefit of the learned importance loss remains speculative.\n- Depth-vs-performance anomaly. Table 2 shows a single-layer decoder achieves the best mAP, while a 6-layer stack collapses to ~32 mAP—counter-intuitive given the added capacity. Please clarify why deeper decoders “don’t work.” Is the drop caused by architectural issues (e.g., state-space over-smoothing or token interference) or by optimisation shortcomings (e.g., vanishing/exploding gradients, insufficient regularisation)? Diagnostics such as training-loss curves, gradient norms, or a variant that stabilises >1 layer would greatly illuminate whether this is a network-design or training limitation.", "was_revised": true, "final_justification": "Thanks for the response. The rebuttal is thorough and adds convincing evidence.\n\n- Novelty: Still moderate, but clarified as the first purely Mamba-based detector with a new token-importance loss.\n\n- Evaluation: New AVA2.2 results strengthen the scope and highlight efficiency.\n\n- Token importance: Random/heuristic ablations confirm the benefit of the proposed mechanism.\n\n- Depth issue & loss weights: Additional training curves and logs provide plausible explanations.\n\nOverall, concerns are largely addressed. I raise my score from 3 → 4 (Borderline accept) since the empirical validation is now much stronger and efficiency advantages are clear.", "rebuttal": "We are grateful for your comments. Below, we address each of the points you identified.\n\n**[Weaknesses addressed]**\n\n**W1: Limited novelty. Prior work...**\n\nR1: Thank you for your feedback. While we agree that prior work has applied Mamba and other state-space models to video tasks, we respectfully clarify the following key points:\n\n(1) To the best of our knowledge, our work is **the first to present a pure Mamba-based architecture** for efficient video action detection (see Sec. 2.2).\n\n(2) **The token construction mechanism with learnable importance** enables effective selection of spatio-temporally relevant tokens (Sec. 2.3). Also, our newly designed loss function effectively distinguishes between important and unimportant tokens (see **Figure 4**). \n\n(3) Our approach achieves strong results in terms of **params, FLOPs, memory, and inference speed**, outperforming or matching Transformer-based SOTA methods (see Table 4, Figure 7).\n\n**W2: Narrow evaluation scope. Results are reported...**\n\nR2: **[New experiments added.]** Our proposed MOGO was originally designed as a lightweight model, focusing on efficient action detection, rather than the *one bbox to multiple label* tasks such as AVA. In standard AVA evaluation, the typical approach involves building a bbox detector (nearly all are based on R-CNN), followed by a separate multi-label classification branch for each detected bbox. However, our MOGO is inherently end-to-end, and each bbox is directly associated with its predicted label(s) in a single stage, without the need for an additional block (No R-CNN and classification blocks). This is why we only report mAP on the two datasets.\n\nNevertheless, we attempted to **adapt** our decoder to enable *one bbox to multiple labels* without introducing an extra classification branch. Also, loss function and other experiment settings are also changed accordingly. The simplified code is shown below:\n\n    class ...(...):\n        def forward(self, x):\n            mem, importance_logits = encoder(x)\n            hs = sdecoder(tgt, mem)\n            outs = [self.heads(output) for output in hs]\n            out = {\n                'pred_actions': outs['class'], # whether a person/subject is detected (binary classification)\n                'pred_boxes': outs['box'],\n                'pred_logits': outs['obj'] # covering all classes in the dataset\n            }\n            return out\n\nIn the original code, class (all classes+1) and coords were one-to-one matched. **On the AVA2.2, our modified MOGO achieved a val mAP of 16.2 (at step 26), which is comparable to CVRL-f32(SlowOnly-R50) with a mAP of 16.3. Notably, our decoder also demonstrates an advantage in GFLOPs (13.782 vs 42).**\n\n**W3: Token-importance mechanism insufficiently validated. Ablations vary k...**\n\nR3: **[New experiments added.]** Thank you for raising this insightful question. We answer your question by 1) previous research evidence and 2) new experiments.\n\nFirst, according to a prior work, *Efficient video action detection with token dropout and context refinement* (ICCV), **selecting non-background tokens as important tokens leads to good performance on action detection tasks**. This provides theoretical support for our token-importance mechanism.\n\nSecond, according to your recommendation, we included new experiments for comparison. \n\n**(1) Random token selection**. We revised our code as following:\n\n    for each frame in video:\n        frame_tokens = get_tokens(memory, frame)\n        selected_indices = random_select(top_k = 40% of frame_tokens)\n        selected_tokens = frame_tokens[selected_indices]\n        add selected_tokens to selected_memory\n    final_memory = concat(selected_memory)\n\nWe removed the *important logits* part and randomly selected tokens for this experiment. After 30 epochs of training, our model achieved fmAP: **67.2** (Initial loss: 7.9832 (epoch 0), Final loss: 0.6982 (epoch 29)). Compare to the benckmark (mAP: **67.4**) with the same environment and setting in Figure 5, our initial method shows superior to the random method.\n\n**(2) Heuristic token selection**. We refer to *Faster Video Transformers with Run-Length Tokenization* and integrated this heuristic RLT-inspired token selection mechanism into our pipeline, and customized it with our token importance scores:\n\n    for patch_idx in range(tokens_per_frame):\n        ...\n        for i in range(frame_number):\n            start_idx = i * tokens_per_frame\n            token = memory[:, start_idx + patch_idx, :]  # [B, D]\n            # compare to previous frame's token\n            if prev_token is not None:\n                diff = (token - prev_token).abs().mean(dim=-1)  # [B]\n                if (diff < threshold).all():\n                    run_length += 1\n                    continue\n            if run_length > 0:\n                patch_tokens[-1]['run_length'] = run_length + 1\n            patch_tokens.append({'token': token, 'run_length': 1})\n            ...\n        if run_length > 0:\n            patch_tokens[-1]['run_length'] = run_length + 1\n        for item in patch_tokens:\n            selected_memory.append(item['token'])\n            run_length_embeds.append(item['run_length'])\n    memory = torch.stack(selected_memory, dim=0).transpose(0, 1)\n\nThis block compresses repeated/unchanged tokens over time, and only keeps tokens when *semantic changes* are detected. After 30 epochs of training, our model with RLT-based token selection achieved fmAP: **71.8** (Initial loss: 7.9931, grad_norm: 68.1378 (epoch 0), Final loss: 0.5432, grad_norm: 9.8033 (epoch 29)). **Compare to the benckmark (mAP:76.7) with the same environment and setting, our method shows superior to this heuristic method and can show the benefit of our learned importance loss we proposed.**\n\n**W4: Depth-vs-performance anomaly. Table 2 shows a single-layer...**\n\nR4: Due to the rebuttal policy, we cannot display figures, but we summarized the training loss, gradient norms, and mAP across all >1-layer decoder experiments in the following. Those experiments were all conducted for 50 epochs with a batch size of 32 and 1 A40 GPU. For 3 decoder layers, the training loss 5.1019 -> 0.4732, grad norm: 87.6617-> 5.8463,with mAP reaching 0.710. However, for 6 layers, loss 4.9990 -> 2.6747 **decreases but plateaus at a higher value**, grad norm: 49.6281 -> 5.2908 with mAP saturating 0.32. \n\nOur explanation is that when the decoder has fewer layers, the network learns stably, loss converges, and the model achieves good performance. This suggests that the network capacity is sufficient to model the task. However, when the decoder is deeper (e.g., 6 layers), we observe that while the training loss still decreases, it plateaus at a much higher value. This is likely to indicate the dataset size is limited and the model is overfitting with more layers.\n\n**[Questions addressed]**\n\n**Q1: Questions for Figure-5. In Fig. 5...**\n\nR5: **[New experiments added.]** Thank you for bringing up this question. We still answer this question by 1) the evidence from existing works, and 2)  new experimental evidence.\n\nFirst, **similar trends have been observed in prior research**. In Figure 5 (the green line) of Efficient Video Action Detection with Token Dropout and Context Refinement (ICCV), we can see keeping more tokens doesn’t always mean better results. \n\nSecond, **k is all about non-keyframe tokens**. Our method is to keep **all the key frame tokens** and **the most important tokens from those non-keyframes** (like the main subject or person) but **only detect the key frame**. The non-keyframes are mainly to help provide some extra context. Therefore, according to your suggestion, we aim to show that **including 100% of non-keyframe tokens may dilute the representation of keyframe tokens**. \n\n| Case                                              | mAP  |\n|:-|:-|\n| Only non-keyframe                                      | 53.7 |\n| Proposed (key-frame + non-keyframe)        | 69.0 |\n\nWe conducted a set of mini-experiments using a small batch size and similar settings, comparing the use of all non-keyframe tokens with our proposed method. The results show that the approach using 100% of non-keyframe tokens yields lower mAP compared to our proposed method.\n\nIn addition, when we split a video into tokens, not every token is **equally helpful** for action detection. For example, many background tokens may add noise and distract the model. We further conducted an experiment that only used low importance tokens, **the mAP drops noticeably (76.7->72.7)** (during training, the loss decreased from 7.98 at epoch [0] to 0.55 at epoch [29], and the gradient norm dropped from 67.6 to around 10).\n\n**Q2: Extreme sensitivity to the triplet weight...**\n\nR6: **[New experiments added.]** To answer your question, we show the experiment records of both the original 1:5:2 run and 5:1:1 baseline for reference. And **we conducted the 1:5:2 experiment once again (a repeat 1:5:2 (20250728))** and found a similar trend. \n\nWhen using 1:5:2 weights, we observed high loss (epoch[0] 6.0x/5.9x -> epoch[29] 2.x) and grad norms > 400 after 20 epochs, while mAP stayed under 0.05. In contrast, with 5:1:1, loss:epoch[0]: 8.0x -> epoch[29] 0.8x, grad norms drops steadily after 10 epochs, and mAP climbs above 0.6. So now we have enough evidence to say that the 1:5:2 setting may overwhelm optimization with box/ovl weights, preventing the model from learning effective action features.\n\n**W3: Non-monotonic behaviour of the total-loss ratio λ...**\n\nR7: Our loss is custom-designed: it combines a **standard action detection loss** and our **own token importance loss**. Since these two terms **may differ in scale**, tuning λ is necessary. Also, mAP differences across λ values falls within an acceptable range, which is common for such hyperparameters. We found that λ = 0.5 gives the best trade-off on this dataset.\n\n**We hope you will kindly reconsider your evaluation in light of these updates.**"}, {"review_id": "WkKilx85NW", "reviewer": "Reviewer_Pwyx", "summary": "this paper propses a mamba based backbone for efficient spatio-temporal video action detection and a novel token-selection scheme to reduce compute while working on video data.", "questions": "Please see the weakness section.\n- How many layers of encoder do you use? How many mamba layers are being used?", "limitations": "Addressed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "i am glad this paper landed in my lap , i haven't read a lot of papers in spatio-temporal action detection for a while, and **this is a really refreshing attempt** to see people still working in this area. \n\nthe question i will leave to discretion of other reviewers is:  **does application of mamba to this domain count as novel**. since mamba is already known, and its application to spatio-temporal action detection is not known, i will count this as a novel work . **my bet is a lot of people will think it is not novel**. but i feel that this work is something that will **greatly benefit** our community, and **open up an area for efficient inference in spatio-temporal video action detection**, and i am excited to see where it goes.\n\nstrengths:\n- to the best of my knowledge, i am not aware of a mamba based architecture in spatio-temporal video action detection , this makes the sequence modelling linear as a function of sequence length, as opposed to transformers. **There is video-mamba paper**, which does it in general video understanding, but not in this area. \n \n- halving the flops (102-104) over 240 flops in josies paper (tuber) seems pretty good. \n- the token importance block seems to be independent of the length of video being processed (table 1). This seems good for efficiency.\n\n\nweaknesses \n\n- how many encoder blocks you use. If it's more than 1, then mamba glances again and again, and i dont know how it justifies the title , mamba glances only once? is it because you only apply token selection once (at the beginning before encoding)?\n\n- line 31: mamba is a good concept, but the performance in table 4 still lacks behind tuber. **Can u plz plz train MOGO on IG+ K400 :-), and not just K400**, that should help close the gap with TubeR.\n\n- i think one baseline should be against flash attention v3.  even though it is not linear, but it would be interesting to see how optimized transformer  fares against mamba. \n\n- where are the results on ava 1.1, ava 2.2? josie showed those when she wrote her TubeR paper, which the authors compared against. so those experiments should also be shown. \n\n- where are the vmap metrics? reporting those at 0.5 iou threshold is a standard practice in literature. table 7 in supp only shows vmap @0.2. it needs more numbers at 0.5 iou also.\n\n- in table 1, decoder takes 100 queries. what if there are more than 100 people in the scene? what if we dont know the number of people to detect for action detection before hand. the network wont work in that case. ucf/jhmdb dont have those cases, but that doesnt make the paper bulletproof. Please add this in limitation.\n\n- token selection blocks assumes a keyframe, and selects top k tokens per frame. This assumes that “equal number” of k tokens per frame are “semantically important”. i guess a better would be to use rle compression over time. check this paper, it should help you guys out :-) [1], it was neurips spotlight last year i think.  the token selection block seems too simple in my opinion, and if authors can make it a bit better, it would be great [1]. \n\n- **formatting suggestions**: the paper seems to sparse, too much white space, and figures dont look pleasing to the eye. while the official guidelines dont penalize authors on their language, and how they wrote the paper, my suggestion would be to shift all the figures to the right, and bring some content from the supplmentary to the main paper. \\vspace is your friend.\n\n- could you guys get any results on a long-video setup, it will greatly strengthen a paper, since action detectors like TubeR generally dont work beyond 64 frames to the best of my knowledge. \n - If long term video is not an option, then you could consider ucf, amir divided a long video into subclips i think when he made ucf101, and those are named as v_basketball_01, v_basketball_02 etc, you could in theory combine them , and show action detection on the long term video. mambas linear cost should allow you to fit those frames in memory, and tuber should fail. that will be a great selling point. \n\n- please add related work w.r.t open world action detectors like SIA. (https://github.com/siatheindochinese/sia_act)\n\n- how do you guys handle the cases where the action switches rapidly, like basketball dunk etc. i guess training a simple action classifier wont do the job. \n- action detection is an inherently difficult problem, i.e. box around person has to be predicted only when the person is starting a action, how do you separate such cases from cases when the person is there in a frame and not doing an actual action (for eg, he is just dribbling basketball, and basketball dunk action has not yet started. i have seen a lot of action classifiers fail in such cases. an inductive bias to encode this constraint may be greatly helpful)\n\n- **I could not find code in the supplementary**\n- results should come before ablations.\n\n\n[1] https://rccchoudhury.github.io/rlt/ \n\n--- my rating is 3, and i will change it once the experiments are more comprehensive. The official guidelines don't permit additional experiments, but without that the paper is incomplete. If those can be done, and added it would be good, otherwise, it might need another revision cycle. **It's a lot of work**, and i apologize for asking so many experiments, but that is a choice that authors might be willing to make. I will stand by my assessment.", "was_revised": true, "final_justification": "Dear Authors:\n\nThank you so much for your detailed rebuttals, additional experiments. I have increased my score accordingly. I would love to see your paper at NeurIPS, and am excited by how this might open up an area for linear-time spatio-temporal action detection. \n\nHowever, please note, as per **what the respected ACs mentioned**, i am not sure what constitutes as ```major revisions.``` \n\nI would suggest categorizing your (promised) changes in a GLOBAL comment as:\n(1) Changes you have made which are just clarifications on the experiments **already** present in the paper.\n(2) Changes which are **brand new**, , and may be considered as ```substancial``` changes,  for eg, adding rle, ava experiments, discussions on table 14 of video-mamba paper not being reliable, flash attention baseline, discussions on training for >64 frames aka trackformer.\n\n\nTo be fair in the review process, the ```ultimate decision lies with the respected AC, PC's who know far more```, and whether they will ```allow``` such substancial revisions, \n\nThanks for your understanding, \n\nreviewer", "rebuttal": "Really grateful! **They mean a lot to us.** Transformers've been everywhere, but we believe there’s room for new ideas, e.g. Mamba. Below are the point-by-point responses.\n\n**W1: how many encoder blocks…**\n\nR1: We use a pretrained 32-block Mamba encoder (32 justified by ablation in Supplementary A.5). However, the “glances only once” actually refers to our custom Decoder block (EQ-Mamba+QVI-Mamba+FFN), which is just ONE single layer. This setup lets us process all the encoded general spatio-temporal info in one single decoding pass. For decoder, our ablations show that just ONE layer already gets good results on datasets like JHMDB. We'll highlight clearly that it’s our newly designed, action-detection-specific Mamba-based Decoder that “glances only once\".\n\n**W2: line 31: mamba is a good concept…**\n\nR2: **[New experiments added.]** We dug deep into resources around the IG dataset (moabitcoin/ig65m-pytorch). But it seems no public access to the *data*, only *pretrained weights* based on R(2+1)D_34. \n\nFor TubeR using IG+K400, we find that TubeR sourced from the pretrained CSN-152 model. And CSN-152 comes from R(2+1)D_34 based on this paper by Facebook: *Video Classification with Channel-Separated Convolutional Networks*. Therefore, the key behind TubeR’s strong results is **not training on IG directly**, but starting from the pretrained WEIGHTS.\n\nThat said, inspired by your suggestion, we tried other available large-scale pretraining sources beyond K400: Something-Something V2 (ssv2) and Breakfast-actions-dataset (breakfast), and integrated them into training.\n\n**ssv2**: We use 2 A40 GPUs and train for 50 epochs with the batch size of 30. And the fmap on the JHMDB is 64.1 (loss: epoch[0] 7.9834  -> epoch[49] 0.4847, grad_norm: epoch[0] 67.6360 -> epoch[49] 6.6854). **So K400's better than ssv2 in terms of pretrained model**.\n\n**breakfast:** 2 GPUs, 30 epochs with the batch size of 4. We tried to input 64 frames (initially, 8) for processing. And the fmap on the UCF101-24 is 42.0 (loss: epoch[0] 1.5368  -> epoch[29] 0.5196, grad_norm: epoch[0] 43.4297 -> epoch[29] 19.2594). map is not satisfactory, so we use K400 back but 64 frames. And the mAP improves to 63.43. **So pretrained model is vital and K400 is better than breakfast**.\n\n**W3: i think one baseline should be against flash attention…**\n\nR3: **[New experiments added.]** First, efficiency. To be completely transparent with you, our AI community, and us, we have to admit that the efficiency of our current mamba is not on par with the latest flash attention kernels. (Since our hardware doesn’t support flash attention v3, we use v2.8.2 released in July 2025.)\n\nWe modified our decoder (as baselined in Table 11) and swapped in flash attention for Mamba. The decoder parameters dropped from 7.228M to 5.321M, and GFLOPs from 2.297 to 1.116. \n\n    class FlashDecoderLayer(...):\n        def __init__(self, ...):\n            super().__init__()\n            self.self_attn = MHA(...)\n            self.cross_attn = MultiheadAttention(...)\n            self.ffn = nn.Sequential(...)\n            self.norm = nn.LayerNorm(...)\n        def forward(self, tgt, memory):\n            tgt = self.norm(self.self_attn(tgt))\n            tgt, _ = self.cross_attn(tgt, memory, memory)\n            return self.norm(self.ffn(tgt))\n\nSincerely hope our work can motivate more bottom-up optimization (e.g., by Tri Dao) to refine mamba C++/CUDA operators. Though still in its early stages, we're already able to show competitive efficiency compared to other transformer applications for this task.\n\nSecond, accuracy. When we trained this flash attention for 30 epochs, the performance on JHMDB was poor, with fmAP50 at only 2.08(loss: epoch[0] 8.7587 -> epoch[29] 3.9476, grad_norm: epoch[0] 9.3831 -> epoch[29] 3.7795). This suggests maybe **simply** swapping in the implementation does not yield good results for our task. \n\n**W4: where are the results on ava...**\n\nR4: **[New experiments added.]** MOGO is initially designed for efficient, end-to-end detection. But AVA is for *multi-label for one bbox* tasks. Unlike standard AVA setups that require a separate bbox detector and multi-label classifier, MOGO predicts bbox and label(s) together in one stage. That’s why we only report mAP on the two datasets.\n\nBut we attempted to **adapt** our decoder to enable *one bbox to multiple labels*, even without introducing an explicit cls branch. Also, loss function and other experiment settings are also changed accordingly. The simplified code is shown below:\n\n    class ...(...):\n        def forward(self, x):\n            mem, importance_logits = encoder(x)\n            hs = decoder(tgt, mem)\n            outs = [heads(output) for output in hs]\n            out = {\n                'pred_actions': outs['class'], # whether a person/subject is detected (binary classification)\n                'pred_boxes': outs['box'],\n                'pred_logits': outs['obj'] # covering all classes in the dataset\n            }\n            return out\n\nIn the original code, *class* (all classes+1) and *coords* were one-to-one matched. **On the AVA2.2, our modified MOGO achieved a val mAP of 16.2 (at step 26), which is comparable to CVRL-f32(SlowOnly-R50) with a mAP of 16.3. Notably, our decoder also demonstrates an advantage in GFLOPs (13.782 vs 42).**\n\n**W5: where are the vmap…**\n\nR5: **[New experiments added.]** In several SOTA works (e.g., Efficient Video Action Detection with Token Dropout and Context Refinement), **fmap50** is widely reported and we included it in Table 4. vmap evaluates the entire video as a tube and uses 3D IoU for evaluation, which is sensitive to tube generation quality. Nonetheless, we have included it.\n\n|Met|vmap20(U)|vmap20(J)|vmap50(U)|vmap50(J)|#Params|GFLOPs|Throughput|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Ours|70.8|67.9|39.4|42.0|82M|102-104|256|\n\nOur vmap50 is relatively low compared to vmap20, mainly due to a few classes with low APs, which significantly affect the average: Basketball, BasketballDunk, TennisSwing, and VolleyballSpiking achieve nearly 0 AP at vmap50, despite reasonable scores at vmap20. This is because that our model is designed for action detection on the keyframe: take in a sequence, use the keyframe for spatial info, and our outputs are centered on those key moments, not continuous tubes.\n\n**W7: token selection blocks assumes a keyframe…**\n\nR7: **[New experiments added.]** We immediately **customized RLT-inspired token selection mechanism with our token importance scores**:\n\n    for patch_idx in range(tokens_per_frame):\n        prev = None\n        for i in range(num_frames):\n            idx = i * tokens_per_frame + patch_idx\n            t = memory[:, idx, :]\n            if prev is None or (t - prev).abs().mean() >= threshold:\n                selected_memory.append(t)\n                prev = t\n\nThis block compresses repeated/unchanged tokens over time, and keeps semantically changed tokens. After 30 epochs of training, our model with RLT-based token selection achieved fmAP: 71.8 (Initial loss: 7.9931, grad_norm: 68.1378 (epoch 0), Final loss: 0.5432, grad_norm: 9.8033 (epoch 29)). Several hyperparameters (e.g., the threshold for rl, norm) remain tunable, and is promising after careful tuning.\n\n**W9&W10: could you guys get any results... & If long term video...**\n\nR11: **[New experiments added.]** We ran 3 new experiments on longer video sequences using UCF. For each setup, we used a matching pre-trained encoder and adjusted the decoder accordingly. Use 2×A40 GPUs with training epoch of 30.\n\n|Ex.|Frames|Pretrained Mode|Batch Size|mAP|\n|--|--|--|--|--|\n|1|16|K400|16|69.50|\n|2|64|Breakfast-actions-dataset|4|41.90|\n|3|64|K400|4|63.43|\n\nThese results show that our method performs well on longer sequences. Due to limited time, we trained on only part of the data and for a relatively small epochs, so we expect mAP could improve further. We also observed that pretraining model is important, using K400 outperforms *breakfast* significantly. Also, our approach supports larger batch sizes than SOTA methods. \n\n**W12: how do you guys handle the cases…**\n\nR12: **[New exp added.]** We'll show that after processing with Mamba, **each token not only contains information about itself but also incorporates global information from all tokens**. So even the rapidly switched action info are also stored in each token in Mamba.\n\n(1) **Theory discussion**. In encoder:\n\nh(t) = A · h(t-1) + B · x(t)\n\ny(t) = C · h(t)\n\nThe system state describes the **entire** system and will contain info from all tokens, meaning that each token will contain both its own information as well as global information from the whole system.\n\nMore importantly, we implemented our encoder using bidirectional Mamba blocks. So it ensures that each token can access information from its **both sides** (preceding and following) from the entire system through the **bidirectional scan**.\n\n(2) **New experiment evidence**. We ran an experiment on JHMDB using only tokens with low importance scores. Even then, fmap50 only dropped from 76.7 to 72.7(during training, the loss decreased from 7.98 at epoch [0] to 0.55 at epoch [29], and the grad norm dropped from 67.6 to around 10). This shows after Mamba encoding, **even less “important” tokens contain enough global info for action detection**, though good token selection still helps accuracy.\n\n**W13: action detection is an inherently difficult…**\n\nR13: When training, we include a separate background class in addition to the dataset action categories. This is not only for handling empty/background objects, but also for *a person is present in the frame but not performing* if the data annotations are of high quality.\n\n**For W6 (add to limitation), W8 (format), W11 (add related work), W14 (code release), and W15 (content order), these are really helpful. and we’ll definitely address them in our final script! We hope you will kindly reconsider your evaluation in light of these updates.**"}]

Author rebuttal:

Meta-review: This paper proposes a Mamba-based framework for video action detection. It includes a pure Mamba-based encoder-decoder with better computational efficiency than Transformers and a video token construction mechanism that selects the most relevant spatio-temporal tokens. The proposed method demonstrates good performance in terms of efficiency and action detection ability on the JHMDB and UCF101-24 datasets though the accuracy (mAP) is considerably lower than for transformer based method (TubeR) .

The reviewers found the experiments in the submitted paper to be not sufficient.During the rebut period, the authors were able to report on additional experiments suggested by the reviewers; these included results on the AVA dataset and tests with different pretraining datasets and variations on token selection.

This paper received only three reviews. Based on the discussion, two reviewers raised their ratings to borderline accept and one to accept. There was also discussion of whether the paper required so many changes that it amounted to a new submission. The authors did a good job of summarizing the changes so the ACs feel that the revised content has been reviewed already. The ACs recommend that the paper be accepted.
Decision: Accept (poster)