Smooth Regularization for Efficient Video Recognition
Authors: Gil Goldman
Raja Giryes
Mahadev Satyanarayanan
Venue: NeurIPS 2025 poster ()
Keywords: Video recognition
Temporal smoothness regularization
Temporal coherence
Lightweight video models
Primary area: deep_learning
TL;DR: Video Recognition models are not smooth as a function of time, but smoothing them improves accuracy

Abstract: We propose a smooth regularization technique that instills a strong temporal inductive 
bias in video recognition models, particularly benefiting lightweight architectures. 
Our method encourages smoothness in the intermediate-layer embeddings
of consecutive frames by modeling their changes as a Gaussian Random Walk
(GRW). This penalizes abrupt representational shifts, thereby promoting low-
acceleration solutions that better align with the natural temporal coherence inherent
in videos. By leveraging this enforced smoothness, lightweight models can more
effectively capture complex temporal dynamics. Applied to such models, our 
technique yields a 3.8%–6.4% accuracy improvement on Kinetics-600. Notably, the
MoViNets model family trained with our smooth regularization improves the 
current state-of-the-art by 3.8%–6.1% within their respective FLOP constraints, while
MobileNetV3 and the MoViNets-Stream family achieve gains of 4.9%–6.4% over
prior state-of-the-art models with comparable memory footprints. Our code and
models are available at https://github.com/cmusatyalab/grw-smoothing.

Smooth Regularization for Efficient Video Recognition
Gil Goldman
Raja Giryes
Computer Science Department Carnegie Mellon University gilg@andrew. cmu.edu
School of Electrical and Computer Engineering
Tel-Aviv University
rajaßtauex tau aau
Mahadev Satyanarayanan Computer Science Department Carnegie Mellon University
satyaccs cmu. edu
Abstract
We propose smooth regularization technique that instills a strong temporal induc- live bias in video recognition models, particularly benefiting lightweight architec- tures. Our method encourages smoothness in the intermediate-le -ayyer embeddings of consecutive frames by modeling their changes as a Gaussian Random Walk (GRW). This penalizes abrupt representational shifts, thereby promoting low-acceleration solutions that better align with the natural temporal coherence inherent in videos. By leveraging this enforced smoothness, lightweight models can more effectively capture complex temporal dynamics. Applied to such models, our technique yields a 3.8%-6.4% accuracy improvement on Kinetics-600. Notably, the Mo ViNets model family trained with our smooth regularization improves the current state- of-the-art by 3.8%-6. 1% within their respective FL.OP constraints, while Mo- bileNetV3 and the Mo LENets-Strean family achieve gains of 4.9% over prior of-he-tf models with comparable memory footprints. Our code and models are available at hattps://githan com/ wtt.///o///cccc////eee cwwsstyyaaa///aaaam/ap/ haggggr-bbbggaaaeae hwwpaaaaaabbgggoosooooii....
Introduction
Video recognition has rapidly evolved over the past decade, with models becoming increasingly capable of learning sophisticated spatial and temporal representations. However, despite remarkable advancements, many architectures still suffer from overfitting or inefficient use of temporal informa- tion [2] (28]. In light of these challenges, we introduce a novel smooth regularization approach designed to instill a strong inductive bias specifically tailored video data. The key insight behind our method is that video content often exhibits continuous motion and gradual changes in appearance, suggesting that representations should vary smoothly over time. By explicitly encouraging this smoothness, we aim to guide neural networks toward more stable and generalizable internal feature representations, ultimately leading to improved performance across range of video recognition tasks.
Our regularization strategy focuses the intermediate- embeddings produced by neural network when processing consecutive frames. Instead of allowing these embeddings to fluctuate arbitrarily across frames, we constrain their dynamics resemble Brownian motion, which translates to imposing Gaussian Random Walk (GRW) behavior in the frames discrete settings that promotes continuous and relatively modest rates of change. The inspiration behind this modeling choice comes from the fact that, in a typical video, adjacent frames exhibit only gradual shifts in object positioning, scale, lighting, or motion. By treating frame-to-frame representation shifts as form of GRW, we incorporate a principled, mathematically grounded way to preserve smoothness in
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GFLOPS vs Top-1 Accuracy
XA8,3
MAIREZAAR
24
56
SOTA
14
24
14
Top- Accurnry
14
24
70
22
24
240
50
05
LDPS (1ogg scale
Nemory (MB3)
Figure Performance Results on Kinetics-600. By simply adding GRW-smoothing to exist- ing models, we achieve significant improvements. Left: Accuracy vs. FLOPs, where each point corresponds to a published model (see Table I for references). GRW-smoothing improves the state- of-the-art performance of efficient models 1886.11% Notably, Mo ViNeiNN--33-OR accuracy at just 56.4 GFLOPs, while the closest model, MVVTvv-BB 222----32x33- requires 18.3 more FLOPs. Right: Accuracy Memory. GRW-smoothing improves the state-of-the-a performance of memory-efficient models by 4.9.6.4%
achieves 85.6%
the learned embeddings. This not only reflects natural representation of videos but also acts as a regularizing force against abrupt or erratic changes in the network's 's internal states.
A core outcome of our approach is that it naturally discourages large jumps between successive embeddings. Mathematically, our formulation involves adding the GRW penalty term to the training objective, which grows whenever the model produces excessively rapid transitions between frames in the network embedding space. By penalizing these abrupt shifts, we encourage networks to learn features that evolve more gently over time. This favors solutions that maintain sense of temporal consistency, that is, low acceleration in the embedding space, leading to more coherent internal representations that can better capture the true temporal dynamics present in real-world videos.
By aligning model training with the natural temporal structure found the data, our approach makes it easier to networks to focus on learning meaningful temporal correlations rather than wasting capacity on fitting noisy or abrupt changes. Consequently, the model becomes more sensitive to subtle motion cues, often crucial for recognizing fine-grained actions or micro-movement semantics without sacrificing robustness to variations due to noise in the network embedding space.
While bigger network may have enough capacity to learn both the variations and noise in the embedding space together with the changes motion, this is more challenging resource constrained networks. Thus, we focus in this work on such type of networks. We demonstrate the effectiveness of our proposed smooth regularization on such lightweight models. To verify benefits, we trained these smaller-scale networks on the popular Kinetics-óre 600 dataset, a large benchmark known for its diversity of human action classes. By simply adding our novel loss function the training of these architectures, we get consistent gains of 3.8%-6.4% in classification accuracy shown in Figure] leading to new state-of-the-art performance under FLOP and memory constraints.
Our main contributions can be summarized as follows:
Smoothness Prior in Video Recognition: We introduce novel regularization technique that enforces smoothness in the intermeciate-layer embeddings of consecutive video frames by modeling their changes as GRW.
State-of-the-Art Performance Under Efficiency Constraints: Our technique outperforms current leading solutions within similar memory and compute range, confirming the broad applicability of our method to resource-limited scenarios.
Flexible Framework: Focusing smoothness as a strong inductive bias provides plug-and-plas regularization option for existing video recognition pipelines. integrates seamlessly into different architectures with negligible computational overhead as illustrated in Figure 3

---

2 Related Work
Video recognition in general. Early approaches to video action recognition learned spatiotemporal representations with 3D CNNs [5 127. with subsequent advances such as two-Stream networks [26 and SlowFast [9] improving the balance between spatial semantics and motion modeling. Temporal segment sampling 30 provided efficient way to cover long videos with sparse clips. With the advent of transformers, attention-based models extended self-attention to spatiotemporal tokens 2 Subsequent work introduced hierarchical and multiscale designs and localized attention improve efficiency [6] 1221. and explored alternative attention mechanisms such as trajectory attention [24), as well as MLP-like backbones [22 and video-specific ViT adaptations [25]. Despite strong accuracy, pure transformers often incur substantial computational and memory costs for long videos, particularly relative to compact CNN baselines 22
Lightweight video recognition. A parallel line of work targets real-time and on-device deployment by reducing compute and parameters. CNN-based designs dominate this space due to depth wise separable convolutions and efficient temporal operators. MoVeNets [I5) introduce a NAS-designer family of efficient 3D CNNs and stream buffer for constant-memory streaming, achieving strong uccuracy-effictie
trade-offs. X3D [8] systematically compounds network width, depth, and temporal resolution to yield compact 3D CNNs. Temporal Shift Module (TSM) [21] augments 2D CNN backbones with a zero paraomerr,,,rr, zero-FLOP temporal exchange, enabling video recognition with image-classfficaionnleee
compute. Further lightweight temporal modules include TEA [9), TDN 29), , and TAdaConv 13] which inject temporal cues with modest overhead. Trunsformer-buxer models such as MVIT 6 and VideoSwin-T [22] reduce attention cost via multiscale hierarchies or windowed attention, yet they often remain heavier than the most compact CNN baselines at strict mobile budgets. Hybrid architectures combine convolutions and attention to leverage local inductive bias with global modeling. UniFormer [18] exemplifies this trend by interleaving convolutional blocks and self-attention. Overall, lightweight video recognition in practice is led by CNNs and convolution-attention hybrids (see Table Min in the Experiments section below).
Temporal coherence: slowness and higher-order smoothness. An early prominent approach to temporal coherence, slow feature analysis (SFA) [31 prefers features that evolve as slowly as possible time by minimizing the expected squared temporal derivative of each feature subject to zero mean, unit variance, and decorrelation constraints. The motivation is that latent factors in natural videos typically change gradually, so maximally slow features capture stable, semantically meaningful structure.
Temporal order and ranking constraints. A complementary line of self-supervised work focuses on verifying or predicting the chronological order of frames or clips. Representative approaches include Shuffle & Learn (23), Sorting Sequences (Order Prediction Networks) (166,, and Odd-One- Out networks [IO]. These objectives encourage representations to respect temporal structure by reasoning about sequence order rather than enforcing slowness.
3 Method
- and encoding of video recognition model's intermediate layer v(X)= = Z (z=) where M and N denote the numbers of inpul frames and embedding time steps (after any temporal subsampling). respectively. The main objective of this work is to guide the optimization process to favor solutions y for which z(t) is a smooth function of t.
Consider a video frame sequence X Mo1
Warm-up Example. Let consider instructive simplified example. We constructed small dataset containing 1,000 short videos of a few model airplanes performing one of three rotations: Yaw, Pitch, or Roll, starting from random initial position, shown in Figure top).
To analyze the geometry of the embeddings, we trained two identical models. In both, we use a pretrained MobileNet as the recognition model that calculates embeddings per frame and then single Transformer layer that process several consecutive frames for the temporal information. The models trained to predict the rotation label using cross-entropy loss. In the second model, we smooth
3

---

Yaw
(b) Pitch
(c) Roll
:
Principal
Principal2
20
-00
-12
Principal
Principal
(d) Embedding without smoothing
(e) Embedding with smoothing
Figure 2: Warm-up Example. Top: The used Airplanes dataset containing 1,000 training and 100
test short videos of model airplanes performing of three rotations, starting from random position. The dataset isolates temporal classification, as any single frame is independent of the rotation label. Bottom: Output embeddings of two identical models trained with and without the smoothness term. In green, blue and red are typical clips embeddings for Yaw, Pitch and Roll, respectively, projected to the first two principal components of the embedded test Each point is single frame embedding. The index is the clip frame index.
the MobileNet embeddings Z with an additional loss term that penalizes high accelerations in z(f). directing the optimization towards keeping d² z= low. This term, which is the main contribution The resulting embeddings without and with this respectively.
of the paper, formally described in Section 3.1
smoothing term presented in Figures 2d and 2e
The geometry in Figure 2d shows that the standard encoder lacks a smooth structure. As it does not utilize the smooth prior have with respect to video sequences, more complicated function was learned than simply the movement in yaw/pitch/roll.
contrast, Figure [2c shows that the model trained with the smoothness term finds intrinsic linear two-dimensional representation where each rotation mapped to a certain direction. Note that the curve in the plot shows that each rotation is smooth, and the accelerations
d z()((() are low.
We
explain how calculate smoothing term that suggest for given normalized embedding in neural network.
3.1 The Gaussian Random Walk (GRW) Smoothing Term
Consider a normalized layer output Z = (z, f=0 prior on the embeddings. We do within a time window, dividing into short subsequences:
N-1 over time. Our goal is to induce smoothness
2 (26- 2 (ZeT.... 11) O..,,.,, 1, = [N/T].
(I)
Imposing direct smooth prior on Z poses difficulty, mapping all z(() to single point is "maximally smooth" but results degenerate solution that is clearly undesired. Therefore, we construct the smooth loss in two steps.

---

Inpul frames
Original model
Clogigsst))
Global Pool
BN
Σ GRW(Z9)
Z
Figure 3: Intermediate layer smoothing, The encodings Z are global-pooled along the spatial dimensions, then normalized across the batch dimension, where we use BN without learnable parameters. The sub-clips Z are fed into GRW.
Input frames
Original model
Affine Light-weight Transfomer
C'(loggit))
Σ 1 E
Figure 4: Final layer smoothing, Output encodings ((X) = 2 of a given video model are affine transformed Z. The sub-clips Z (Zer,... (( are fed into GRW regularization, as an additional loss term, then further processed using few Attention layers.
1. Frame Ordering: We first introduce contrastive loss that directs the optimization towards mappings that maintain the structure of the order of the frames.
2. Smooth Prior: We then impose distribution that favors low-acceleration mappings and plug in the contrastive loss, resulting in our smoothing prior.
Frame Ordering. Consider the following right-frume-order contrastive loss
f(zū, zi, 22 zT-1)
C,(0)= =- E log ExES(I.T) f (zj,, (1( 2- x2))) ... 2x
(2)
where ff a probability distribution that we will define in the next step of the smooth prior, and S(1 : 7") is the group of all permutations æ of the elements {1, (1...., T 11-1 1). That is, we fix the first frame and contrast the correct ordering the remaining frames with all their permutations. This
prevents the loss term from degenerate solutions that collapse to the same point.
Smooth Prior In the loss term in Equation 22) can be chosen freely. We impose the smoothness prior by setting f as a distribution favoring low acceleration embedding.
Define the velocities and accelerations of the embedding
d z = (v = (zf 2h 24 - -2p-2);
dl
d
pc A (as) (vi -V. vg..., 2 vz
dt
5

---

To smooth zt) we model the distribution of the velocities as random walk with Gaussian increments,
v = Vū +
a
T t T T 2,
(3)
==
where (af)=z) are i..dd, af NO, I). Under this assumption
T-3
f(29) plvi.
-(A)) =
II M(a))
(4)
where, with abuse of notation, / denotes the density of the standard normal distribution. To put it all together, the loss in Equation becomes
p(49)
((p) = E N.o
log Σ
(5)
EES(1 p(A+)
where A are the accelerations according to the permutation n, AG = a ( -1)-
d (Zo, 2 (2)))
The loss (5) requires a scaling parameter. For any embedding y the scaling of the embedding by a to aup introduce an inverse temperature parameter
p(aA9)
[(a+) = E N.c
log
ExES(1.T) p(aA))
We determine the scaling by adding another term (?(V) controlling the unconditional speeds
T-2
~ ~~(,0, 7),
22V*) = log II N(v)).
t=
The final smooth prior is
log p(A))
Lemooth(p)) -
+aN(V9)
(6)
X.c Σ SES( p(Ag)
and the final loss is
Loe ALeemooth-
Note that the sum in Equation taken over all permutations, which grow factorially with T. For large T. uniformly sample k permutations, for some given k. In practice, enumerate all (T - 1)! orderings when T < 7 and uniformly sample k: = 1000 permutations when T > 7. Since (T' 1)! < 1000 for T < the number of evaluated orderings per clip is < 1000, keeping the computational cost of the denominator in Equation (6) effectively independent of T.
3.2 Applying GRW to a neural network
We propose to apply the GRW loss intermediate, typically higher, layers of video recognition models to induce the smoothness inductive bias. Section we demonstrate empirically the advantage of using this loss term.
We propose applying the GRW term in two possible locations in neural network: (i) smoothing of an intermediate layer, (ii) smoothing of the final layer. We describe each option next.
Intermediate Layer Smoothing (Figure 3). For an intermediate layer output 2 E |RCNN*X where N is the temporal dimension, C is the channel dimension, and K is the (flattened) spatial dimension. the encodings are globally pooled along the spatial dimensions and then normalized to have expected value of R and mean unit length,
VC BNia(GP(21) e CCNN

---

where the BN1a does not have learnable shift and rescale parameters. Then, we extract sub-clips Z from Z and use them as input to the GRW loss.
Final Layer Smoothing (Figure D. We smooth the final encodings of the model, just before the classification head and then further process the smoothed embeddings with lightweight temporal model, namely, a Transformer with 1-2 layers. Formally, the final layer output 2 is normalized using learnable affine transformation, Z Linear(Z), which is applied each embedding separately. In the supplementary material we show that optimizing this transformation together with the GRW loss, put the embeddings in Z close the unit sphere and therefore we refer to this affine transformation as normalization. Also here, we create from Z the sub-clips Ze and use them in the GRW loss.
Experiments
To demonstrate the effect of smoothing using method, we compare the performance of models trained with and without GRW regularization.
Datasets. We report our results on Kinetics-60M (K600) 4 and Kinetics-400 (K400) [4]. Both datasets consist of 10-second videos of varying resolutions and frame rates, labeled with 600 and 400 action classes.
Models and Implementation. We focus on video recognition models with lower computational requirements to control training time, and since introducing inductive bias becomes increasingly important when efficiency a factor. We selected the current of-t--of-the-ar models in the lowest categories of FLOPS and memory, and tnnettned them by applying GRW-smoothe we applied GRW to the MoViNet model family A,, and their streaming versions AO-S,... A22- S. The streaming variants, denoted as Ai-S, are memory memory-efficient efficient versions of the Mo ViNet-ii models that process videos frame by frame. The baseline performance of these streaming models is lower than that of their non-streaming counterparts due to the use of causal operations.
Specifically,
At the lowest end of memory requirements, we applied GRW MobileNerV3 Small with the following modification. We extract a "base frame" every T frames, where T is the GRW clip window (see Section 3). All frames within T -frame window are processed individually, but alongside their corresponding base frame. To support this, we modified the first layer to accept 6 input channels.
We used Final Layer GRW-smoothing (see Figure which produced better results than Intermediate Layer GRW -smoohigg (Figure3) 33) However, both methods improve accuracy results, as discussed in Section 4.2 We maintain the original model's final layer output dimension in the normalization affine transformation and apply GRW weoosoooggin on the output. We replace the classification head with a 2-layer vanilla transformer with a standard x4 MLP expansion factor (see Section 3 for details).
We set 10-1 as the balancing factor Equation 7 and a = 1/2 as the scaling factor in Equation (6) We found that the results robust to perturbing the values of X, which suggests that gradients in the direction of smooth solutions align with gradients with respect to the classification likelihood; see ablation studies in Subsection
In all experiments we set the GRW window to span 0.5-1.0s of video. Specifically, for MoNeNet- A0/-A1/-A2-GRW and SabileNetV3-S-GRRRE
we use 5 fps with T=5 (covering 1 s), and for MoViNet-A3-GRB we use 12 fps with T-6 (covering 0.5 s). For these values of T we enumerate the full set of orderings in Equation ( of size 1)} (i.e., 24 for 5 and 120 for T-6), so no permutation subsampling was required (i.e., k was not used).
Remark 1. GRW-smoothing operates on per-frame embeddings Z = (Zz) rather than raw frames, so the added computation is small relative to the backbone. For example, in MoViNet-A2-S-- the input frame x, E 22222224X33 (280,528 dims) is mapped to 2; E 16400 i.e. an R 235x reduction in dimension. Consequently, the O(k- T) vector operations and log-probability evaluations in Equation (6) are negligible compared the early convolutional blocks. Empirically, with the T values used in this section (full enumeration of permutations), the wall-clock time per training epoch for Mo MoViNet-A2-SS --WS---W--R is only 2% higher than Mo MoViNee-A2-S without smoothing; even for larger T where one might sample up to k=1000 permutations, the overall cost remains dominated by the backbone forward/backward pass.
Training. On K600, we fine-tune starting from existing weights. On K400, we use transfer learning from K600. We employ simple training process, not applying augmentations except when training
7

---

the A2 and A3 models on K400. We use different training rates for the transformer head and model backbone, decreasing with cosine learning rate scheduler in the range [10- 4 10 #I for the model backbone and [10-9, 10-5) for the transformer head. We fine-tune for 14 epochs on K600 and 10 epochs on K400.
The smaller models, A0-Al and MobileNet, were trained on single dgx-A100 for 3-5 days, while the A2 and A3 models were trained on 2 dgx-A100 for days.
4.1 Results
For results, we use single clip evaluation for our models. We report Top-1 accuracy results against FLOPS and memory. The resolution column refers to the resolution of the input video, with 224 indicating 224 x 224. The frames column is given by num num_clips nun fromee used the evaluation. The GFLOPs column indicates the total computation for the evaluation of single video sample.
As seen in Table 1 all models with GRW-smoothing achieve significant improvements in accuracy and set new SOTA results in their corresponding GFLOPs group. Specifically, MoViNet-AO-SS GRGW MoViNet-A1-S-On
NoViNet-A2-S-GRY and MoViNet-A3-GRW improve the SOTA results by 6.1% 5.2%, 4.7% and 3.8,, respectively. For JoViNer-A3-GRY the next model achieving similar accuracy, MViTv2-B-32x3, requires x8.3x more GFLOPs.
We compare current SOTA memory-efficient models, namely MobileNet and streaming versions of the VoNe model family, before and after smoothing them with GRW; see Table MobileNetV3: GRW, MoViNet-AO-S-GRA versions by 6.0%, 6.4%, 5.5% and 4.9%, respectively, see also Figure righht)
MoViNet-AA---GRRE MoViNet-A2-S-GRE improve their non-smooth
4.2 Ablation Studies
We study the effect of GRW by (i) disentangling the contribution of smoothing versus the added attention layers, and (ii) analyzing sensitivity to the key hyperparameters: the GRW window T, the scaling factor in Equation 6 and the balance À in Equation 7
Placement and attention vs. smoothing. We ablate where GRW applied (Intermediate vs. Final layer) and whether the attention head alone explains the gains. Concretely, on K600 we train: (i) MoViNee-A2-S-GRY (Final Layer Smoothing; Figure 4. (ii) MoViNet-A2-S-GRW (Intermediate
Layer Smoothing: Figure 3), and (iii) Mo ViNee--22-- + attention, the baseline equipped with the same 2-layer Transformer head but without GRW. Table shows that adding attention alone yields a small gain over the bascline (+0.9 Top-1), while training with GRW yields an additional +4.0 absolute (for a total +4.9 over the baseline) when applied at the final layer; applying GRW at an intermediate layer also improves accuracy (+2.4) even without attention.
Sensitivity to T, a, and A. We further ablate the GRW hyperparameters on MoViNet-AO-S-GRR trained on K600, varying one parameter at time and keeping all other settings fixed (Final Layer Smoothing; training protocol as in Sec. 4h. Results are summarized in Table
Summary. Across these ablations, GRW is not overly sensitive near the settings used in our main results: 7=5, a= in Equation (6), and x=10-1 in Equation 7 Short windows under-exploit temporal coherence, very long windows may blur distinct motions, and excessively large À can over-regularize.
Conclusion
In summary, we introduced novel smooth regularization technique designed to enhance temporal understanding video recognition models, particularly lightweight architectures. By modeling the evolution of frame embeddings as Gaussian Random Walk, our method penalizes abrupt representational changes, effectively promoting low-acceleration solutions that align with the natural temporal coherence of video data. This approach has demonstrated significant accuracy improvements, notably 3.8%-6.49 6.4% gain on Kinetics-600, and has established new state-of-the-an performance in compute-constraing settings. By combining proposed GRW regularization with models such as

---

Table 1: K600 by FLOPS
Model
Top-1
GFLOPs
RES
FRAMES
MoViNet-AO-SGGRW
78,4
2.7
172
1 X 50
MoViNet-AO(55]
72.3
2.7
172
1 50
II
61.3
2.8
224
1 X 50
MobileNeeVV----TSM211
65.5
2.8
224
X 50
X3D-XSI8I
70.2
3,9
182
X 20
MoViNet-AT-S-GRW
81.9
172
1 50
MoViNet-AI
76.7
6.0
172
1 X 50
X3D-S
73.4
7.8
40
MoViNet-A2
78.6
10.3
224
X 50
10
MobileNeeV3-LLL11
68.1
11.0
224
50
11
MobileNeeVV--+TTSMDII
71.4
11.0
224
50
12
MoViNet-A2-S-GRR
83.3
11.3
224
X 50
13
X3D-MB8
76.9
19.4
256
50
14
X3D-XSI8
72.3
23.3
182
30 X
15
MoVINet-A3-GRW
85.6
56.4
256
120
16
MoViNet-A3
81.8
56.9
256
1 x 120
X3D-S
76.4
76.1
30 13
18
X3D-L
79.1
77.5
356
x 50
Mo
83.5
105
290
80
20
UniFormer-S
82.8
167
224
4 16
21
X3D-M
78.8
186
256
30 X 16
22
X3D-L
80.7
187
356
1 120
23
13D
71.6
216
224
1 X 250
24
MoViNet-A5
84.3
281
320
120
25
MViT-B-16x410
82.1
353
224
5 X 16
26
MoViNet-A6[15
84.8
386
320
1X 120
27
UniFormer-B
84.0
389
224
4 16
28
XVIT (8x)
82.5
425
224
3 8
29
XVTT (1 6
84.5
850
224
3 x 16
30
WVIT-B-32x311
83.4
850
224
32
31
MViTv2-Bi B- 32x3
85.5
1030
224
5 x 32
Table 1: Top-1 accuracy, total video evaluation (in GFLOPs), input resolution (RES), and FRAMES = clips x frames per clip used for evaluation on Kinetics-60n Models enhanced with our proposed smooth regularization are marked with GRW. These models consistently outperform their baselines and other caate-of-the-dee methods under similar FLOP constraints. Variance: for MoViNe--AO-SGGRY across three seeds obtain 78.4 0.05 Top-1 (mean + std).
MoViNet-A0/1/2/3 and their streaming counterparts, well MobileNetV3,, we improve overall performance within their respective FLOP and memory constraints. The GRW regularization acts flexible, plug-and-plas component with minimal computational overhead, guiding networks towards more stable and generalizable feature representations.
While our approach presents promising results, it has certain limitations. The core assumption of Gaussian Random Walk dynamics, while beneficial for many natural videos, might not universally optimal for content characterized by extremely abrupt transitions or intentionally discontinuous motion. A future work may explore such vidcos and how to extend GRW for such cases. Furthermore, eopeeimeemnn
while our experiments demonstrate significant gains lightweight models, the extent of improvement on very large-capacity models, which we could not do due to computational constraints, require further investigation. Finally, the necessity of the frame ordering component in the contrastive loss, while effective in preventing degenerate solutions, does introduce an additional layer of complexity to the training objective. A more efficient variants be studied in future work.
Looking ahead, several avenues for future research emerge. Extending application of our GRW smoothing to a wider array of video architectures, including more complex Transformer-basee models, could yield further insights into generalizability. Investigating its efficacy across diverse video understanding tasks beyond action recognition, such temporal action localization or video

---

Table K600 by Mem
Model
Top-1
Mem MB
Table 3: K400 by FLOPS
MobileNerV3-S111
61.3
29
Model
Top-1
GFLOPs
MobileNetV3-S-GRY
67.3
30
Mo GOW
70.4
2.7
MoViNet-AO-S-GRW
78.4
53
MoViNet-AO[IS
65.8
2.7
MoViNet-AO-SJ15]
72.0
53
MoViNet-A2
75.0
10.3
MoViNet-A1-S-GRW
81.9
67
MoViNet-A2-GRY
77.6
11.3
MoViNer-AI-S[13]
76.4
67
X3D-XS
69.5
23.3
MoViNet-A2-S-GRW
83.3
78
MoViNet-A3-GRW
81.7
56.4
MoViNet-A2-S(13
78.4
78
78.2
56.9
Table 2 Top-1 accuracy and memory usage (in MB) for menory-efficient models on Kinetics-600. Models enhanced with smooth regularization (GRW) are shown in bold and consistently out- perform their baselines under identical memory constraints.
X3D-S18
73.5
76.1
VideoMambal 17
76.9
108
Top-1 accuracy, total video evaluation cost, Kinetics-000.
Table 3
Table 4: Ablation on placement (K600, MoViNet-A2-S family). Model
Top-1
GFLOPs
MoViNet-A2-S-GRRW (final layer)
83.3
11.3
MoViNet-A2-SSGRYY (intermediate layer)
80.8
10.3
MoViNet-A2-S + attention (no GRW)
79.3
11.3
Mo MoViNettA2-S(5
78.4
10.3
anomaly detection, presents another promising direction. Additionally, dynamic smoothing window that adapts to video content is favorable. Finally, a more in-depth theoretical understanding of how GRW regularization influences the optimization landscape and feature learning process would be beneficial.
Acknowledgements
Parts of this research conducted using ORCHARD, a high-performance cloud computing cluster. The authors would like to acknowledge Carnegie Mellon University for making this resource available its community.
This material is based upon work supported by the United States Navy under award number N00174- 23-1-0001 and by the National Science Foundation under grant number CNS-2106866 The content of the information does not necessarily reflect the position or the policy of the government and no official endorsement should be inferred. This work was done in the CMU Living Edge Lab, which is supported by Intel, Arm, Vodafone, Deutsche Telekom, CableLab, Crown Castle, InterDigital, Seagate, Microsoft, the VMware University Research Fund, IAI, and the Conklin Kistler family fund. Any opinions, findings, conclusions or recommendations expressed in this document are those of the authors and do not necessarily reflect the view(s) of their employers or the above funding sources.
This work was partially supported by grant from The Center for Al and Data Science at Tel Aviv University (TAD).
(a) Window T
(b) in Equation
(c) Equation
Top-1 (%)
Tup-1 (%)
Top-1 (%)
77.3
77.9
0.25
78.0
0.01
78.4
0.5
78.4
78.0
78.3
1.0
75.5
1.0
72.0
smoothing
no smoothing
72.0
smoothing
Table 5: Hyperparameter ablations for GRW on K600 with AO- (a) Window T
peaks at 7-5. (b) Scaling a shows mild optimum a=0.5 in Equation (6) (c) Balance À is
robust {0.01, 0.1) and degrades at A=1.0 in Equation (
10

---

References
[1] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6836-6846, October 2021.
[2] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021.
[3] Adrian Bulat, Juan Manuel Perez Rua, Swathikiran Sudhakaran, Brais Martinez, and Geor- gios Tzimiropoulos. Space-time mixing attention for video transformer. Advances in neural information processing systems,
19607, 2021.
[4] Joao Carreira, Eric Noland, Andras Banki-Horvatie Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600, arXiv preprint arXiv:/808.01 2018.
[5] Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2017.
[6] Haoqi Fan, Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6824-6835, 2021.
[7] Mohsen Fayyaz, Emad Bahrami, Ali Diba, Mehdi Noroozi, Ehsan Adeli, Luc Van Gool, and Jurgen Gall. 3d cnns with adaptive temporal feature resolutions. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 4731 1731-4740, 2021.
[8] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 203-213,200 2020.
[9] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[10] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervieed video representation learning with odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3636-3645, 2017
[11] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
[12] De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. What makes video a video: Analyzing temporal information in video understanding models and datasets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7366-7375 2018.
[13] Ziyuan Huang, Shiwei Zhang, Liang Pan, Zhiwu Qing, Mingqian Tang, Ziwei Liu, and Marcelo H Ang Jr. Tada! imporaaly--ddatiiv convolutions for video understanding. arXiv preprint arXiv:2110006178 2021.
[14] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 1705006950000 2017.
[15] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 16020-16030,
2021.
11

---

[16] Hsin- Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Sing-Hsuann Yang. Unsupervised represen- tation learning by sorting sequences. In Proceedings of the IEEE international conference on computer vision, pages 667-676, 2017.
[17] Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding. In ECCV, 2024.
[18] Kunchang Li, Yali Wang, Gen Peng, Guannan Song, Yu Liu, Hongsheng Li, and Yu Qiao. UniFormer: Unified transformer for efficient spatial-temporal representation learning, In ICLR, 2022.
[19] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 909-918, 2020.
[20] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages
2022.
[21] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video under- standing. Proceedings of the IEEE/CVF international conference on computer vision, pages 7083-7093, 2019.
[22] Ze Liu, Jia Ning, Yuxin Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
[23] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European conference on computer vision, pages 527-544. Springer, 2016.
[24] Mandela Patrick, Desmond Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In NeurIPS, 2021.
[25] Oran G. Sharir, Asher Noy, and Lihi -elnik-Manor. An image is worth 16x16 words, what is a video worth? In arXiv:2 101.1113..111333 13915, 2021.
[26] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[27] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2015.
[28] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling vidco masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14549-14560, 2023.
[29] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1895-1904, 2021.
[30] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pages 20-36, 2016.
[31] Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
[32] Daijun Zhang. Kunchang Li, Yi Wang, and et al. Morphmlp: An efficient mip-like backbone for
representation learning. ECCV, 2022.
12

---

A Scaling
Here we show the scaling behavior of ERR-ssmothinng The main result we prove that the optimal solution for smoothing window of size T, applied to approximately centered data, lies within a ball of radius bounded by OTTvlnT). We prove this result in the one-dimensional case.
Let us recall the setup:
Consider T' 2 3 points in [0, R with fixed endpoints
0 = 22 S 2T R.
Z = 12))=11
(8)
and define velocity and acceleration vectors as
V(Z) = (22 21, 2T 2T- E ERT-L
V (m))= T-1
A(Z) (V' U1 - 2r-2) E R
A= (0))) 2
We will denote by 2 T the set of all point configurations of the form ttht is, non-decreasing sequences with 2 = O. For any Z E z we denote R(Z) 2T.
For any such configuration define the loss components:
T
T-1 (26+1 22)2
(z =
and
-T= (Z
exp
(Z log
Σn exp
-- az())*)
where the sum is over all permutations π of ( TT fixing 21, and
z (21, (2)) ,
With the above notation and (see Equation (6 in the paper), GRW loss is given by
[(Z) = Ca(Z) + (,,
Theorem (GRW -smoohiggg scale). Given 2 3. let Z = arg minzezr ((Z). Then
R(Z*) - O O(TVnT)
Remark 2. Theorem suggests that for approximately centered embedding, it is beneficial to allow scaling degree freedom before feeding embedding vectors into ERW-smoothing to allow the solution to converge into the O VITTT ball. Therefore, when applying GRW- ERW-smoothing to sequence model embedding, we either directly scale and center the embedding using non-learnable batch normalization (BN), or allow for a learnable linear transformation to set the scaling. Empirically, we found that centering is not required when linear transformation is applied.
Proof. We will use the following propositions in the proof, providing their proofs subsequently:
Proposition (Uniform Lower Bound on L). For any 2 E 2
R2(Z)
((Z) 2 (z N 2(T 1)
(9)
Proposition 2 (Uniform Configuration Upper Bound), Consider the uniform configuration
Zng (0, 1'
R
2R
(10)
T
For R -
C(Z,) 0(T
(11)
13

---

Assuming Proposition I and Proposition 2 we will now complete the proof of Theorem 1 their proofs subsequently.
and provide
From Proposition and Proposition
2 follows that
R² 2(T 1) < &
for some constant C₁. Rearranging, we get the desired result. Now we prove the auxiliary claims.
Proof of Proposition / Since La (Z) is non-negative, we have ((2) L.( Recall that L. -d' uf. For any Z the velocities are non-negative and satisfy Σ. 0z žт 21 -[(2) = R. Therefore, (Z) 2 minyeRT
[ 1 VV The last is a classic quadratic program with n²² the minimizer V (1, 1), attaining the minimum 2(T-1)- where these velocities are realized
by the uniform configuration of the points. Hence, obtain ((Z) (Z)
(2)
2(T
Proof of Proposition
Fix R and consider the uniform configuration
10
We have
R² 2(T
R
((Zn) In Σ
1)2 S(Z)
exp
2(T 1)
(=)(1)=
2.(2.)
Lu(Zu)
where
T-2
S(a) = E (T(1 2) (1 +1)))))((())
The velocity term simplifies
R2 2(T - 1)
(T 1)2 --1
Co(z
(12)
2(T - 1) 2
The acceleration term simplifies
Σ exp (-19(m) )
L La(zz) = In
Inn(T 1)!)= O(ThT)
(13)
(=((1)=1
Then by 12 and 13 ((zu) = O(TInT)
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Throughout the paper, we describe the proposed method and show through various experiments its advantage the constrained compute setting.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
We explicitly discuss the limitations of our work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
There are no theoretical results shown in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide in Section detailed descriptions of all experimental settings, including model architectures, initialization procedures, hyperparameters, and evaluation metrics. This information is sufficient to reproduce the main experimental results that support our claims.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will publish our code and instructions upon acceptance, enabling faithful reproduction of all experiments. All datasets used are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ctaaps..c/publcc/guii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide full training and evaluation details in Section 4 Where possible, adopt hyperparameter settings from prior work; otherwise, perform grid search. These details are specified to ensure clarity and reproducibility.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Training models is very computationally demanding and therefore cannot afford training multiple models. Yet, the advantage they achieve are with very high margin, which clearly shows that the improvement is indeed significant and not marginal.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the GPU used and training time in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: hett://nneuisppe wtwpursps.ccppulc///eiiieee cc/ppulicetthiicccudee.........
Answer: [Yes]
Justification:
We followed the code of ethics while conducting the research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper provide a regularization for the task of video action recognition, which makes action recognition video models more efficient.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The proposed algorithm is not of high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide reference to all the models and datasets we use in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We use existing datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not use crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We did not use crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We use LLMs only in their standard use of improving writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurre /0tps///ecccoooeeeeee////2////// for what should or should be described.
21


Reviews:
[{"review_id": "Y0zQF6WxFE", "reviewer": "Reviewer_YNFs", "summary": "The paper introduces a smooth regularization method specifically tailored for videos. The method relies on the idea that the consecutive frames in a video have a lot of redundancy and this, therefore, should be reflected in how the embeddings also change. Gaussian random walk is used to model the frame embedding evolution and to avoid sudden representational changes in the embeddings. This in theoery should encourage slow developing (low-accelations) solutions that are aligned with video data. The evaluations are provided on k400 and k600 datasets.", "questions": "Following the sake of brevity, i will refer to the points mentioned in the weakeness section. \n\nHow can the proposed technique be adapted to account for the sudden temporal changes? (weaknesses: point 4)\nWhy are the evaluations on SSV2 missing? \nWhy is the method not compared with latest state of the art methods? (point 7)", "limitations": "NA", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\nOverall, the idea is simple yet intriguing. \nThe paper is written decently and is overall easy to read and understand. \n\n\nWeaknesses:\nThe following weaknesses are not in the order of importance:\n1. The related works section is quite lacking and not comprehensive. It does not talk about other temporal regularization techniques at all which have been explored in this domain quite extensively. Moreover, the realted works section also doesnt explore the lightweight networks literature which is the setting for the proposed paper. \n\n2. From the writing point of view, the mathematical symbols introduced lack the definitions (normally given below as soon as the symbol/variable is introduced). E.g. line 96 introduces X but does not define M or N and later C in the equation 1. There other instances in the rest of the paper as well. \n\n3. The introduction lacks the buildup of the motivation and the problem at hand. This can be made better. \n\n4. A big issue with the approach is it's assumption of smooth transitions between the frames in the videos. This may be true for some applications (e.g. surveillance/security footage etc), but not for most video content, which has sudden transitions. For example, I would be very interesed in knowing the results on the SSV2 dataset and other temporally challenging datasets. Even in normal videos encoded with prevalent video codecs, it may be reasonable to assume the smoothness in the \"p-frames\" but there are also I-frames where the difference is simply too huge to be encoded as a p/b frame. This proposed technique won't work good with fast moving objects, scenes or other videos of this nature.\n\n5. The evaluations are quite small and need to be expanded to other video datasets such as SSV2 and can also be shown to work with localization and detections datasets as well to show the effectiveness of the proposed technique. \n\n6. The additional complexity introduced by frame ordering component in the contrastive loss is not quantified. \n\n7. The comparison methods/models are quite old...\n\n8. Ablation on lambda (eq 7) and alpha (eq 6) is not provided.", "was_revised": true, "final_justification": "The answers provided in the rebuttal provide useful information therefore, I have increased my rating from reject to borderline reject but the current state of the paper still falls short of borderline accept.", "rebuttal": "Thank you for your careful assessment and constructive feedback. We appreciate the opportunity to clarify the issues you raised. Should the paper be accepted, we will incorporate the revisions below into the camera-ready version.\nBelow, we first contextualize our contribution and then respond point-by-point to your comments.\nScope and main claims\nOur goal is to show that temporal regularization that biases optimization toward **piecewise-smooth** solutions is particularly effective for **efficient** video-recognition models. Although the technique is architecture-agnostic, its benefit is most pronounced in the low-compute regime. We will emphasize this in the introduction and, with your approval, update the title accordingly.\nOur experiments substantiate the claim. On Kinetics-600 we **advance the state of the art by 2.8–3.4 pp Top-1 for all models within ≤ 1 TFLOP** (Table 1). We compared against every published model in this budget to the best of our knowledge; we would be grateful for pointers to any we might have missed.\nCNN and hybrid backbones dominate this compute range. We therefore applied GRW smoothing to MoViNet, which remains the strongest lightweight backbone on Kinetics-600. For instance, our MoViNet-A3-GRW reaches 84.8 % Top-1 at 56.4 GFLOPs, while the first model that surpasses this accuracy (MViTv2-B-32 × 3) requires 1,030 GFLOPs, **over 18× more computation**.\nWe also note **Video-Mamba (2024)**, the latest sub-1 TFLOP model: it lags behind MoViNet-A3-GRW by 3.8 pp Top-1 while using nearly twice the compute (Table 3; see Supplement §1 for the corrected typo).\n\nPoint-by-Point Responses\n\n1. *“The related works section is quite lacking and not comprehensive. It does not talk about other temporal regularization techniques at all which have been explored in this domain quite extensively. Moreover, the realted works section also doesnt explore the lightweight networks literature which is the setting for the proposed paper.”*\n\nWe will expand the section to survey prior temporal-regularization methods and the lightweight video-model literature.\n\n2. *“From the writing point of view, the mathematical symbols introduced lack the definitions (normally given below as soon as the symbol/variable is introduced). E.g. line 96 introduces X but does not define M or N and later C in the equation 1. There other instances in the rest of the paper as well.”*\n\nWe agree that explicit definitions improve readability. We will define M, N, C, and any other symbols at first mention.\n\n3. *“The introduction lacks the buildup of the motivation and the problem at hand. This can be made better.”*\n\nWe will strengthen the motivation and problem statement in the introduction.\n\n4. *“A big issue with the approach is it's assumption of smooth transitions between the frames in the videos. This may be true for some applications (e.g. surveillance/security footage etc), but not for most video content, which has sudden transitions. For example, I would be very interesed in knowing the results on the SSV2 dataset and other temporally challenging datasets. Even in normal videos encoded with prevalent video codecs, it may be reasonable to assume the smoothness in the \"p-frames\" but there are also I-frames where the difference is simply too huge to be encoded as a p/b frame. This proposed technique won't work good with fast moving objects, scenes or other videos of this nature.”* \n\nOur smoothness prior is imposed **within short windows (~1 s), not across an entire video**, analogous to assuming continuity only between I-frames. In other words, we assume that the representation can be made **piecewise smooth** (see § 3.1).\n\nSSV2: Google has not released MoViNet weights or training recipes for SSV2. Reproducing SOTA on SSV2 from scratch requires tuning many hyper-parameters (weight decay, label smoothing, dropout, stochastic depth, batch size, LR schedule, layer-wise LRs, several augmentation parameters, AutoAugment, etc.), which exceeds the capacity of the two DGX machines available to us. We therefore focused on Kinetics, where strong baselines exist, allowing us to isolate the effect of GRW smoothing.\n\n5. *“The evaluations are quite small and need to be expanded to other video datasets such as SSV2 and can also be shown to work with localization and detections datasets as well to show the effectiveness of the proposed technique.”*\n\nPlease see the SSV2 explanation above. Extending GRW to localization/detection is promising future work but beyond the scope of our supervised-classification study.\n\n6. *”The additional complexity introduced by frame ordering component in the contrastive loss is not quantified.” \n\nThe **overhead is negligible**. GRW operates on the frame embeddings\n$Z = (z_i)$, where each frame $x_i$ is significantly reduced in size. For example, in MoViNet-A2-GRW, each input frame $x_i$​ is 224 × 224 × 3, whereas $z_i$ is 640-D—> **230 × smaller**. A full training epoch for A2-GRW is only **2 % slower** than baseline A2. We will report this explicitly.\n\n7. *\"The comparison methods/models are quite old...\"      \n\nWe compare **all models** with ≤ 1 TFLOP on Kinetics-600 (Table 1) and ≤ 100 GFLOPs on Kinetics-400 (Table 3 and the typo correction in Supplement §1). Video-Mamba (2024) is included and still trails our best model by 3.8 pp while using nearly 2 × the compute.\n\n8. *“Ablation on lambda (eq 7) and alpha (eq 6) is not provided.”*\n\nWe thank the reviewer for this important comment. \nWe include the following ablation studies on different values of $\\lambda$ and $\\alpha$:\n\n| Model               | Top-1 (%) | α             |\n|---------------------|-----------|---------------|\n| MoViNet-A0-S-GRW    | 77.2      | 0.25          |\n| MoViNet-A0-S-GRW    | 77.4†     | 0.5           |\n| MoViNet-A0-S-GRW    | 76.9      | 1             |\n| MoViNet-A0-S-GRW    | 75.7      | 2             |\n| MoViNet-A0-S        | 72.0      | no-smoothing  |\n\n†We include here our most current results, which improve upon the submitted version and **advance the SOTA by 5.4 pp.**  \n\n| Model               | Top-1 (%) | λ             |\n|---------------------|-----------|---------------|\n| MoViNet-A0-S-GRW    | 77.4      | 0.01          |\n| MoViNet-A0-S-GRW    | 77.4†     | 0.1           |\n| MoViNet-A0-S-GRW    | 77.3      | 1             |\n| MoViNet-A0-S        | 72.0      | no-smoothing  |\n\n\nWe would gladly make the above additions to the paper. \n\nReviewer Questions: \n\n1. *”How can the proposed technique be adapted to account for the sudden temporal changes?”*\n\nPlease see response on smoothness assumption in 4. above (window-level, § 3.1).\n\n2. *”Why are the evaluations on SSV2 missing?”*\n\nPlease see SSV2 explanation in 4. above\n\n3. *”Why is the method not compared with latest state of the art methods?”*\n\nLatest sub-1 TFLOP model (Video-Mamba 2024) is included and outperformed; \nPlease see our response to 7. above.\n\nThank you again for your constructive feedback."}, {"review_id": "20cdxbtoCj", "reviewer": "Reviewer_8e7P", "summary": "This paper proposes a novel smooth regularization technique designed to enhance temporal understanding and improve the efficiency of video recognition models, especially lightweight architectures. The core idea is to instill a strong temporal inductive bias by modeling changes in intermediate-layer embeddings of consecutive frames as a Gaussian Random Walk (GRW). This regularization penalizes abrupt representational shifts, encouraging smoother, more temporally coherent feature evolutions. The method is presented as a plug-and-play loss term that can be applied to either intermediate or final layers. Extensive experiments on the Kinetics-600 and Kinetics-400 datasets demonstrate that this GRW-smoothing consistently yields significant accuracy improvements, establishing new state-of-the-art performance within their respective FLOP and memory constraints.", "questions": "1. Computational Overhead of GRW Loss. Could the authors provide more specific details on the computational overhead (e.g., FLOPs or actual runtime increase during training) introduced by the GRW loss term, especially considering the permutation sampling?\n2. Sensitivity to Hyperparameters: How sensitive are the results to the choice of hyperparameters \\lambda and \\alpha? The paper states robustness to perturbing \\lambda, but a more detailed analysis of the hyperparameter space for both parameters would be valuable.", "limitations": "The authors has explicitly discussed limitations in Section 5.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Novel and Intuitive Regularization. The concept of enforcing smoothness in video frame embeddings through a Gaussian Random Walk (GRW) is novel, intuitive, and well-motivated by the inherent temporal coherence of video data. This provides a strong inductive bias.\n\n2. Significant Performance Gains for Lightweight Models. The paper demonstrates impressive and consistent accuracy improvements (2.8%-6.0%) on various efficient architectures (MoViNets, MobileNetV3) on standard benchmarks (Kinetics-600, Kinetics-400). This directly addresses the practical challenge of deploying video models on resource-constrained devices.\n\n3. Plug-and-Play Framework. The GRW regularization is designed as a simple, additive loss term, making it easy to integrate into existing video recognition pipelines with negligible computational overhead. This enhances its practical applicability.\n\nWeaknesses\n\n1. Computational Cost of Permutations. The loss function in Equation (5) involves a sum over all permutations of frames within a sub-clip, which grows exponentially with the sub-clip length T. While the authors mention sampling k permutations for large T, the computational overhead of this sampling and the contrastive loss itself could still be substantial, especially during training. More details on the practical values of T and k used in experiments and their impact on training time would be beneficial.\n\n2. \"Frame Ordering\" Complexity: The introduction of a \"frame ordering\" contrastive loss to prevent degenerate solutions adds complexity. While necessary, a simpler or more efficient mechanism to achieve this goal could be explored in future work.\n\n3. The figures and tables of this work can be improved. Some figures are blurry and some tables take up too much space.", "was_revised": true, "final_justification": "My major concerns have been addressed in the rebuttal. Accordingly, I am raising my score from borderline reject to borderline accept.", "rebuttal": "We thank Reviewer 8e7P for the careful review and constructive feedback. We appreciate the opportunity to clarify the points you raised. If the paper is accepted, we will incorporate the appropriate changes in the camera-ready version.\n\nPoint-by-Point Responses\n\n1. *“Computational Cost of Permutations. The loss function in Equation (5) involves a sum over all permutations of frames within a sub-clip, which grows exponentially with the sub-clip length T. While the authors mention sampling k permutations for large T, the computational overhead of this sampling and the contrastive loss itself could still be substantial, especially during training. More details on the practical values of T and k used in experiments and their impact on training time would be beneficial.”*\n\nGRW-smoothing assumes piecewise temporal coherence in video data. On Kinetics, we used temporal windows of 1 second. Depending on the FPS of each model, this corresponds to the following values of T: 5, 5, 5, 5, and 12 for MobileNetV3-GRW, A0-GRW, A1-GRW, A2-GRW, and A3-GRW, respectively.\n\nIn our implementation, we compute all permutations when T ≤ 7, and uniformly sample k = 1000 permutations when T > 7. Since the number of permutations for T ≤ 7 is (T − 1)! < 1000, this keeps the computational complexity effectively constant.\n\nWe include the following ablation study on different values of T:\n\n| Model               | Top-1 (%) | T             |\n|---------------------|-----------|---------------|\n| MoViNet-A0-S-GRW    | 77.4†     | 5             |\n| MoViNet-A0-S-GRW    | 77.4      | 10            |\n| MoViNet-A0-S-GRW    | 75.4      | 2             |\n| MoViNet-A0-S        | 72.0      | no-smoothing  |\n†We include here our most current results, which improve upon the submitted version and **advance the SOTA by 5.4 pp**.  \n\nWe emphasize that the **training cost is small**. GRW-smoothing is applied to the frame embeddings $Z = (z_i)$, where each frame $x_i$ is significantly reduced in size. For example, in A2-GRW, the input frame $x_i$ has dimension 224×224×3, while $z_i$ has dimension 640—over 230× smaller. Thus, even with k = 1000, the cost of GRW-smoothing is negligible compared to the early layers of the model.\n\nMoreover, the wall-clock time of a single training epoch for A2-GRW is only 2% higher than A2 without smoothing, making the difference practically negligible.\n\nGRW-smoothing is not applied during inference, so there is no runtime overhead.\n\nWe thank the reviewer for this important comment and, if permitted, will clarify this in the final version.\n\n2. *“Frame Ordering\" Complexity: The introduction of a \"frame ordering\" contrastive loss to prevent degenerate solutions adds complexity. While necessary, a simpler or more efficient mechanism to achieve this goal could be explored in future work.”*\n\nWe agree that exploring simpler alternatives is a promising direction. As noted above, the current implementation incurs minimal cost relative to the model’s overall computation.\n\nWe also highlight that GRW-smoothing has desirable mathematical properties: it keeps the embeddings bounded within a ball of a certain radius, as proven in the supplementary material.\n\n3. *“The figures and tables of this work can be improved. Some figures are blurry and some tables take up too much space.”*\n\nWe appreciate this suggestion and will gladly revise the figures and tables to improve clarity and layout in the camera-ready version.\n\nReviewer Questions\n1. “Computational Overhead of GRW Loss. Could the authors provide more specific details on the computational overhead (e.g., FLOPs or actual runtime increase during training) introduced by the GRW loss term, especially considering the permutation sampling?”\n\nPlease see our response to Point 1.\n\n2. *“Sensitivity to Hyperparameters: How sensitive are the results to the choice of hyperparameters $\\lambda$ and $\\alpha$? The paper states robustness to perturbing $\\lambda$, but a more detailed analysis of the hyperparameter space for both parameters would be valuable.”*\n\nWe include the following ablation studies on different values of $\\lambda$ and $\\alpha$:\n\n| Model               | Top-1 (%) | α             |\n|---------------------|-----------|---------------|\n| MoViNet-A0-S-GRW    | 77.2      | 0.25          |\n| MoViNet-A0-S-GRW    | 77.4†     | 0.5           |\n| MoViNet-A0-S-GRW    | 76.9      | 1             |\n| MoViNet-A0-S-GRW    | 75.7      | 2             |\n| MoViNet-A0-S        | 72.0      | no-smoothing  |\n\n\n| Model               | Top-1 (%) | λ             |\n|---------------------|-----------|---------------|\n| MoViNet-A0-S-GRW    | 77.4      | 0.01          |\n| MoViNet-A0-S-GRW    | 77.4†     | 0.1           |\n| MoViNet-A0-S-GRW    | 77.3      | 1             |\n| MoViNet-A0-S        | 72.0      | no-smoothing  |\n\n\n\nWe hope these clarifications fully address your concerns. If they do, we would be grateful if you could consider updating your score to reflect the new information. Thank you again for your constructive feedback."}, {"review_id": "xuX1N9krqA", "reviewer": "Reviewer_hQEB", "summary": "This paper proposes a smooth regularization technique that instills a strong temporal inductive bias in video recognition models and validates it on the lightweight MoViNets model family, demonstrating the effectiveness of this technique.", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe proposed smooth regularization technique yields a 2.8%–6.0% accuracy improvement on Kinetics-600. The MoViNets model family trained with the smooth regularization improves the current state-of-the-art by 2.8%–3.4% within their respective FLOP constraints, while MobileNetV3 and the MoViNets-Stream family achieve gains of 2.8%–6.0% over prior state-of-the-art models with comparable memory footprints.\n\nWeaknesses:\n(1) The verification is mainly conducted on lightweight models. Is the smooth regularization technique also effective when verified on non - lightweight networks?\n(2) The latest base used currently is from 2021. It is suggested to add ablation verification of GRW on the latest base.\n(3) The current verification results are from fully - supervised scenarios. Can consistent effects be achieved in semi - supervised and few - shot scenarios?\n(4) Is this smoothing technique applicable to the network structure of transformers?", "was_revised": true, "final_justification": "After reviewing the authors' response, I have increased my rating from reject to borderline reject but the current state of the paper still falls short of borderline accept.", "rebuttal": "Thank you, Reviewer hQEB, for your careful review and constructive feedback. We appreciate the opportunity to clarify the points you raised. If the paper is accepted, we will make the appropriate changes in the camera-ready version.\nBelow, we provide a brief overview to contextualize our contribution, followed by point-by-point responses to your comments.\n\n**The scope** of this work is to show that temporal regularization, which steers optimization toward solutions that are (piecewise) smooth in time, is highly beneficial for building **efficient** video-recognition models. Although the method is general and can be applied across a wide range of architectures, its advantage is especially pronounced at the low-compute end of the spectrum. We will emphasize this in the introduction and, if the reviewer concurs, update the paper’s title to reflect this focus.\n\nThe results we present strongly substantiate our claims. Specifically, we **advance the state of the art on Kinetics-600 by 2.8–3.4 pp** Top-1 for all models in the ≤1 TFLOP range (Table 1). While some reviewers expressed concerns about comparisons with the most recent models, we compared against **every** published model within this budget to the best of our knowledge. If there are models that we missed, we would greatly appreciate it if you could point us to them.\n\nEfficient video recognition in this compute range is dominated by CNN and hybrid architectures. We therefore applied our GRW smoothing to the MoViNet family, released in 2021, because it remains the **strongest** lightweight backbone on Kinetics-600. For example, our MoViNet-A3-GRW achieves 84.8% Top-1 at only 56.4 GFLOPs. The first model that surpasses this accuracy, MViTv2-B-32×3, requires 1,030 GFLOPs—more than **18× greater computation**.\n\nFinally, we draw the reviewers’ attention to Video-Mamba (2024), the most recent sub-1 TFLOP model. It trails our MoViNet-A3-GRW by 3.8 pp Top-1 while consuming nearly twice the computational budget (see Table 3 and the typo correction in Supplement §1).\n\nPoint-by-point responses\n\n1. *“The verification is mainly conducted on lightweight models. Is the smooth regularization technique also effective when verified on non-lightweight networks?”*\n\nOur regularization technique may apply to non-lightweight video models; however, these typically require more than 2 DGX machines to train, which is the infrastructure that we have, and indeed much more to run several experiments. This is out of scope for many university labs.\n\n2. *“The latest base used currently is from 2021. It is suggested to add ablation verification of GRW on the latest base.”*\n\nThe 2021 MoViNet family holds the current state of the art on Kinetics for models with a compute budget ≤1 TFLOP. We compare against **all** models in that range (Table 1) and improve the SOTA by **2.8–3.4** pp Top-1.\nNo newer lightweight architecture is truly competitive in this regime. To the best of our knowledge, the most recently published lightweight model which is somewhat competitive to the SOTA is Video-Mamba (2024) (Table 3, ref. [15]), which reaches 76.9% Top-1 at 108 GFLOPs on Kinetics-400. By contrast, vanilla MoViNet-A3 already attains 78.2% Top-1 at 56.9 GFLOPs, and our MoViNet-A3-GRW pushes this to **80.7% Top-1 at 56.4 GFLOPs** (see Table 3 and the typo correction in Supplement §1). Thus, GRW provides a substantial gain even over the strongest current baseline.\n\n\n3. *“The current verification results are from fully-supervised scenarios. Can consistent effects be achieved in semi-supervised and few-shot scenarios?”*\n\nThe scope of this work is supervised learning. Investigating the applicability of GRW smoothing to semi-supervised learning is an exciting direction we are currently exploring in the lab.\n\n\n4. *“Is this smoothing technique applicable to the network structure of transformers?”*\n\nThe current work demonstrates the effectiveness of GRW smoothing on CNN architectures. CNN and hybrid architectures dominate accuracy in the ≤1 TFLOP regime, and we outperform all of them by 2.8–3.4 percentage points.\n\nTo the best of our knowledge, the only transformer models in this budget are MViT-B-16×4 (82.1 Top-1 at 353 GFLOPs) and MViT-B-32×3 (84.3 Top-1 at 850 GFLOPs). In comparison, MoViNet-A3-GRW achieves 84.8 Top-1 at just 56.4 GFLOPs—higher accuracy with roughly **6× and 15× less compute**, respectively.\n\nExtending GRW to transformer backbones is underway but was outside the scope of this submission.\n\n\nWe hope these clarifications fully address your concerns. If they do, we would be grateful if you could consider updating your score to reflect the new information. Thank you again for your constructive feedback."}, {"review_id": "1RNBCf2f4M", "reviewer": "Reviewer_nKeo", "summary": "This paper proposes a novel smooth regularization technique for video action recognition, aimed at enforcing temporal coherence in lightweight models. The key idea is to apply a Gaussian Random Walk (GRW) prior on the intermediate or final layer embeddings across frames, penalizing high accelerations in the feature space. The authors show that this inductive bias leads to improved temporal consistency and significant performance gains on the Kinetics-400 and Kinetics-600 datasets. The method is lightweight, architecture-agnostic, and shows consistent benefits across multiple efficient models such as MoViNets and MobileNetV3.", "questions": "I am interested in seeing the performance on downstream tasks. However, even if the results are not particularly strong, I will not lower my score because of that.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1) Novel concept. The GRW-based smoothing loss is a principled and intuitive formulation, encouraging low-acceleration transitions across temporal embeddings, which aligns well with the inherent continuity in natural videos.\n\n\n2) Method Simplicity and Generality. The proposed loss is simple to implement, requires minimal tuning, and can be seamlessly plugged into existing architectures.\n\n3) Well-Written and Clear Presentation:\nThe paper is well-structured, with clear motivation, visualizations (e.g., warm-up toy dataset), and an understandable exposition of the loss function.\n\n------------------------------------------------------------------------------\n\n\nWeaknesses:\n\n1)   The method is only evaluated on action classification with Kinetics datasets. It remains unclear how well it generalizes to other video understanding tasks (e.g., action detection, video captioning) or datasets with higher temporal discontinuities.\n\n2) Te paper lacks error bars or variance analysis across multiple training seeds. This could help solidify the claims, especially given the relatively small number of epochs and resource-constrained training setup.", "was_revised": true, "final_justification": "I am happy that the authors have addressed my concerns in their rebuttal.", "rebuttal": "We thank Reviewer 8e7P for the careful reading of our submission and the thoughtful feedback. Below we address each point in turn.\n\nPoint-by-Point Responses\n\n1. *“The method is only evaluated on action classification with Kinetics datasets. It remains unclear how well it generalizes to other video understanding tasks (e.g., action detection, video captioning) or datasets with higher temporal discontinuities.”*\n\nOur primary claim is that taking a trained model and applying GRW smoothing over short temporal windows, thereby making the representation **piecewise smooth**, yields more efficient models. To isolate that effect, we begin with publicly released weights and fine-tune with GRW.\nExtending to other tasks hinges on the availability of lightweight backbones with released checkpoints. Unfortunately, such weights are scarce for lightweight models. For example, the MoViNet family reports results on Charades, but releases checkpoints only for Kinetics. When weights are unavailable, reproducing a strong baseline requires extensive hyper-parameter tuning (weight decay, label smoothing, dropout, stochastic depth, batch size, LR schedule, layer-wise LRs, data augmentation, AutoAugment, etc.), which exceeds our compute budget (two DGX machines).\nFor these reasons, we focused on Kinetics-400/600, where strong baselines exist, allowing a clean evaluation of GRW smoothing. We will nevertheless attempt to add an additional classification dataset for the camera-ready version.\n\n2. “The paper lacks error bars or variance analysis across multiple training seeds. This could help solidify the claims, especially given the relatively small number of epochs and resource-constrained training setup.”\n\nVariance on Kinetics-400/600 is low. We trained MoViNet-A0-GRW four times with different seeds, obtaining Top-1 accuracies of 77.42, 77.36, 77.30, 77.30 (mean 77.35 ± 0.05). We will report these values with error bars in the camera-ready version.\n\n\nReviewer Questions: \n\n*”I am interested in seeing the performance on downstream tasks. However, even if the results are not particularly strong, I will not lower my score because of that.”*\n\nPlease see our response to the first comment above regarding downstream tasks and compute constraints.\n\nThank you again for your constructive feedback."}]

Author rebuttal:

Meta-review: The paper ended up receiving mixed reviewer with reviewer nKeo a strong champion for the work. The AC reviewed the reasons for rejection from reviewers YNFs and hQEB, and have not found any convincing arguments in their reviews. Below is a point by point look at those two reasons for rejection. 


The main weaknesses of the work that were called out by YNFs are as follows: 

>Method assumed smooth transition and should test on SSV2 to see what happens when this smoothness constraint is violated.

The AC disagrees with the analysis. First Kinetics does have many shot boundaries and the method still works well. Second SSV2 doesn't have any shot boundaries, just some fast motion. So Kinetics is a reasonable test for discontinuities (shot boundaries) and the method proves robust still. 

> Evaluation of the method is limited to Kinetics

The AC agrees that the paper would be stronger if more datasets were used to validate the method and prove its not over fit to the kinetics setting. The argument that the weights for SSV2 weren't released for a specific model is a little weak since the method could be tested on other OSS models with weights for SSV2.

>The comparison is only to out of data models

The reviewers comments were made without referencing the newer methods and did not point out newer models to compare to in follow ups. The authors pointed out that the "older" model is still SOTA over a number of more recent works and they still improve over this older model.

> Request for more ablations

Authors provide these ablations 

>Method is complex

The AC believe the authors do a good job justifying that the method is not overly complex to implment. 

> Related work section missing some discussion

The AC does not believe this is cause for rejection and the authors promise to improve it. There also doesn't seem to be some other method missing from related work that would make the proposed method irrelevant, which would have been a bigger concern.

>Poor writing/notation

Authors agree to fix

>Poor motivation of work

Not a reason to reject but the comments will help the authors create a better camera ready.

The AC thanks YNFs for their review but would have liked to see a bit more interaction with the authors as YNFs only said the changes to get the paper up to standard would be "too much". The lead the AC to believe that for the most part the rebuttal covered all needed points but YNFs just didn't believe the authors would actually make the changes outlined in the rebuttal.



The main weaknesses of the work that were called out by hQEB are as follows: 

> Experiment were done only on lightweight models

That is the target of the method so this is ok. Labs with constraints can show promising methods on small models, and other better funded labs can try to scale these methods.

>The comparison is only to out of data models (same as above)

The reviewers comments were made without referencing the newer methods and did not point out newer models to compare to in follow ups. The authors pointed out that the "older" model is still SOTA over a number of more recent works and they still improve over this older model.

>Should try method on semi-supervised

This could be a good idea but not a reason for rejections

>Should try on transformers

This could be a good idea but not a reason for rejections

Reviewer hQEB did not respond to the rebuttal at all and only gave a reasons of the paper is not in good state for remaining on the borderline reject side.

Conversely all reviewers and the AC acknowledge the strong results for low flop models. The proposed smooth regularization is proven effective on two model families showing it can be transfer. Even if the method proves to be only useful in the low compute domain this is an important problem space for embedded/smart devices. Additionally reviewer nKeo (the strong champion for the work) pointed out that the reasons for rejection given by the other reviewers seemed weak and the AC agrees. 

Considering this the AC recommends acceptance into NeurIPs. The authors are highly encouraged to take a full pass on the paper to fix all the presentation issues called out during the review process.
Decision: Accept (poster)