Title: Smooth Regularization for Efficient Video Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video recognition

Temporal smoothness regularization

Temporal coherence

Lightweight video models
Abstract: We propose a smooth regularization technique that instills a strong temporal inductive 
bias in video recognition models, particularly benefiting lightweight architectures. 
Our method encourages smoothness in the intermediate-layer embeddings
of consecutive frames by modeling their changes as a Gaussian Random Walk
(GRW). This penalizes abrupt representational shifts, thereby promoting low-
acceleration solutions that better align with the natural temporal coherence inherent
in videos. By leveraging this enforced smoothness, lightweight models can more
effectively capture complex temporal dynamics. Applied to such models, our 
technique yields a 3.8%–6.4% accuracy improvement on Kinetics-600. Notably, the
MoViNets model family trained with our smooth regularization improves the 
current state-of-the-art by 3.8%–6.1% within their respective FLOP constraints, while
MobileNetV3 and the MoViNets-Stream family achieve gains of 4.9%–6.4% over
prior state-of-the-art models with comparable memory footprints. Our code and
models are available at https://github.com/cmusatyalab/grw-smoothing.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel smooth regularization technique for video action recognition, aimed at enforcing temporal coherence in lightweight models. The key idea is to apply a Gaussian Random Walk (GRW) prior on the intermediate or final layer embeddings across frames, penalizing high accelerations in the feature space. The authors show that this inductive bias leads to improved temporal consistency and significant performance gains on the Kinetics-400 and Kinetics-600 datasets. The method is lightweight, architecture-agnostic, and shows consistent benefits across multiple efficient models such as MoViNets and MobileNetV3.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1) Novel concept. The GRW-based smoothing loss is a principled and intuitive formulation, encouraging low-acceleration transitions across temporal embeddings, which aligns well with the inherent continuity in natural videos.\n\n\n2) Method Simplicity and Generality. The proposed loss is simple to implement, requires minimal tuning, and can be seamlessly plugged into existing architectures.\n\n3) Well-Written and Clear Presentation:\nThe paper is well-structured, with clear motivation, visualizations (e.g., warm-up toy dataset), and an understandable exposition of the loss function.\n\n------------------------------------------------------------------------------\n\n\nWeaknesses:\n\n1)   The method is only evaluated on action classification with Kinetics datasets. It remains unclear how well it generalizes to other video understanding tasks (e.g., action detection, video captioning) or datasets with higher temporal discontinuities.\n\n2) Te paper lacks error bars or variance analysis across multiple training seeds. This could help solidify the claims, especially given the relatively small number of epochs and resource-constrained training setup.\n\nQUESTIONS:\nI am interested in seeing the performance on downstream tasks. However, even if the results are not particularly strong, I will not lower my score because of that.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a smooth regularization technique that instills a strong temporal inductive bias in video recognition models and validates it on the lightweight MoViNets model family, demonstrating the effectiveness of this technique.

STRENGTHS AND WEAKNESSES:
Strengths:
The proposed smooth regularization technique yields a 2.8%–6.0% accuracy improvement on Kinetics-600. The MoViNets model family trained with the smooth regularization improves the current state-of-the-art by 2.8%–3.4% within their respective FLOP constraints, while MobileNetV3 and the MoViNets-Stream family achieve gains of 2.8%–6.0% over prior state-of-the-art models with comparable memory footprints.

Weaknesses:
(1) The verification is mainly conducted on lightweight models. Is the smooth regularization technique also effective when verified on non - lightweight networks?
(2) The latest base used currently is from 2021. It is suggested to add ablation verification of GRW on the latest base.
(3) The current verification results are from fully - supervised scenarios. Can consistent effects be achieved in semi - supervised and few - shot scenarios?
(4) Is this smoothing technique applicable to the network structure of transformers?

QUESTIONS:
see weaknesses

LIMITATIONS:
see weaknesses
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel smooth regularization technique designed to enhance temporal understanding and improve the efficiency of video recognition models, especially lightweight architectures. The core idea is to instill a strong temporal inductive bias by modeling changes in intermediate-layer embeddings of consecutive frames as a Gaussian Random Walk (GRW). This regularization penalizes abrupt representational shifts, encouraging smoother, more temporally coherent feature evolutions. The method is presented as a plug-and-play loss term that can be applied to either intermediate or final layers. Extensive experiments on the Kinetics-600 and Kinetics-400 datasets demonstrate that this GRW-smoothing consistently yields significant accuracy improvements, establishing new state-of-the-art performance within their respective FLOP and memory constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Novel and Intuitive Regularization. The concept of enforcing smoothness in video frame embeddings through a Gaussian Random Walk (GRW) is novel, intuitive, and well-motivated by the inherent temporal coherence of video data. This provides a strong inductive bias.\n\n2. Significant Performance Gains for Lightweight Models. The paper demonstrates impressive and consistent accuracy improvements (2.8%-6.0%) on various efficient architectures (MoViNets, MobileNetV3) on standard benchmarks (Kinetics-600, Kinetics-400). This directly addresses the practical challenge of deploying video models on resource-constrained devices.\n\n3. Plug-and-Play Framework. The GRW regularization is designed as a simple, additive loss term, making it easy to integrate into existing video recognition pipelines with negligible computational overhead. This enhances its practical applicability.\n\nWeaknesses\n\n1. Computational Cost of Permutations. The loss function in Equation (5) involves a sum over all permutations of frames within a sub-clip, which grows exponentially with the sub-clip length T. While the authors mention sampling k permutations for large T, the computational overhead of this sampling and the contrastive loss itself could still be substantial, especially during training. More details on the practical values of T and k used in experiments and their impact on training time would be beneficial.\n\n2. \"Frame Ordering\" Complexity: The introduction of a \"frame ordering\" contrastive loss to prevent degenerate solutions adds complexity. While necessary, a simpler or more efficient mechanism to achieve this goal could be explored in future work.\n\n3. The figures and tables of this work can be improved. Some figures are blurry and some tables take up too much space.\n\nQUESTIONS:\n1. Computational Overhead of GRW Loss. Could the authors provide more specific details on the computational overhead (e.g., FLOPs or actual runtime increase during training) introduced by the GRW loss term, especially considering the permutation sampling?\n2. Sensitivity to Hyperparameters: How sensitive are the results to the choice of hyperparameters \\lambda and \\alpha? The paper states robustness to perturbing \\lambda, but a more detailed analysis of the hyperparameter space for both parameters would be valuable.\n\nLIMITATIONS:\nThe authors has explicitly discussed limitations in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a smooth regularization method specifically tailored for videos. The method relies on the idea that the consecutive frames in a video have a lot of redundancy and this, therefore, should be reflected in how the embeddings also change. Gaussian random walk is used to model the frame embedding evolution and to avoid sudden representational changes in the embeddings. This in theoery should encourage slow developing (low-accelations) solutions that are aligned with video data. The evaluations are provided on k400 and k600 datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nOverall, the idea is simple yet intriguing. \nThe paper is written decently and is overall easy to read and understand. \n\n\nWeaknesses:\nThe following weaknesses are not in the order of importance:\n1. The related works section is quite lacking and not comprehensive. It does not talk about other temporal regularization techniques at all which have been explored in this domain quite extensively. Moreover, the realted works section also doesnt explore the lightweight networks literature which is the setting for the proposed paper. \n\n2. From the writing point of view, the mathematical symbols introduced lack the definitions (normally given below as soon as the symbol/variable is introduced). E.g. line 96 introduces X but does not define M or N and later C in the equation 1. There other instances in the rest of the paper as well. \n\n3. The introduction lacks the buildup of the motivation and the problem at hand. This can be made better. \n\n4. A big issue with the approach is it's assumption of smooth transitions between the frames in the videos. This may be true for some applications (e.g. surveillance/security footage etc), but not for most video content, which has sudden transitions. For example, I would be very interesed in knowing the results on the SSV2 dataset and other temporally challenging datasets. Even in normal videos encoded with prevalent video codecs, it may be reasonable to assume the smoothness in the \"p-frames\" but there are also I-frames where the difference is simply too huge to be encoded as a p/b frame. This proposed technique won't work good with fast moving objects, scenes or other videos of this nature.\n\n5. The evaluations are quite small and need to be expanded to other video datasets such as SSV2 and can also be shown to work with localization and detections datasets as well to show the effectiveness of the proposed technique. \n\n6. The additional complexity introduced by frame ordering component in the contrastive loss is not quantified. \n\n7. The comparison methods/models are quite old...\n\n8. Ablation on lambda (eq 7) and alpha (eq 6) is not provided.\n\nQUESTIONS:\nFollowing the sake of brevity, i will refer to the points mentioned in the weakeness section. \n\nHow can the proposed technique be adapted to account for the sudden temporal changes? (weaknesses: point 4)\nWhy are the evaluations on SSV2 missing? \nWhy is the method not compared with latest state of the art methods? (point 7)\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-03"}