Title: SpikingVTG: A Spiking Detection Transformer for Video Temporal Grounding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spiking Neural Networks

Efficient AI
Abstract: Video Temporal Grounding (VTG) aims to retrieve precise temporal segments in a video conditioned on natural language queries. Unlike conventional neural frameworks that rely heavily on computationally expensive dense matrix multiplications, Spiking Neural Networks (SNNs)—previously underexplored in this domain—offer a unique opportunity to tackle VTG tasks through bio-plausible spike-based communication and an event-driven accumulation-based computational paradigm. We introduce SpikingVTG, a multi-modal spiking detection transformer, designed to harness the computational simplicity and sparsity of SNNs for VTG tasks. Leveraging the temporal dynamics of SNNs, our model introduces a Saliency Feedback Gating (SFG) mechanism that assigns dynamic saliency scores to video clips and applies multiplicative gating to highlight relevant clips while suppressing less informative ones. SFG enhances performance and reduces computational overhead by minimizing neural activity. We analyze the layer-wise convergence dynamics of SFG-enabled model and apply implicit differentiation at equilibrium to enable efficient, BPTT-free training. To improve generalization and maximize performance, we enable knowledge transfer by optimizing a Cos-L2 representation matching loss that aligns the layer-wise representation and attention maps of a non-spiking teacher with those of our student SpikingVTG. Additionally, we present Normalization-Free (NF)-SpikingVTG, which eliminates non-local operations like softmax and layer normalization, and an extremely quantized 1-bit (NF)-SpikingVTG variant for potential deployment on edge devices. Our models achieve competitive results on QVHighlights, Charades-STA, TACoS, and YouTube Highlights, establishing a strong baseline for multi-modal spiking VTG solutions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SpikingVTG, a novel SNN-based architecture designed for the Video Temporal Grounding (VTG) task. The method segments a video into fixed-length clips and predicts, for each clip, three values: a binary foreground indicator, a temporal offset pair, and a relevance score.\nA key contribution is the Saliency Feedback Gating (SFG) mechanism, which uses intermediate outputs to compute dynamic saliency scores and update video input features over discrete timesteps. The paper also formulates the convergence dynamics of the spiking transformer to steady-state ASR values and utilizes these in the training and inference pipeline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThis is likely the first attempt to apply spiking neural networks to the VTG domain, introducing a new perspective on low-power temporal reasoning tasks.\nThe SFG mechanism is biologically inspired and dynamically adjusts input features based on feedback from average spiking rates (ASR), allowing for adaptive and sparse computation.\nWeakness\nAssumption of Equilibrium: While the paper claims that ASR values and saliency scores converge to equilibrium and provides partial visualizations, it lacks a rigorous empirical or theoretical analysis of the convergence process. A more precise treatment of convergence criteria, dynamics, generalizability, and other related aspects would strengthen the claims made in this section.\nFeature Space Compatibility: In the SFG module, the cosine similarity is computed between pooled text representations M and ASRs from deep layers a_i^{Nv[t]}. However, the compatibility between these feature types (from different depths and modalities) is not discussed.\nGating Mechanism Scalability: The computation of cosine similarity per clip and timestep may add overhead, and the potential latency of SFG as a feedback loop in larger-scale settings is not discussed.\n\nQUESTIONS:\nCould the authors clarify how convergence to equilibrium is defined and detected during training?\nHow do the authors ensure the compatibility of textual features and deep-layer video ASRs in the cosine similarity calculation within SFG?\nHas the computational cost of SFG (per-timestep cosine similarity) been evaluated on longer video sequences or real-time settings?\nIs the SFG module quantized in the 1-bit NF-SpikingVTG model? If not, could its floating-point operations impact the claimed energy efficiency?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SpikingVTG, a novel Spiking Neural Network based model designed for Video Temporal Grounding tasks, which aims to identify precise temporal segments in a video corresponding to natural language queries.\nThe SpikingVTG architecture consists of three key components: a spiking Transformer core, a Saliency Feedback Gating mechanism, and a spiking decoder for output generation. The spiking Transformer core utilizes Leaky Integrate-and-Fire neurons as its fundamental computational units and replaces the traditional floating-point multiply-accumulate operations in standard Transformers with more efficient floating-point accumulate-only operations, enabling it to capture temporal and cross-modal dependencies effectively.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The paper introduces a novel application of Spiking Neural Networks (SNNs) to the domain of Video Temporal Grounding (VTG), which has been traditionally dominated by conventional neural networks. This innovation leverages the computational efficiency and sparsity of SNNs, offering a bio-plausible and energy-efficient alternative.\n* By exploiting the inherent sparsity and accumulation-based computation of SNNs, the model achieves substantial reductions in computational overhead and energy consumption. The introduction of normalization-free and 1-bit quantized variants further underscores the model’s suitability for deployment on resource-constrained edge devices, directly addressing a critical practical challenge in VTG.\n* The authors provide extensive experimental results, including ablation studies and comparisons with state-of-the-art non-spiking models. This comprehensive evaluation supports the claims made in the paper and provides insights into the model's behavior.\n\nWeaknesses\n* The paper does not provide in-depth discussion or ablation on how the spiking framework handles the fusion of video and language modalities, which is a non-trivial challenge in VTG. The reader is left to assume that the spiking transformer core adequately captures cross-modal dependencies, but more detailed analysis or visualization would strengthen the claim.\n* Although the SFG mechanism and other architectural innovations are well-motivated, the paper lacks thorough ablation studies to quantify the individual contributions of each component (e.g., normalization-free design, Cos-L2 transfer). This makes it difficult to assess which parts of the design are most critical to performance.\n* The experimental section primarily compares SpikingVTG to non-spiking transformer models. It does not benchmark against other lightweight, quantized, or pruned VTG architectures, which would be necessary to convincingly demonstrate the unique advantages of SNNs in this context.\n* The model is positioned as suitable for edge deployment, but the paper does not report real-time inference latency or throughput on actual hardware. Without these metrics, the practical viability for time-sensitive applications remains somewhat speculative.\n* While the paper claims significant energy efficiency, the methodology for measuring energy consumption and the hardware context are not described in sufficient detail. This lack of transparency may limit reproducibility and make it difficult to directly compare with other efficient VTG models.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SpikingVTG, a spiking neural network based architecture for video temporal grounding tasks. Unlike conventional transformers, it uses spike-based, event-driven computation for higher energy efficiency. Key innovations include a Saliency Feedback Gating mechanism to focus on relevant video clips, a knowledge transfer method (CLRM) from non-spiking models, and quantized, normalization-free variants for edge deployment. The results on the benchmarks shows good performance at low cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is intuitive and driven by biologically motivated. The Saliency Feedback Gating (SFG) re-uses the network’s own spiking activity as a guiding signal, boosting meaningful spikes in clips. \n2. The 1-bit version fo the model is impressive with only 11MB  in size and reduces one-pass energy to 1.3 mJ, this has true potential to be deployed to the edge devices.\n\nWeaknesses\n1. the proposed method, to match the performance, relies on a heavy teacher models. This multi-stage pipeline (teacher > CLRM > fine-tune) adds additional computes and makes it not clear if the proposed architecture is effective or the distillation part.\n2. The performance of proposed model Still lags behind the SOTA. I would encourage the authors to provide more in-depth analysis of the errors, and the limitation of current models. \n3. Seems the efficiency claims come from FLOP-based calculations with cost model; this is not very convincing, the author could simply use GPU energy consumption to estimate the efficiency.\n\nQUESTIONS:\n1. seems the model accuracy depends on Cos-L2 Representation Matching with a heavyweight teacher; without CLRM the performance drops about 3%on qvh. How important is the teacher model, what if we use a smaller teacher, is the the SNN architecture itself can learn strong multimodal representations or it has to rely on the knowledge distillation?\n2. The SFG mechanism produces saliency scores based on average spiking rates, but there is no visualize or explain why certain clips are gated.  Please further explain if certain behaviors are explainable or the behavior of SFG are purely empirical.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SpikingVTG, a novel spiking neural network-based detection transformer designed for Video Temporal Grounding (VTG). The model features a bio-inspired, energy-efficient architecture comprising a spiking transformer core, a Saliency Feedback Gating (SFG) mechanism for relevance-aware filtering, and a spiking decoder for output generation. To enhance generalization, the authors introduce a Cos-L2 Representation Matching (CLRM) loss for knowledge transfer from pre-trained non-spiking models. Furthermore, they develop normalization-free and 1-bit quantized variants for deployment on resource-constrained edge devices. Experimental results across four VTG benchmarks (QVHighlights, Charades-STA, TACoS, and YouTube Highlights) demonstrate competitive performance and notable energy savings, establishing a solid foundation for future research on spiking-based VTG systems.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper introduces the first spiking transformer architecture for VTG, replacing conventional dense matrix multiplications with event-driven, accumulation-based operations using leaky integrate-and-fire neurons. This design aligns well with neuromorphic computing principles and targets the high energy cost of traditional VTG models.\nThe proposed Saliency Feedback Gating (SFG) mechanism enables dynamic focus on query-relevant video segments using spike-based saliency estimation. Combined with the use of implicit differentiation for BPTT-free training, the model achieves both efficiency and scalability.\nThe experimental evaluation is comprehensive, spanning four VTG datasets with multiple model variants (including normalization-free and 1-bit versions). The authors conduct thorough ablation studies and analyze the energy-accuracy tradeoff, enhancing the paper's empirical depth and practical relevance.\n\n**Weaknesses**\n\nDespite its architectural efficiency, the proposed model underperforms compared to state-of-the-art pretrained non-spiking VTG models (e.g., UniVTG), especially at higher IoU thresholds. This may limit its adoption in applications requiring high localization precision.\nThe SFG module introduces additional floating-point operations, but its computational overhead is not fully quantified or compared against the energy gains, making its efficiency claims less conclusive.\nWhile the authors assert that the model is \"well-suited for edge-based deployment,\" no experiments are conducted on actual edge or neuromorphic hardware. This lack of validation undermines the strength of the deployment claim.\nAll energy efficiency results are estimated via theoretical cost models. The absence of real-device power profiling or runtime measurements limits the practical credibility of the claimed energy savings.\n\nQUESTIONS:\nThe traditional neural networks like CNN, transformer can be accelerated by hardware like GPUs with CUDA. Is their any matured and popular hardware and hardware acceleration libs to support Spiking Neural Networks (SNNs)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}