SpikingVTG: A Spiking Detection Transformer for Video Temporal Grounding
Authors: Malyaban Bal
Brian Matejek
Susmit Jha
Adam D. Cobb
Venue: NeurIPS 2025 poster ()
Keywords: Spiking Neural Networks
Efficient AI
Primary area: other
TL;DR: A spiking detection transformer for video temporal grounding tasks.

Abstract: Video Temporal Grounding (VTG) aims to retrieve precise temporal segments in a video conditioned on natural language queries. Unlike conventional neural frameworks that rely heavily on computationally expensive dense matrix multiplications, Spiking Neural Networks (SNNs)—previously underexplored in this domain—offer a unique opportunity to tackle VTG tasks through bio-plausible spike-based communication and an event-driven accumulation-based computational paradigm. We introduce SpikingVTG, a multi-modal spiking detection transformer, designed to harness the computational simplicity and sparsity of SNNs for VTG tasks. Leveraging the temporal dynamics of SNNs, our model introduces a Saliency Feedback Gating (SFG) mechanism that assigns dynamic saliency scores to video clips and applies multiplicative gating to highlight relevant clips while suppressing less informative ones. SFG enhances performance and reduces computational overhead by minimizing neural activity. We analyze the layer-wise convergence dynamics of SFG-enabled model and apply implicit differentiation at equilibrium to enable efficient, BPTT-free training. To improve generalization and maximize performance, we enable knowledge transfer by optimizing a Cos-L2 representation matching loss that aligns the layer-wise representation and attention maps of a non-spiking teacher with those of our student SpikingVTG. Additionally, we present Normalization-Free (NF)-SpikingVTG, which eliminates non-local operations like softmax and layer normalization, and an extremely quantized 1-bit (NF)-SpikingVTG variant for potential deployment on edge devices. Our models achieve competitive results on QVHighlights, Charades-STA, TACoS, and YouTube Highlights, establishing a strong baseline for multi-modal spiking VTG solutions.

SpikingVTG: A Spiking Detection Transformer for Video Temporal Grounding
Malyaban B Brian Matejek Susmit Jha?, Adam Cobb²
'The Pennsylvania State University Computer Science Laboratory, SRI International
Abstract
Video Temporal Grounding (VTG) aims to retrieve precise temporal segments in a video conditioned on natural language queries. Unlike conventional neural frameworks that rely heavily computationally expensive dense matrix multi- plications, Spiking Neural Networks (NNNs)-preiiouss) underexplored in this domain-offer a unique opportunity to tackle VTG tasks through bio-plau tio-pluusible spike-based communication and an event-driven aecumulation-basee computa- tional paradigm. We introduce SpikingVTG, multi-modal spiking detection transformer, designed to harness the computational simplicity and sparsity of SNNs for VTG tasks. Leveraging the temporal dynamics of SNNs, our model intro- duces Saliency Feedback Gating (SFG) mechanism that assigns dynamic saliency scores to video clips and applies multiplicative gating highlight relevant clips while suppressing less informative ones. SFG enhances performance and reduces computational overhead by minimizing neural activity. We analyze the layer- wise convergence dynamics of SFG-enabled model and apply implicit differentiation at equilibrium to enable efficient, BPTT-free training. To improve generalization and maximize performance, we enable knowledge transfer by optimizing Cos-L2 representation matching loss that aligns the layer-wise representation and attention maps of non-spiking teacher with those of our student Spiking VTG. Additionally, we present Normalization Free (NF)-SpikingV') VTG, which eliminates non-local operations like softmax and layer normalization, and extremely quantized 1-bit (NF)-SpikingVTT) variant for potential deployment on edge devices. Our models achieve competitive results on HViighiights, Charades-STA, TACoS, and YouTube Highlights, establishing a strong baseline for multi-modal spiking VTG solutions.
Introduction
The rapid expansion of various social medias and portable smart technologies has triggered an unprecedented surge in video content. This vast influx data has intensified the need for efficient methods to retrieve and analyze video information. Consequently, the field of Video Temporal Grounding (VTG) [ has emerged as an important area of research. The main objective of VTG is to identify the precise segment of video that corresponds accurate and context-driven video content retrieval. In this paper, we focus on two tasks: moment retrieval 2 3. which aims to identify video intervals relevant given query, and highlight detection []. which retrieves the best candidate segment of the video response to the query. Our work involves analyzing multimodal data-combinin video content with natural language queries-to develop an effective solution to the problem. With the rise of transformer based architectures the field of VTG has seen significant advancements 5 However, these models demand substantial
given natural language query, enabling
Work completed while SRI.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

power and energy 7 to operate. Furthermore, VTG is inherently resource-intensive requiring the analysis of long video sequences, leading significant computational overhead. Additionally, applications such as edge-based event detection in video e.g., identifying accidents from traffic cameras-requit deploying VTG models on resource-constrained devices near the data source, in order to reduce data transfer to the cloud. These devices often operate under limited energy availability. Inspired by neural dynamics in the brain, this paper leverages bio-plausible neuronal models and learning dynamics to develop an efficient and brain-inspired solution for VTG tasks. Our model enables flexible inference-time tradeoff between energy consumption and accuracy, making it well-suited for edge-based deployment in VTG tasks
We introduce SpikingVTG, spiking detection transformer designed for efficient and accurate VTG. Built on the sparse, event-driven communication and accumulation- based computation of Spiking Neural Networks (SNNs) R) Spiking leverages the intrinsic dynamics of SNNs to offer a lightweight yet competitive alternative to conventional transformer- based eliminntnnn the need for costly dense, real-valued matrix multiplications. The architecture comprises three key components: (i) spiking transformer core, (ii) Saliency Feedback Gating (SFG) mechanism, and (iii) spiking decoder for output generation. The spiking transformer captures temporal and tross-modal dependencies, while the decoder produces task-specific outputs. The SFG mechanism addresses a critical challenge unique to VTG: given an input video composed of many clips, how can we identify those most relevant to the query? Tailored specifically for multi-modal VTG tasks, the SFG module leverages the temporal dynamics of SNNs to attend more towards the most salient segments while suppressing irrelevant clips.
SpikingVTG Model:
Saliency Feedback Gating Mechanism: Operating over discrete time steps, Spiking uses the intermediate spiking activity of the transformer core to dynamically estimate the relevance of each video segment. We compute a feedback-base saliency score for each segment based on the average spiking rate (ASR) of the output of the transformer, conditioned on the query. These scores serve as soft attention masks within a multiplicative gating mechanism, suppressing less informative segments to reduce computational overhead while enhancing focus on the most relevant candidate clips. From a neuroscience perspective, feedback connections are known to play a critical role in object recognition in the visual cortex 19]. Furthermore, our feedback mechanism maintains layer-wise convergence of ASR to equilibrium, enabling us to adopt an efficient training mechanism leveraging the equilibrium dynamies [TO.. This approach circumvents the need for computationally intensive BPTT [11] and instead updates model parameters using a single backward pass, significantly improving memory efficiency.
Cos-L2 Representation Matching (CLRM): While our base Spiking VTG model achieves perfor- mance comparable to non-pretrained, non-spiking VTG models such as M-DETR and UniVTG [1. achieving state-of-the-arr results on VTG tasks typically requires transformer-based non-spiking models like UniVTG to undergo extensive pre-training. This pre-training significantly boosts their generalization and task-specific capabilities. To enable our SpikingVTG to benefit from similar pre- pre-training advantages--wihhe
incurring the high computational cost of training on large-scale datasets like Ego4D, Video-CC [12] we propose a direction and scale aware CLRM loss for efficient knowledge transfer. Optimizing CLRM loss aligns the hidden states and attention score maps of a pre-trained non-spiking multi-modal transformer with the hidden state converged ASR and mean attention scores of our Spiking VTG model. By minimizing this alignment loss on the downstream task, we allow the student model to imbibe the generalization capability learnt by the pre-trained teacher without having to perform the extensive pre-training from scratch.
Optimizations and Application to VTG Tasks: Traditional transformer-base VTG solutions 13) rely heavily non-local normalization operations, such softmax and layer normalization, which pose significant challenges for efficient implementation on ncuromorphic hardware [4). To address this limitation, which eliminates all layer normalization operations and substitutes softmax spiking attention with a ReLU and scaling- based spiking attention mechanism. Although alternative Softmax-fit free attention approaches have been explored in the literature [15] they have primarily been applied uni-modal vision tasks. We empirically demonstrate that while these optimizations enhance computational efficiency, they result in minimal performance degradation.
develop and evaluate Normalization-Free (NF)-SpikingVTC model,
To further reduce computational complexity and memory footprint [7 18 introduce 1-bit (NF))-pikingVTT VTG model, which rely primarily on integer accumulation operations and eliminate
2

---

QUER Person Entering A foom
Feedback Connection
Spiking Transtormer Block
Spiking Decoder
Paruna Towe Oway
Sallency Food dback Geting
SuPe Boreets
aldry Meneeeaaeee
LAP PPP
Hightight
Moment Retrieval
Sau
Couc 1br1
Repeat times
Reanat
LIF Neurons
Figure 1: High-level overview of the proposed Spiking VTG model. The model employs a spiking transformer core that utilizes Saliency Feedback Gating through temporal feedback connections. The model also incorporates spiking decoder module that takes the output of the transformer core to predict parameters for the VTG task.
all non-local operations. This design makes it well-suited for deployment on resource-constrainee edge devices. To our knowledge, this work is the first to evaluate an operational spiking detection transformer across VTG tasks, including moment retrieval and highlight detection, on datasets such as QVHighlights, Charades-STA TACoS and Youtube Highlights. To further highlight the benefits of using SpikingVTG, we present energy-accuran tradeoff analysis. This demonstrates that model is well-suited for deployment on edge devices, where its performance can be dynamically adjusted based on the available energy budget.
2 Related Works
VTG Advancements:
Moment-DETR D. a transformer encoder-decoder model introduced alongside the QVHighlights dataset, laid strong foundation for subsequent VTG architectures. UMT [5] introduced an unified framework for solving both highlight detection and moment retrieval tasks. Due to the limited availability of trainable video data, UniVTG [6] proposed innovative solution by unifying various VTG tasks and labels under a single formulation. This enables the development of an LLM-like pretraining framework, achieving state-of- ooee-offthh of-the-art performance VTG tasks. Although baased
fully spiking- based model has been explored for VTG tasks, SpikeMba [19 primarily a non-spiking model-integrates SNN components generate proposal sets from video data. However, since its core framework is derived from Mamba (0) and relies on footingg-ooin matrix multiplications, SpikeMba cannot be considered baseline for spiking models, which predominantly accumulation- based operations. While recent VTG models have significantly improved task-specific performance, adopting spiking framework enables us to leverage solutions on edge devices with limited or dynamic energy supply.
tradeoff madeoff-making suitable
Spiking neural networks (SNNs): SNNs have been implemented in neuromorphic systems like IBM TrueNorth (21 and Intel Loihi 2 [22], demonstrating approximately 75x greater energy efficiency compared to traditional networks running on low-power GPUs 23 SNNs, with their energy -efigy-efficient computational framework, offer a promising solution to the resource- resource-intensive demands of multimodal VTG tasks. While SNNs for long time were confined in simpler vision-based tasks (24] with simple architectures, recent developments have scaled them to transformer-basee
models for tasks ranging
from vision to language modelling (25 26 27]. however majority of them uni-modal and rely on non-local operations not implementable on a neuromorphic chip.
3 Video Temporal Grounding (VTG)
For given video V and language query Q, we start by segmenting V into a sequence of L. fixed- length clips, denoted as (vi...., UL. }. Each clip v; has a length and is centered at timestamp t, The textual query Q consists of Ly tokens, denoted as {q), (q)...., qr.. }. Following previous studies on VTG (6), define three parameters for each clip i (fr, d, st), where f. = 1 if the
3

---

clip is in foreground, i.e. relevant else f: = 0. d; [ds,, de.] ] E R represent the temporal distance that converts the clip timestamp i, to its interval boundaries. Here, d, valid when ft The term du; denotes the distance between the start of the interval and 4, while de, denotes the distance between the end of the interval and 4, & E [0, 1] a continuous score that quantifies the relevance between the visual content of clip i and the query Q. Our proposed Spiking VTG predicts these three parameters for each video clip. this paper, we focus on the following VTG tasks:
Moment Retrieval: We rank the predicted clip boundaries {b+) where b, [4 da,, 4 + deil.
based on their associated probabilities given by ( Since the predicted L, boundaries are dense, we employ a I-dimensional Non-Maximum Suppression (NMS) [28) with threshold of 0.7 to eliminate highly overlapping boundary boxes, resulting in a final prediction.
Highlight Detection For each clip, we rank all clips based on their combined scores {f. + + M This combined value represents how well the chip match with the underlying query. We then return the top clips (e.g., Top-1) 1) as predictions.
SpikingVTG: Architecture Overview
The core computational unit of the proposed Spiking VTG model is a leaky integrate-and-fir (LIF) neuron [29]. Neurons communicate with each other using sparse, spike-based activations instead of real-valued signals, thus we can replace floating-pont point matrix multiplications with accumulative operations, resulting in improved computational efficiency.
4.1 Spiking Neural Networks
The discrete time dynamics of an LIF-based spiking neuron can be given as follows,
u. [t + 8] yu,[t] + W(: (3(- 1)(t) by. ui(l 1] u.ll + 6] Ven, si 1].
(I)
at = Σt x
7
where, at time t, u;[f] is the membrane potential of the 1th neuronal layer, b, indicates a bias term and y is the leaky term. W(--1) represents the layer-specific operation; 4- 8 is an intermediate time step determine if the neuron fired; Vih, is the threshold of layer i. We use ternary spiking model (30 in our work for spike (s(t- + 1)) generation. This improves performance while avoiding the introduction of additional floating-pon -point multiplicative and accumulative (FP-MAC) operations. The average spiking rate (ASR (t) of neurons within each layer at time can be defined as a weighted-average function.
4.2 Spiking Transformer Core
The high-level overview of each encoder block of our spiking transformer architecture is demonstrated in Fig. [] The model consists of N encoder layers, each consists of spiking multi-headed attention block, followed by an intermediate layer and an output layer. Communication within and between encoder layers occurs via spikes. Furthermore, all matrix multiplications involved linear layers and attention layer comprises of more efficient fp-accumulative (FP--CCC) operations instead of FP-MAC operations in conventional neural architectures. Detailed descriptions of each layer are provided in the Appendix A Following architectural optimizations (Section (.6), we replace softmax-bai based attention with ReLU and scaling- basedd spiking attention mechanism, remove all layer normalization operations, and explore extreme quantization of linear weights.
4.3 Saliency Feedback Gating (SFG)
SpikingVTCC operates over a specific number of convergence time steps (T,), with the convergence dynamics detailed in Section 4.5 This temporal processing allows to leverage intermediate temporal outputs to dynamically update the input to the model at every time step for improved performance. This approach conforms to the feedback connections observed in the human visual

---

cortex (9) providing bio-plausible explanation for its efficacy. The ASR of the final encoder layer of the Spiking Transformer core is used as a temporal feedback to compute dynamic saliency score with the input query enabling the design of a gating mechanism (Fig. 21), allowing selective focusing on relevant segments of the video while minimizing computation on irrelevant segments. The saliency feedback gating mechanism is shown below,
al, [t] - M
F [t] cos( aN [t] M) ||a|,, [t]]la |
(2)
VIL+ + 1] V[[ * [1
SVG Q aN.) ) = Vtt + 1] a Q,
where, using attentive pooling operation, sentence representation M Q Softmar p). M RD, input textual query features Q xD, input video features V R and W D is learnable embedding and D is the hidden dimension. F [t] is the dynamic saliency score, at time t, for the i-th segment of the video. We compute F by applying min-max normalization to F,,, allowing us to obtain per-clip scores within the range [0, 1]. The ASR of the output of the spiking transformer core is given as a E Rl,++ "x [f] is ASR of output of the spiking transformer core, corresponding to video segment i, at time t. The output of SFG module the concatenation of saliency feedback gated video features and query features and serves as the input to the spiking transformer core at time f
To ensure compatibility between the two modalities during cosine similarity computation, both repre- sentations are projected into shared latent space of dimensionality D. The query representation is aggregated into global sentence embedding via attention-based pooling with learnable embedding vector. This adaptive mapping aligns the textual representation with the spatiotemporal semanties of the spiking video features, thereby facilitating effective cross moosl compatibility.
SFG Computational Overhead: The SFG layer comprises O D) floating- peiiingg-oonttto multiplication operations; however, the computational overhead of this layer is significantly less than that of the transformer core which has complexity of ( D + L D²), where I. Top + Tug.
4.3.1 Visualizing effect of SFG:
The dynamic saliency score (FP) achieves equilibrium value F fol- lowing the convergence of ASR the neuronal layers of the SpikingVTG model (Fig. 3). As shown in Fig. 2p, we empirically analyze the scores per clip at equilibrium to gain in- sights into the functioning of the SFG based multiplicative gating mecha- nism. Neighboring salient clips of the target clip exhibit higher F., scores equilibrium, while irrelevant clips show lower scores, highlighting the effectiveness of the SFG mechanism.
SALIENCY FEEDBACK GATING (SPG)
Higher,, Noartry popeernn
Score (F22) at Equilbbruu
Pooker
dkedurty
Elemastrvlae
Video Features
Y-Quaou
Geneet
32
Text
Fenturen
10 15 20
25
130
35
45 50 55
Video Cip (b)
Inclex
(6)
Figure 2: (a) Overview of the internal operations of the saliency-
gating mechanism. The ASR of the output of the spiking transformer core at each time step is leveraged as the feedback signal (Fig. D.. (b) Heatmap showing the scores per clip (F.) at equilibrium, with the target frame for highlight detection corresponding to clip index 28.
The SFG mechanism not only results in better performance of our Spik- VTG architecture (Table (5) but also reduces overall neural activity by sparsifying input spikes. Empirical
results (Fig. (3) confirm that the model with the gating mechanism exhibits a lower neural activity particularly in the input and spiking attention layers, compared to the model without SFG.
4.4 Spiking Decoder Module
The spiking decoder comprises of stacked 1-D convolutions followed by integrate-fire (IF) neuron layers (7 = 1 in Eqn. [. for spike generation. The spiking decoder used for predicting foreground indicator (f+) per clip, applies n1 1-D convolution operations with kernel size k1, each followed by an
5

---

IF layer. The final layer consists of single output channel. and its temporal mean is passed through sigmoid activation produce the prediction. The spiking decoder used for d, applies n2 1-D convolution operations with kernel size k2, each followed by an IF layer, and the final convolution layer has two output channels to predict d, (dx;, dez], after which we compute b;-
4.5 SpikingVTG: Convergence Dynamics
The membrane potential (u1) of the input LIF layer to the spiking transformer core, following the SFG mechanism can be formulated as below.
u1 [t 1| = yu [t] + SFG(V, Q, aN. a.. + Ven, s1 +
(3)
where, aN [4] is the ASR of the final layer corresponding to the video features at time I, V is the input
video features, Q is the query features and SFG is defined in Eqn. 2 The layer-wise convergence
dynamics of the SpikingVTG with SFG is demonstrated in Fig.
Following Eqn. 1 the layer-wise ASR is, a, 1] 1 (f(a 1])
m.. [t]+++]] Σ
where,
Vin,
is operation of layer i. Following empirical evidence (Fig. 3) and theoretical formulation 10 26 as time + oo, the layer-wise ASRs converge equilibrium, enabling the derivation of layer-wise steady-state equations given as,
of = ol (
(f(at11) + b,))
Vehi
The equation describing the steady-state ASR dynamics of the input layer is given as,
ai = ( Viha
(SFG(V, ax, +b))
(5)
where, clipping function (x) clamps the values within [-1, 1]. This is because we allow ternary spikes thus ASR must be with [-1,1]. Furthermore, the dynamic saliency score also achieves an equilibrium value F since the ASR at the final layer achieves equilirbrium state a Convergence dynamics of other layers are given in Appendix A To analyze the overall layer-wise neural activity, which includes both positive and negative spiking event, we present the layer- wise dynamics of the absolute spiking activity events in Fig. b. i.c. act;(t] Ei_yle |e.[c]
Training: As described in the Section 4.4] the Spiking decoder is responsible for predicting J: and d, for individual video clip i and &; is computed using the SFG module at equilibrium. Using these three predictions, we design loss function that combines various components. The total loss over Nr clips in the training set is given by
Luyer-wise Convergence Dynamica
Layer-wine Spilknn Activity (ac()
0.40
Mean Activity
0.60
Recuctionis
Mean ASR
komm
0.50
0.30
0.20
0.30
0.10
0.20
0.10
Sealing
0.00
20
30 40
30 Time Steps (b)
20
40
Time Steps
Figure 3: Results from random QVHighlights input passed through Spiking VTG models. (a) Layer- mean ASR conver- gence over operating time steps a sample spiking transformer encoder layer (Fig. M note ASR can be negative due to ternary spikes. (b) Layer-wise mean spiking activity (act, [t]. averaged over neurons) versus time steps. Model with SFG exhibit reduced activity in both input and spiking attention layers, highlighting SFG's role in minimizing neural activity.
1 ENT (Ly. + Ld, + L.,) Ny
where Ls is the binary cross- entropy loss for the indicator variable ft. La combines smooth LI loss with IoU loss [1 for the predicted boundaries, and Le an optional loss incorporating intra- and inter-video contrastive learning 332 Detailed formulation of the loss functions in Appendix
During training, leveraging implicit differentiation [33] at equilibrium, only ASR values at equilibrium
used, aL(a") aL((a) [J=1 la-) 00
(fa(a") where, 0 is the model parameters, ga(a) fa(a)
00

---

is the steady-state equation of ASR, J_1 is the inverse Jacobian of when = a', i... at equilibrium. Thus, unlike BPTT. we do not need store the intermediate computational graph and the model parameters can be updated using single backpropagation step.
4.5.1 Cos-L2 Representation Matching (CLRM)
We train the base Spiking model on VTG tasks such as QVHighlights M. achieving comparable performance to a similarly scaled non-spiking transformer-he based model, UniVTG [ as demonstrated in Table I The later to improve its performance further uses pre-training on large datasets such as Video-CO and Ego4D, Since, extensive pretraining the SpikingVTG is considerably resource intensive we propose an effective knowledge transfer strategy. This mechanism enables SpikingVTG to inherit the generalization capabilities learned by the pre-trained detection transformer models like UniVTG model. Importantly, since the primary benefit of spiking architectures manifests during inference on resource-constraine edge devices, this knowledge transfer a one-time process, which significantly improves performance.
Hidden State Matching Loss: To align the hidden state (output of in- dividual encoder layer) of the non- spiking multi-modal transformer model with the converged ASR (4.5) of the corresponding layer of the SpikingVTG model, we propose hybrid loss function combining
Table 1: Ablation study of the effect of CLRM on Spik-
ing VTG evaluated on the evaluation set of QVHighlights. Method
OVHighlights MR
OViighlight HD
0F1.7
mAPoiavg
mAP
UNIVT 6
59.74
40.90
36.13
38.83
61.81
Spiking VTG w/o CLRM
60.12
39.68
36.23
38.84
62.49
UniVTO with
67.35
52.65
68.77
SpikingVTG TG w/ CLRM
67.58
50.82
44.07
40.81
68.64
squared cosine similarity based directional loss and a L2- norm based loss for minimizing the seale difference. For each layer, the student representation is projected into the feature space of the teacher via a learnable linear transformation Wa E xde, The total loss is:
B
L
Acos (1 cos (0.9..) )) + de,
(1.)))
((1,))12
Crep = BXL ==j==
(6)
where N is total number of encoder layers, B is batch size, L is length of sequence, Aens dι₃
a.2.0) (3.0)
hyperparameters and the cosine similarity is computed as: cos( i.j.k) 1| (. l2 1|te where,
si s(2.k) = ar;
E Rd is the teacher representation at layer i, sequence position k, and batch index j. at, E BxLxd, and Tr, E BxLLxd are the pre-activation outputs from student and teacher, respectively.
((ia) Wa E Rd is the projected student representation and t (i.k) = T(iik)
Attention Score Matching Loss: We align the attention score map (AT" E IR BxLXX) of encoder
layer of the non-spiking model with the mean attention score (A I. Σ R of our corresponding converged SpikingVTG model. The total attention map aligmment loss (Cau) is then computed following Equ. [6] This loss term encourages the student to learn cross- cross-modal attention behavior consistent with the pre-trained teacher.
The cumulative representation matching loss is given as LCLRM = Lrep + Lan- Optimizing this loss enables our SpikingVTG model to learn the generalability enabling better performance as demonstrated in Table
4.6 Normalization-Fre and Quantized Spiking
We perform two key optimizations: removal of non-local operations and 1-bit weight quantization. Although SpikingVTG replaces floating-poin -oint MACs with ternary spikes, it retains softmax and layer normalization, which are inefficient for resource-constrained hardware. We eliminate these by introducing Normalization-Free (NF) SpikingVTG.
In MF--ppiingVTGG softmax in the spiking attention is replaced with a ReLU followed by scal- ing with 1/L, reducing compute overhead. Given d-dimensional queries, keys, and values (gill (1), skg [4], sv, [0] at time the attention weights oi are computed as follows:
azy = ([g]1]] 8k, [0] ... (0...)) [4] 8kz [1])
(7)

---

where, ő is a custom kernel consisting of ReLU operation followed by scaling with L-!. We further remove all layer normalization layers eliminate additional non-local operations.
To reduce memory and computational cost, we introduce 1-bit NF-SpikingVTG by binariz- ing weights. Each weight matrix W E Raxm is zero-centered and quantized as W sgn ( mm E, Wo);
am Ei, Weyl, where W, (-1,+1]. The output of each quantized linear layer is scaled by 3. yielding binary-weight, Cemary-activalle model that supports efficient integer-ACe ACC operations. 1-bit NF-SpikingVIG achieves competitive performance while drastically reducing memory and compute, making it well-suited for edge deployment.
Table Performance comparison of our Spiking VTG model against non-spiking VTG solutions on test sets of QVHighlights-Me MR and Charades STA datasets for moment retrieval task.
Method
SNN
OViighlights MR
Charades-STn
670.3
@F0.7
mAP@0.3
mAP
6F0.3
6/0.5
W1l.
mloU
33.02
54.82
29.40
UMT
5
No
56.21
41.18
53.83
37.01
49.35
26.16
58.86
40.86
35.59
70.81
UniVTG-P1 6
No
65.43
50.06
64.06
45.02
72.63
60.19
38.55
52.17
UVCOM 24
No
63.55
47.47
63.37
42.67
56.69
34.76
SpikcMba
19
64.13
49.42
43.67
71.24
59.65
36.12
51.74
BAM-DETR 335
No
62.71
48.64
64.57
46.33
72.93
59.95
39.38
52.33
CG-DETE
No
65.41
48.18
64.51
42.77
70.40
58.40
36.30
50.10
LLMEPET
3
66.73
49.94
65.76
43.91
70.91
36.49
50.25
Sp
Yes
65.29
48.18
64.31
42.25
71.20
58.73
37.16
50.62
Experimentation
We evaluate SpikingVTG variants on moment retrieval and highlight detection tasks using the QVHighlights, Charades-STA, TACoS and Youtube Highlight datasets. Since, the best of our knowledge, our proposed model is the first spiking detection transformer evaluated VTG tasks, we compare its performance against sota non-spiking detection transformers. Addi- tional training, dataset, evaluation metric, hyper- parameter and experimental details are provided in Appendix C
Table 3: Performance comparison on the test set of TACoS for moment retrieval task. Method
SNN
TACoS
000.7
mloU
M.DERR 1
No
37.97
24.67
11.97
25.49
No
UniVTG
No
51.44
34.97
17.35
33.60
QD-DETR
39
No
52.39
36.77
21.07
35.76
CG-DETR
No
52.23
22.23
UniVIG-PT 6
No
56.11
43.44
24.27
38.63
SpikeMbu 19
22.83
Spiking TG
Yes
54.71
39.27
21.84
36.02
The experiments were run NVIDIA RTX A6000 GPU with 48GB memory.
5.1
Results
SpikingVTG establishes baseline for spiking models VTG tasks. The results shown in Table Our model achieves competitive results compared to the current SOTA non-spiking models.
2
&
Training & Inference Metrics: In the CLRM-based knowledge transfer stage, the memory requirement 20GB when using a batch size of 32 on the QVHighlights dataset. In contrast, replacing our training method with BPTT would require over 100GB of memory for T. > 10, making BPTT computationally infeasible. Training on true labels with similar batch size requires 8GB of
Table 4: Performance comparison of our Spiking model against other non- non-sii VTG solutions on the test set of the QVHighlights and Youtube Highlights for highlight detection task.
Melhod
SNN
OVHighlights HD
Youtube Highlights
mAP
11T
Dog
Gym.
Skaling
Skiing
Purking
Suriing
Avg.
UNI S
No
38.18
59,99
65.9
75.2
71.8
72.3
81.5
82.7
74.9
UniVTG 64
38.20
60.96
71.8
76.5
73.3
73.2
73.9
82.2
75.2
UniVTG+PT 6
No
40.54
66.28
74.3
79.0
84,9
75.1
74.4
83.9
78.6
QD-DETR
72.2
SpikeMba
No
74.4
75.4
74.3
75.5
75.5
CG-DETR
40.30
76.3
76.1
UVCOM 34
No
39.98
65,58
73.8
77.1
76.0
75.1
75.7
82.7
76.4
M.E.PTT
No
38.18
59.99
73.6
73.2
75.3
74.0
72.5
82.5
75.3
Spiking VTG
73.9
8

---

memory. The clock time for 50 epochs of training on QVHighlights is = HOURS. Inference latency on entire test set of QViigllights ranges from 25 see (T, 2) 4 mins (T, 20).
Table 5: Ablation Study Spiking VTG variants on the evaluation set of QVHighlights.
Methou
OVIlighlights MR
@Vihghlight HD
Operations
Local
Activity
Energy
600.5
650.7
mAP
HIT@
UniVTG (sola)
52.65
НP-MAC
23.92m
SpikinyVTG without SEG
64,94
47.21
40.49
67.37
FP-ACC
0.41
15.2ml
SpikingVTG with SFG
67.58
50.82
40.81
68.64
FP-ACC
X
0.34
13.8m]
(NT) Spikkngg VTG w/ SEG
1-hit (NF) (NF), w/ SPG
65.31
47.48
40.35
67.30
INT-ACC
0.19
1.3m]
I-bit -NitS(NNPSSl w/ ReLU
65.91
47.04
40.16
67.07
INT-ACC
0.19
1.3mJ
Ablation Study: As demonstrated in Table 5 the inclusion of the SFG mechanism enhances performance compared to the model without SFG. It results in reduced neuronal activity and overall less energy consumption. Thus, we enable SFG for the Spiking VTG variants we discuss next. Without non-local operations, the (NK)-SpikingVTG model achieves competitive performance even compared other SOTA non-spiking VTG models. Although the 1-bit (NF)-Spiking VTG VTG variant shows a slight reduction in performance, it is highly memory efficient and involves simpler INT-ACC computations, resulting order of magnitude less energy consumption. Along with energy consumption (I, 10) Table 5 also highlights the sparsity in the Spiking VTG variants particularly in the 1-bit version underscoring considerably reduced mean neural activity. For more hardware friendly model, we train a variant of 1-bit (NF)-Spiking VTG by replacing all GELU layers with ReLU layer [40 and observe minimal degradation performance. Extensive
study done in Appendix
P
Analysis of Energy Efficiency: We analyze the test-time energy efficiency of SpikingVTO variants compared non-spiking transformer- based model with same depth and hidden di- mensions. The analysis considers arithmetic operation costs using 45mm CMOS technology with 32-bit precision, where FP-MAC, FP-ACC, and INT-ACC ops. consume pJ6 0.9 pJ, and
200
Energy Efficiency (E,)
175
OVHighlights. HD: HIT@
150
125
65
100
64
0.1 pJ respectively [41] The energy cost for
75
non-spiking transformer encoder layer based on total FLOPSs used in attention and linear layers, is E (3LD²) + (LD2 L D) (2LD*)) 4.6 pJ. Considering = 200, compute E 5.98 mJ. Since, Uni has 4 encoder layers, so total energy 23.92m.J.
HIT@1
-63
1024, we
10
Time Steps
Figure 4: Graph showing tradeoff energy effi- ciency (E,) and HIT@ on QVHighlights for varying time steps.
In contrast, the 1-bit (NF)-Spiking VTG oper- ates over Te time steps. Its per time-step cost
is: Es, = [33 IF Rmn LD2) (IF LDD
TFR, L2DD) + (IFRam Runn LD²) + + (i..
LDD)] 0.1 pJ, where TF R4 denotes the mean activity of component 1. For our 1-bit (NF)- SpikingVTG model, which uses INT-ACC operations, we empirically measure: IFR, = 0.40, IF 0.18, LR, 0.19. FF 0.03, and IF Ruttem. With T. = 10, the total spiking energy is Es = T. Es. 0.3 mJ, yielding an energy efficiency of E, E Es 19.93. Model-wise energy comparison is shown in Table 5
0.09, resulting in Es, 0.03 mJ.
Energy-Accurace Tradeoff: Unlike conventional non-spiking VTG solutions, SpikingVTG enables a controllable trade-off between performance and energy consumption, as total energy usage scales with the number operating time steps. In Fig. 4 we demonstrate this performance-energy tradeoff achieved using 1-bit (NFF)-Spking VTG model by running it for different number of time steps. Notably, when running the model for just time steps, Spiking VTG attains HIT@ score 66.4 (compared to the SOTA score of 68. 77), while achieving energy efficiency factor E, ss 100.

---

6 Conclusions
In this paper we propose SpikingVTG, bio-inspired computationally efficient solution for VTG tasks in resource constraint environment. We propose saliency feedback gating mechanism, which leverages the temporal dynamics of the model improve performance while lowering computational costs by reducing overall neuronal activity across the model. We empirically demonstrate the convergence dynamics of the SpikingVTG model and leverage the formulated steady-state equations to efficiently train our model using implicit differentiation at equilibrium. To improve generalization, we propose Cos-L2 Representation Matching loss, enabling knowledge transfer from non non-spiking VTG models and yielding substantial performance gains. We further optimize the model by removing non-local operations and applying extreme quantization, leading to the 1-bit NF-SpikingVTC. Our framework significantly improves computational efficiency and model compactness, enabling tunable energy-aceuracy tradeoffs on low-resource devices. While this work presents the first spiking solution for VTG with competitive performance, there remains room for improvement in fully closing the performance gap. Future research can build upon our framework to further advance spiking models in this domain.
7 Acksowledgments
This work was supported in part by the United States Air Force and Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8875023-C-00519 and the U.S. Army Research Laboratory Cooperative Research Agreement I9INT-17-2-0016. Any opinions, findings and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the Department Defense or the United States Government.
References
[1] Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:11846-111 11:1184--1188,, 2021.
[2] Songyang Zhang, Houwen Peng. Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[3] Jonghwan Mun, Minsu Cho, and Bohyung Han. Local-global video-text interactions for temporal grounding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattemn Recognition, pages 10810-10819, 2020.
[4] Fa- Ta-Ting Hong, Xuanteng Huang, Wei-Hong Li, and Wei-Shi Zheng. Mini-net: Multiple instance ranking network for video highlight detection. In Computer ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 345-360, Springer, 2020.
[5] Ye Liu, Siyuan Li, Yang Wu, Chang-WWe Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3042-3051,
2022.
[6] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin- peng Wang. Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2794-2804, 2023.
[7] Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1 1-9. IEEE, 2023.
[8] Samanwoy Ghosh-Dastida Dastidar and Hojjat Adeli. Spiking neural networks. International journnaal of neural systems, 19(0)):95-308 2009.
10

---

[9] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical the ventral stream's 's execution of core object recognition behavior. Nature neuroscience, 22(6))974-983, 2019.
[10] Mingqing Xiao, Qingyan Meng, Zongpeng Zhang. Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation on the equilibrium state, Advances in Neural Information Processing Systems, 34: 14:11111-15222 2021.
[11] Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 1-63 2019.
[12] Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. In European Conference on Computer Vision, pages 407-426. Springer, 2022.
[13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
[14] Amar Shrestha, Haowen Fang, Zaidao Mei, Daniel Patrick Rider, Qing Wu, and Qinru Qiu. survey on neuromorphic computing: Models and hardware. IEEE Circuits and Systems Magazine, 653, 2022.
[15] Soroush Abbasi Koohpayegani and Hamed Pirsiavash. Sima: Simple softmax-free attention for vision transformers. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2607-2617, 2024.
[16] Boxun Xu, Hejia Geng, Yuxuan Yin, and Peng Li. Ds2ta: Denoising spiking transformer with attenuated spatiotemporal attention. arXiv preprint arXiv:2409.15377 2024.
[17] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang. Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint ar Xrxiv:2310. 1453, 2023.
[18] Malyaban Bal. Yi Jiang, and Abhronil Sengupta. Exploring extreme quantization in spiking language models. In 2024 International Conference on Neuromorphic Systems (ICONS), pages 272-276. IEEE, 2024.
[19] Wenrui Li, Xiaopeng Hong, and Xiaopeng Fan. Spikemba: Multi-modal spiking saliency mamba for temporal video grounding. arXiv preprint arXiv:2404.0/ /74, 2024.
[20] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprinl arXiv:2312.00775 2023.
[21] Michael V DeBole, Brian Taba, Amon Amir, Filipp Akopyan, Alexander Andreopoulos, William P Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, al. Truenorth: Accelerating from zero to 64 million neurons in 10 years. Computer, 52(5):20-
29, 2019.
[22] Mike Davies, Andreas Wild, Garrick Orchard, Yulia Sandamirskaya, Gabriel A Fonseca Guerra, Prasad Joshi, Philipp Plank, and Sumedh R Risbud. Advancing neuromorphic computing with loihi: A survey of results and outlook. Proceedings of the IEEE, 109(5):91 1-934, 2021.
[23] Guangzhi Tang, Neelesh Kumar, and Konstantinos P. Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6090-6097, 2020.
[24] Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le. Spiking neural networks and their applications: A review. Brain Sciences, 12(7):863, 2022.
[25] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint 22X.:2209.15425, 2022.
11

---

[26] Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10998-1 19998-11006, 2024.
[27] Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint urXiv:2302. 13939, 2023.
[28] Jan Hosang, Rodrigo Benenson, and Bernt Schiele. Learning non-maximum suppression. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4507-4515, 2017.
[29] Sangya Dutta, Vinay Kumar, Aditya Shukla, Nihar R Mohapatra, and Udayan Ganguly. Leaky integrate and fire neuron by charge-discharge dynamics in Roating-body mosfet. Scientific reports, (():8257,2011)
[30] Yufci Guo, Yuanpei Chen, Xiaode Liu, Weihang Peng, Yuhan Zhang, Xuhui Huang, and Zhe Ma. Ternary spike: Learning ternary spikes for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12244-12252, 2024.
[31] Hamid Rezatofighi, Nathan Tsoi, Jun JonYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[32] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[33] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
[34] Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma, Hengwei Bian, Yatai Ji, Yujiu Yang, and Xiu Li. Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection, 2023.
[35] Pilhyeon Lee and Hyeran Byun. Bam-detr: Boundary-aligned moment detection transformer for temporal sentence grounding in videos, 2024.
[36] Hao Sun, Mingyao Zhou, Wenjing Chen, and Wei Xie. Tr-detr: Task-reciprocal transformer for joint moment retrieval and highlight detection, 2024.
[37] WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae-Pil Heo. Correlation-guided query- dependency calibration for video temporal grounding, 2024.
[38] Yiyang Jiang, Wengyu Zhang, Xulu Zhang, Xiao Yong Wei, Chang Wen Chen, and Qing Li. Prior knowledge integration via I1m encoding and pseudo event regulation for video moment retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24 page 7249-7258. ACM, October 2024.
[39] WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query- dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23023-23033, 2023.
[40] Jonathan Timcheck, Sumit Bam Shrestha, Daniel Ben Dayan Rubin, Adam Kupryjanow, Garrick Orchard, Lukasz Pindor, Timothy Shea, and Mike Davies. The intel neuromorphic dns challenge. Neuromorphic Computing and Engineering, 3(3):034005, 2023.
[41] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks, 2015.
[42] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 2277-525..001. 2017.
12

---

[43] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 12,-36,220 2013.
[44] Min Sun, Ali Farhadi, and Steve Seitz. Ranking domain-specific highlights by analyzing edited videos. In Computer Vision ECco 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 787-802. Springer, 2014.
13

---

A Extended Architecture Overview
The Spiking Transformer layer primarily consists of a spiking multi-head attention (MHA) block, followed by a spiking feedforward network comprising an intermediate layer and an output layer with both inter- and intra-layer communication happening using spikes. Details of the operations in each layer provided below.
In Spiking MHA, to enable computationally efficient accumulate based operations the input to the attention layer are spikes instead of real-valued data. The spiking attention mechanism 26 is given as follows,
Spiking Attention Block:
(X, K. f.. V.it)) = o(d * Q(x,(t) (K,14)²) (t)))
(8)
Here, (t)) represents the Query, obtained by passing the input spikes Xg (t) at time t through linear layer (Wo). The spikes for the Key layer (t) are generated by passing X, (t) through a linear mapping (Wx). followed by LIF neuron layer. Similarly, we generate spikes for Value. d is scaling constant. Since the input, key, and value matrices consist of spike trains rather than real-valued data, the primary computations in all matrix multiplications are floating-pon point accumulation operations rather than floating point multiplicative and accumulative operations. In the (NF) NNFi-Sik VTG variant, as discussed in the paper, use e the ReLU and scaling operation, significantly reducing the computational overhead compared to employing & as the non-local ftmar operation. The output of the attention layer fed to an LIF neuron, which outputs spikes. The convergence dynamics of the layer at equilibrium is given as, aattn = of Vin (Attn(a", a a.)) + buttn), where a represents the ASR of the layer used to generate the Query, ag denotes the ASR of the Key, and at corresponds to the ASR of the Value. buttn is bias term.
Intermediate Layer: The intermediate layer takes as input the spikes generated from the preceding layer and maps it through an LIF layer. The convergence dynamics of the layer at equilibrium is given as, interm. o ( (act((( (Wctt(Wnterm..... +bnterm..) )). where Winterm is the linear weight and gelu() is the activation used for the layer. a is the ASR at equilibrium for the previous layer. binterm. is a bias term. In this paper, we have used explored different choices for act, such as GELU and ReLU. During inference, all matrix multiplications involve accumulative operations due to the nature of the input.
intermediate dimension with linear layer. The output is then passed
Output Layer: The output layer takes as input the spikes generated from the preceding layers as shown in Fig. [ The output then passed through an LIF layer. The convergence dynamics of the layer at equilibrium is given as, Coutput al Ven (wormiWWoutpee outpue
az)) + boutput) )), where W outpuu is the linear weight and layer norm is used for normalization. @ntterm. is the ASR at equilibrium for the previous intermediate layer. boutput a bias term. During inference, all matrix multiplications involve accumulative operations due to the nature of the input. In the (NF)- Spiking VTG model we further remove the layer normalization to improve on-chip deployability.
B Loss Function Details
As described in the main paper, the total loss over N clips in the training set is defined as =
N DM (Ly. + Ltd, + Le,), ), where Ly represents the binary cross-entropy loss for the indicator variable f., La combines the smooth LI loss with the generalized IoU loss BI for the predicted boundaries, and Le is an optional loss term incorporating intra- and inter-video contrastive learning
33]. We follow similar loss function construction as previous works on VTG T6. The loss for fore-ground parameter is given as follows,
L = -Ay log + +11 (1- fi) log(1 f.)
where, fr is the true label and fo is the model prediction. The loss for predicted boundaries given as follows,
La = 11 (Au (ALI Smmoottl (dl, de) + Aro (bs)b1))
(10)
14

---

where, d, b, are the true label and d, by is the model prediction. Lc Aineer Ammra Lintra is used for inter-video and intra video contranstive learning [6] For each video we randomly select a clip with fore-ground indicator= I and positive saliency score. Clips from the same video, denoted as Uj, with saliency scores $j < 8; are treated as negative samples. i.e., = { 8i, Sjj h and perform intra-video contrastive learning using the loss
cxp($;/T)
Luur - log
(11)
exp(3,/T) + Ejea exp(3,/T)
Furthermore, treat textual queries from other samples within the batch (I: S) as negative samples, enabling inter-video contrastive learning for cross-sample supervision:
exp($:/T)
Linter = loog Ekes exp /T)
(12)
where S is the training batch, a cos(us, M&) and Mj is the sentence representation (Fqn. cos is cosine similarity.
and
C Dataset Details
QVHighlights: The QVHighlights dataset [ stands out the sole dataset providing annotations for both moment retrieval and highlight detection, making it an excellent resource for benchmarking on both the VTG tasks. Comprising 10,148 videos with average duration of 150 seconds. It features a total of 10,310 queries linked to 18,367 moments, resulting in an average of 1.8 distinct moments per query within each video. The dataset spans variety of scenarios, including daily vlogs, travel vlogs, and news events.
Charades-STA: The Charades-STA 42 dataset comprises 16,128 indoor videos, each with an average duration of 30.6 seconds. includes 12,408 query-interval pairs designated training and 3,720 query-interval pairs reserved for testing.
TACoS: TACoS 443 consists of 127 videos, each averaging 4.78 minutes length. The dataset is split into 75 videos for training, 27 for validation, and 25 for testing.
Youtube Highlights: YouTube Highlights 44 consists of 433 videos across 6 domains, using the domain names as text queries.
D Evaluation Metrics:
For QVHighlights, following previous work 11 use Recall with IoU thresholds 0.3, 0.5 and 0.7 and avg. mean average precision (mAP), mAP@0.5 and mAP@0.7 0..7 as the evaluation metric for moment retrieval tasks. For highlight detection, we use mAP and HIT@ [I] where clip is considered a true positive if it receives score of "Very Good" [5]. For Charades-STA and TACoS, we employ Recall @ I with IoU thresholds of 0.3, 0.5, and 0.7, along with the mean IoU (mloU). For Youtube Highlights we use mAP.
E Additional Experimental Details
In this subsection, we provide a concise overview the implementation details and provide additional experimental details. The GPU specifications for the experiments detailed in the main paper, while the CPU utilized an AMD Ryzen Threadripper 3960X 24-Core Processor. We have used Python and the PyTorch framework to write the code. The video and textual feature are developed following previous work [ 6). We have used the Adam optimizer to train our model. We list the hyper-parameters used in the work in Table 6. We used grid search to find optimal values.
E.1 Training Stages
Training a multi-modal spiking architecture like Spiking VTG resource-intensive. enhance the efficiency of this process and develop computationally efficient variants of our model, we leverage
15

---

Hyper-parament parameters
Range
Optimal
Encoder Layers
(2-6)
D: Hidden Dimension
(768-2048)
1024
Tu fedecoerr depth
(1-5)
k1 f f-decoerr kernel size
(3-9)
n2: d-decoder depth
(1-5)
k2 d-decoder kernel size
(3-9)
TCLRM: Timesteps for CLRM
(5-100)
Ty: Timesteps for Finetuning
(5-50)
16
Vea: Threshold Potential
(0.5-2.0)
1.0
7: Leakk--fctorr
-0.9-100)
0.99
Xy :L., co-eflicient
(1-20)
10
AL1 :LLsmoothL1 co-efficient
(1-20) 20)
Amttru :Lintra Co-efficient
(0 1.0)
0.05
Ainter
(0 1.0)
0.01
Niou Mioo co-efficient
(1 20)
10
Acos: Sq. cosine weight (CLRM)
00 1.01
0.2
Na. L2 weight (CLRM)
(0-1.0)
0.8
Ir: Learning Rate
(Ie 8 1e 9)
wa: weight decay
(le 1e-9)
le
Batch Size
(8-64)
32
Epochs: CLRM
10-100
Epochs: Finetuning
20-200
100
Table 6:
of our SpikingVTG model. Optimal values for QVHighlights dataset is
also shown.
multi-staged training framework. We utilize a transformer- non-spiking VTG model (such as UniVTG) perform CLRM loss optimization. After this initial stage, fine-tune Spiking VTG using the true labels. Once the base SpikingV'TC model is established, we modify its architecture to remove non-local operations and perform extreme quantization, followed by additional fine fine-tuning to create computationally efficient variants with minimal performance degradation. The resulting computationally efficient, lightweight models well-suited for deployment neuromorphic chips, enabling efficient inference.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims made in the abstract and Introduction are upheld in the method-
ological discussions and empirical results provided in Section and 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
We discuss the limitation of the proposed model in the Conclusions Section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our work is primarily empirical; we demonstrate the model's convergence dynamies and validate the performance gains of our proposed techniques through compre- hensive experiments on relevant benchmarks.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section and extensively goes over the primary contributions. We also provide additional experimental details (hyper-parameterr, dataset details, etc.) in the Appendix.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We are uploading the code as part of the submission. If accepted we will make it open access github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiips.cc/puulic//uui
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
19

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have added detailed discussion on datasets, hyper- parameters,, experimental setup in Section [5 and Appendix E
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have added the experimental results following similar literature in Section
S
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have thoroughly discussed computational resource requirement in Section
5
20

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't Midn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:[Yes)
Justification:
The research conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This work aims to develop sustainable Al solutions that reduce the carbon footprint of machine learning models, thereby contributing to environmentally responsible and socially impactful AI.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
21

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: Yes]
Justification: Proper citations to all resources provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
22

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy attee:///aarrpppcccc////feeerrse for what should or should not be described.
23


Reviews:
[{"review_id": "6jnr5MtJJA", "reviewer": "Reviewer_vi5H", "summary": "This paper proposes SpikingVTG, a novel spiking neural network-based detection transformer designed for Video Temporal Grounding (VTG). The model features a bio-inspired, energy-efficient architecture comprising a spiking transformer core, a Saliency Feedback Gating (SFG) mechanism for relevance-aware filtering, and a spiking decoder for output generation. To enhance generalization, the authors introduce a Cos-L2 Representation Matching (CLRM) loss for knowledge transfer from pre-trained non-spiking models. Furthermore, they develop normalization-free and 1-bit quantized variants for deployment on resource-constrained edge devices. Experimental results across four VTG benchmarks (QVHighlights, Charades-STA, TACoS, and YouTube Highlights) demonstrate competitive performance and notable energy savings, establishing a solid foundation for future research on spiking-based VTG systems.", "questions": "The traditional neural networks like CNN, transformer can be accelerated by hardware like GPUs with CUDA. Is their any matured and popular hardware and hardware acceleration libs to support Spiking Neural Networks (SNNs)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper introduces the first spiking transformer architecture for VTG, replacing conventional dense matrix multiplications with event-driven, accumulation-based operations using leaky integrate-and-fire neurons. This design aligns well with neuromorphic computing principles and targets the high energy cost of traditional VTG models.\nThe proposed Saliency Feedback Gating (SFG) mechanism enables dynamic focus on query-relevant video segments using spike-based saliency estimation. Combined with the use of implicit differentiation for BPTT-free training, the model achieves both efficiency and scalability.\nThe experimental evaluation is comprehensive, spanning four VTG datasets with multiple model variants (including normalization-free and 1-bit versions). The authors conduct thorough ablation studies and analyze the energy-accuracy tradeoff, enhancing the paper's empirical depth and practical relevance.\n\n**Weaknesses**\n\nDespite its architectural efficiency, the proposed model underperforms compared to state-of-the-art pretrained non-spiking VTG models (e.g., UniVTG), especially at higher IoU thresholds. This may limit its adoption in applications requiring high localization precision.\nThe SFG module introduces additional floating-point operations, but its computational overhead is not fully quantified or compared against the energy gains, making its efficiency claims less conclusive.\nWhile the authors assert that the model is \"well-suited for edge-based deployment,\" no experiments are conducted on actual edge or neuromorphic hardware. This lack of validation undermines the strength of the deployment claim.\nAll energy efficiency results are estimated via theoretical cost models. The absence of real-device power profiling or runtime measurements limits the practical credibility of the claimed energy savings.", "was_revised": true, "final_justification": "The authors have addressed many of my concerns, I'd happy to lift the score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed feedback. We address each concern below and clarify our contributions with additional evidence and improvements for the final version.\n\n**Question 1**\n> The traditional neural networks like CNN, transformer can be accelerated by hardware like GPUs with CUDA. Is their any matured and popular hardware and hardware acceleration libs to support Spiking Neural Networks (SNNs)?\n\n**Response**\n\nThank you for this insightful question. Yes, mature hardware platforms and toolchains for SNNs are emerging. **Intel's Loihi 2** supports asynchronous event-driven computation and spiking neuron models, making it well-suited for models like our 1-bit NF-SpikingVTG. Other neuromorphic platforms include IBM's TrueNorth, SpiNNaker, and Akida’s BrainChip. Software Librarires such as NxSDK, SpikingJelly, and LAVA enable SNN training and deployment on these platforms.\n\nImportantly, matmul-free transformer architectures similar to ours have been deployed on Loihi 2 [1], demonstrating improved energy efficiency than deploying transformers on edge-based GPUs, confirming the practicality and efficiency of such models for edge deployment. In response to Weakness 3, we have also discussed methodology to deploy our proposed SpikingVTG model on the Intel Loihi 2 using the Intel Lava framework.\n\n**Weakness 1**\n> Despite its architectural efficiency, the proposed model underperforms compared to state-of-the-art pretrained non-spiking VTG models (e.g., UniVTG), especially at higher IoU thresholds. This may limit its adoption in applications requiring high localization precision. \n\n**Response**\n> We thank the reviewer for this valuable feedback. While our focus is not on directly surpassing full-precision non-spiking models like UniVTG, SpikingVTG aims to offer a first-of-its-kind neuromorphic, computationally-efficient alternative for video temporal grounding in resource-constrained settings. It achieves competitive performance on both QVHighlights (e.g., HD @0.7: 50.82 vs. 52.65) and Charades-STA (e.g., @0.7: 37.16 vs. 38.55), while the 1-bit (NF)-SpikingVTG variant sustains strong performance (47.48 and 36.10, respectively) at over 18× lower energy cost (Table 5).\nA key strength of our model is its tunable energy-accuracy tradeoff (Fig. 4), enabled by varying the number of inference time steps. For example, with just 2 time steps, SpikingVTG achieves HIT@1 = 66.4 on QVHighlights with ~100× greater energy efficiency. This level of test-time adaptability is not possible with conventional VTG architectures, making SpikingVTG particularly well-suited for edge and low-power deployments.\n\n**Weakness 2**\n> The SFG module introduces additional floating-point operations, but its computational overhead is not fully quantified or compared against the energy gains, making its efficiency claims less conclusive.\n\n**Response**\n\n**Computational Cost of SFG:**  \n The SFG module computes per-clip saliency scores via cosine similarity between the output layer ASR $\\( a^{N_v}_i[t] \\in \\mathbb{R}^D \\)$ and the global textual embedding $\\( M \\in \\mathbb{R}^D \\)$.  This results in a total computational complexity of  $\\[\\mathcal{O}(L_v \\cdot D)\\]$  which is *linear* in the number of video segments $\\( L_v \\)$. In contrast, the spiking attention mechanism in the transformer core has quadratic complexity  $\\[\\mathcal{O}(L^2 \\cdot D + L \\cdot D^2)\\]$  where $\\( L = L_v + L_q \\)$. Thus, *even for long video sequences*, the computational cost of SFG is negligible compared to the spiking transformer core.  Moreover, the operations involved in SFG consist of dot products—element-wise in nature—and are computationally lightweight.\n\n**Energy Considerations of SFG**  While cosine similarity involves floating-point arithmetic (element-wise multiplication and summation), it does **not** require floating point matrix multiplications.  Therefore, its energy footprint is **significantly lower** than that of transformer attention layers.  For example, for the 1-bit NF SpikingVTG, considering cost of arithmetic operations on 45nm CMOS, while the transformer core consumes 1.3mJ the SFG mechanism only consumes ~0.01mJ only (Considering a video of 400 secs).\n\nMore importantly, the inclusion of SFG leads to **reduced spike activity** across layers (see Figure 3b), translating into **lower computational cost and improved accuracy**. This is highlighted below with an ablation with and without SFG:\n\n| Variant                | @0.5   | @0.7   | mAP   | HIT@1 | Energy (mJ) | Neural Activity |\n|------------------------|--------|--------|-------|--------|--------------|------------------|\n| SpikingVTG w/o SFG     | 64.94  | 47.21  | 40.49 | 67.37  | **15.2**     | 0.41            |\n| **SpikingVTG w/ SFG**  | **67.58** | **50.82** | **40.81** | **68.64** | **13.8**     | **0.34**        |\n\n**Weakness 3**\n> While the authors assert that the model is \"well-suited for edge-based deployment,\" no experiments are conducted on actual edge or neuromorphic hardware. This lack of validation undermines the strength of the deployment claim. All energy efficiency results are estimated via theoretical cost models. The absence of real-device power profiling or runtime measurements limits the practical credibility of the claimed energy savings.\n\n**Response**\n\nIn order to validate the feasibility of on-chip deployment, we have implemented our 1-bit (NF)-SpikingVTG to be compatible with Intel Loihi 2 using the Intel LAVA framework. The model is architecturally aligned with Loihi 2’s hardware capabilities, being fully matmul-free (replacing floating-point MACs with integer accumulation operations), removing all non-local normalization layers such as softmax and layer norm. We implement our ternary spiking neurons using **lava.proc.lif.models.PyTernLifModelFixed** and the 1-bit weights are implemented as a custom **ProcessModel** similar to the BitLinear implementation done by [1].\n\nLoihi 2 supports two execution modes: **pipelined** mode for full-input processing and **fall-through** mode for autoregressive token generation. For VTG tasks, we use pipelined mode for offline batch settings (e.g., full video moment retrieval) and fall-through mode for real-time highlight detection (clip-by-clip processing). While [1] implemented a 12-layer matmul-free transformer (370M parameters), our proposed model has only 4 layers with approximately $10\\times$ fewer parameters, further reducing memory and compute requirements.\n\nIn fall-through mode that is with realtime processing, reported Loihi 2 benchmarks for a matmul-free transformer akin 1-bit NF SpikingVTG, demonstrate a throughput of 41.5 tokens/sec compared to 14.3 tokens/sec for an iso-param conventional transformer on an NVIDIA Jetson GPU (around $3 \\times$ speedup). The energy cost per token when we process the entire sequence is 3.7 mJ/token on Loihi 2 versus 11.7 mJ/token on the Jetson GPU, i.e. $>3 \\times$ less. \n\nApart from leveraging computationally efficient accumulation based operations, these improvements stem from Loihi 2’s on-core storage of weights and recurrent states in local SRAM, which minimizes memory movement and reduces both latency and energy consumption.\n\nReferences:\n\n[1] Abreu, Steven, Sumit Bam Shrestha, Rui-Jie Zhu, and Jason Eshraghian. \"Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2.\" arXiv preprint arXiv:2503.18002 (2025).\n\n**We thank the reviewer for their constructive feedback, which has helped us strengthen the work. In light of our clarifications, we respectfully request the reviewer to reconsider their rating. We are happy to address any additional questions or comments.**"}, {"review_id": "4Od6bfWs73", "reviewer": "Reviewer_Y4Qk", "summary": "The paper proposes SpikingVTG, a spiking neural network based architecture for video temporal grounding tasks. Unlike conventional transformers, it uses spike-based, event-driven computation for higher energy efficiency. Key innovations include a Saliency Feedback Gating mechanism to focus on relevant video clips, a knowledge transfer method (CLRM) from non-spiking models, and quantized, normalization-free variants for edge deployment. The results on the benchmarks shows good performance at low cost.", "questions": "1. seems the model accuracy depends on Cos-L2 Representation Matching with a heavyweight teacher; without CLRM the performance drops about 3%on qvh. How important is the teacher model, what if we use a smaller teacher, is the the SNN architecture itself can learn strong multimodal representations or it has to rely on the knowledge distillation?\n2. The SFG mechanism produces saliency scores based on average spiking rates, but there is no visualize or explain why certain clips are gated.  Please further explain if certain behaviors are explainable or the behavior of SFG are purely empirical.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper is intuitive and driven by biologically motivated. The Saliency Feedback Gating (SFG) re-uses the network’s own spiking activity as a guiding signal, boosting meaningful spikes in clips. \n2. The 1-bit version fo the model is impressive with only 11MB  in size and reduces one-pass energy to 1.3 mJ, this has true potential to be deployed to the edge devices.\n\nWeaknesses\n1. the proposed method, to match the performance, relies on a heavy teacher models. This multi-stage pipeline (teacher > CLRM > fine-tune) adds additional computes and makes it not clear if the proposed architecture is effective or the distillation part.\n2. The performance of proposed model Still lags behind the SOTA. I would encourage the authors to provide more in-depth analysis of the errors, and the limitation of current models. \n3. Seems the efficiency claims come from FLOP-based calculations with cost model; this is not very convincing, the author could simply use GPU energy consumption to estimate the efficiency.", "was_revised": true, "final_justification": "i carefully read the rebuttal and other reviews and decided to keep my rating. \nPlease make sure to include the rebuttal numbers and contents to the final paper.", "rebuttal": "Thank you for your detailed and valuable feedback on our paper. In this rebuttal, we address your comments.\n\n**Question 1**\n> seems the model accuracy depends on Cos-L2 Representation Matching with a heavyweight teacher; without CLRM the performance drops about 3%on qvh. How important is the teacher model, what if we use a smaller teacher, is the the SNN architecture itself can learn strong multimodal representations or it has to rely on the knowledge distillation?\n\n**Response**\n\nThank you for this insightful question. We would first like to emphasize that even without incorporating Cos-L2 Representation Matching, our model achieves competitive performance (as highlighted below & Table 1 in paper) compared to the non-pretrained transformer-based VTG baseline, UniVTG.\n\n**Table 1:** Ablation study of the effect of CLRM on SpikingVTG evaluated on the evaluation set of QVHighlights.\n\n| Method                  | QVHighlights-MR @0.5 | QVHighlights-MR @0.7 | mAP@avg | mAP   | HIT@1 |\n|-------------------------|----------------------|----------------------|---------|-------|-------|\n| UniVTG              | 59.74                | 40.90                | 36.13   | 38.83 | 61.81 |\n| SpikingVTG w/o CLRM     | 60.12                | 39.68                | 36.23   | 38.84 | 62.49 |\n\nTo achieve improved performance, transformer-based VTG models such as UniVTG typically rely on extensive pre-training on large-scale datasets (e.g., Ego4D, VideoCC). However, pre-training a spiking model from scratch is substantially more computationally expensive. To address this, we employ Cos-L2 Representation Matching (CLRM), which enables direct knowledge transfer from a pre-trained UniVTG to the downstream task—eliminating the need for costly large-scale pre-training. To demonstrate that SpikingVTG is fully capable of learning strong multimodal representations on its own, we also pre-trained it on Ego4D and VideoCC. Achieving similar performance (on QVHighlights), however, required approximately 45 hours of wall-clock time for full pre-training, compared to only ~4 hours for knowledge distillation via CLRM (50 epochs, each epoch ~300 sec). This highlights CLRM as an efficient and compute-friendly solution for rapidly training SpikingVTG with significantly reduced resource requirements.\n\n**Question 2**\n> The SFG mechanism produces saliency scores based on average spiking rates, but there is no visualize or explain why certain clips are gated. Please further explain if certain behaviors are explainable or the behavior of SFG are purely empirical.\n\n**Response**\nThank you for this excellent question. The SFG mechanism computes saliency scores by measuring the cosine similarity between the final-layer ASR representation of each video clip and the textual query embedding (Eq. 2). These similarity scores are then min–max normalized to lie within $[0, 1]$, where **higher values indicate greater relevance to the query**.  \n\nFigure  2b provides a visualization of SFG’s behavior. In this example, the ground-truth highlight corresponds to clip index 28. We observe that clips temporally close to index 28 exhibit higher saliency scores (warmer colors, more red), while clips further away from the relevant segment have lower scores (cooler colors, more blue). This pattern is consistent across examples and indicates that the SFG mechanism tends to assign higher weights to clips that are semantically and temporally aligned with the query.  \n\nThus, the per-clip scores $\\(F_s\\)$ are interpretable as a relevance measure between the query and video content. The visualization confirms that SFG is able to focus attention on query-relevant regions in a manner consistent with the ground truth.\n\n**Weakness 1**\n>the proposed method, to match the performance, relies on a heavy teacher models. This multi-stage pipeline (teacher > CLRM > fine-tune) adds additional computes and makes it not clear if the proposed architecture is effective or the distillation part.\n\n**Response**\n\n We have responded to this in the response to **Question 1**.\n\n**Weakness 2**\n>The performance of proposed model Still lags behind the SOTA. I would encourage the authors to provide more in-depth analysis of the errors, and the limitation of current models.\n\n**Response**\n\nThank you for highlighting this point. While SpikingVTG is, to the best of our knowledge, the first fully spiking solution for VTG tasks, we acknowledge that there remains room for improvement toward matching the best-performing full-precision models. One key factor contributing to the performance gap is the quantization of activations to ternary spike states `{−1, 0, 1}`, which inevitably introduces information loss and reduces representational precision.  Future work can address this by exploring graded or multi-level spiking activations (supported in neuromorphic hardware such as Loihi 2), which can improve representational capacity while retaining the energy efficiency of spike-based computation.  We will include a detailed discussion of these limitations and potential improvements in the revised manuscript.\n\n**Weakness 3**\n> Seems the efficiency claims come from FLOP-based calculations with cost model; this is not very convincing, the author could simply use GPU energy consumption to estimate the efficiency.\n\n**Response**\n\nIn our paper we provided a FLOP based estimate following energy consumption of MUL, ACC, operations on 45nm CMOS. In order to validate the feasibility of on-chip deployment, we have implemented our 1-bit (NF)-SpikingVTG to be compatible with Intel Loihi 2 using the Intel LAVA framework. The model is architecturally aligned with Loihi 2’s hardware capabilities, being fully matmul-free (replacing floating-point MACs with integer accumulation operations), removing all non-local normalization layers such as softmax and layer norm. We implement our ternary spiking neurons using **lava.proc.lif.models.PyTernLifModelFixed** and the 1-bit weights are implemented as a custom **ProcessModel** similar to the BitLinear implementation done by [1].\n\nLoihi 2 supports two execution modes: **pipelined** mode for full-input processing and **fall-through** mode for autoregressive token generation. For VTG tasks, we use pipelined mode for offline batch settings (e.g., full video moment retrieval) and fall-through mode for real-time highlight detection (clip-by-clip processing). While [1] implemented a 12-layer matmul-free transformer (370M parameters), our proposed model has only 4 layers with approximately $10\\times$ fewer parameters, further reducing memory and compute requirements.\n\nIn fall-through mode that is with realtime processing, reported Loihi 2 benchmarks for a matmul-free transformer akin 1-bit NF SpikingVTG, demonstrate a throughput of 41.5 tokens/sec compared to 14.3 tokens/sec for an iso-param conventional transformer on an NVIDIA Jetson GPU (around $3 \\times$ speedup). The energy cost per token when we process the entire sequence is 3.7 mJ/token on Loihi 2 versus 11.7 mJ/token on the Jetson GPU, i.e. $>3 \\times$ less. \n\nApart from leveraging computationally efficient accumulation based operations, these improvements stem from Loihi 2’s on-core storage of weights and recurrent states in local SRAM, which minimizes memory movement and reduces both latency and energy consumption.\n\nReferences:\n\n[1] Abreu, Steven, Sumit Bam Shrestha, Rui-Jie Zhu, and Jason Eshraghian. \"Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2.\" arXiv preprint arXiv:2503.18002 (2025).\n\n**Thank you for your help in improving our paper. In light of our clarifications, we respectfully request the reviewer to reconsider their rating. We are happy to address any additional questions or comments.**"}, {"review_id": "esX3whYUyN", "reviewer": "Reviewer_UKZa", "summary": "The paper introduces SpikingVTG, a novel Spiking Neural Network based model designed for Video Temporal Grounding tasks, which aims to identify precise temporal segments in a video corresponding to natural language queries.\nThe SpikingVTG architecture consists of three key components: a spiking Transformer core, a Saliency Feedback Gating mechanism, and a spiking decoder for output generation. The spiking Transformer core utilizes Leaky Integrate-and-Fire neurons as its fundamental computational units and replaces the traditional floating-point multiply-accumulate operations in standard Transformers with more efficient floating-point accumulate-only operations, enabling it to capture temporal and cross-modal dependencies effectively.", "questions": "Please refer to Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n* The paper introduces a novel application of Spiking Neural Networks (SNNs) to the domain of Video Temporal Grounding (VTG), which has been traditionally dominated by conventional neural networks. This innovation leverages the computational efficiency and sparsity of SNNs, offering a bio-plausible and energy-efficient alternative.\n* By exploiting the inherent sparsity and accumulation-based computation of SNNs, the model achieves substantial reductions in computational overhead and energy consumption. The introduction of normalization-free and 1-bit quantized variants further underscores the model’s suitability for deployment on resource-constrained edge devices, directly addressing a critical practical challenge in VTG.\n* The authors provide extensive experimental results, including ablation studies and comparisons with state-of-the-art non-spiking models. This comprehensive evaluation supports the claims made in the paper and provides insights into the model's behavior.\n\nWeaknesses\n* The paper does not provide in-depth discussion or ablation on how the spiking framework handles the fusion of video and language modalities, which is a non-trivial challenge in VTG. The reader is left to assume that the spiking transformer core adequately captures cross-modal dependencies, but more detailed analysis or visualization would strengthen the claim.\n* Although the SFG mechanism and other architectural innovations are well-motivated, the paper lacks thorough ablation studies to quantify the individual contributions of each component (e.g., normalization-free design, Cos-L2 transfer). This makes it difficult to assess which parts of the design are most critical to performance.\n* The experimental section primarily compares SpikingVTG to non-spiking transformer models. It does not benchmark against other lightweight, quantized, or pruned VTG architectures, which would be necessary to convincingly demonstrate the unique advantages of SNNs in this context.\n* The model is positioned as suitable for edge deployment, but the paper does not report real-time inference latency or throughput on actual hardware. Without these metrics, the practical viability for time-sensitive applications remains somewhat speculative.\n* While the paper claims significant energy efficiency, the methodology for measuring energy consumption and the hardware context are not described in sufficient detail. This lack of transparency may limit reproducibility and make it difficult to directly compare with other efficient VTG models.", "was_revised": true, "final_justification": "The issues about fusion method of video and language, individual contributions of each component, comparison with other lightweight VTG architectures and the deployment issues were mostly solved. Although there are some minor issues, the overall impact of these issues is relatively small and does not significantly affect the overall justification of this paper. Therefore, I recommend a borderline accept.", "rebuttal": "We thank the reviewer for their insightful feedback. Below, we provide our responses to each of the comments.\n\n**Weakness 1**\n> The paper does not provide in-depth discussion or ablation on how the spiking framework handles the fusion of video and language modalities, which is a non-trivial challenge in VTG. The reader is left to assume that the spiking transformer core adequately captures cross-modal dependencies, but more detailed analysis or visualization would strengthen the claim.\n\n**Response**\n\nWe thank the reviewer for raising this point. In SpikingVTG, cross-modal fusion is achieved through joint spiking attention over concatenated video and query tokens. Given the spike-based input \n$X[t] = [V_s[t]; Q_s[t]] \\in \\mathbb{R}^{L \\times D}$, attention weights are computed following Eqn. 8 (in paper) as:\n\n$\\alpha_{ij}[t] = \\varphi\\left(s * Q(X_i[t]) \\cdot (K_j[t])^\\top\\right),$\n\nenabling interaction between video and query tokens $(\\(i \\in V, j \\in Q\\))$. This joint attention mechanism allows the model to learn cross-modal dependencies efficiently. Furthermore, the Saliency Feedback Gating (SFG) mechanism applies to the input video tokens. The gated video input is:\n\n$\\bar{V}[t+1] = V[t]  * \\bar{F}_s^v[t], \\quad \\text{with } \\bar{F}_s^v[t] \\in [0, 1]^{L_v},$ where $F^v_s[i,t] = \\cos(a^{N_v}_i[t], M) = \\frac{a^{N_v}_i[t] \\cdot M}{\\|a^{N_v}_i[t]\\|_2 \\|M\\|_2}.$\n\nTo ensure compatibility of these two modalities in the cosine similarity computation, we incorporate the following design choices:\nBoth $\\( a^{N_v}_i[t] \\)$ and $\\( M \\)$ reside in a latent space of dimensionality $\\( D \\)$. The query representation $\\( Q \\in \\mathbb{R}^{L_q \\times D} \\)$ is projected to a global sentence embedding ($\\mathbb{R}^{D}$) via attention-based pooling:$\\[M = Q^\\top \\text{Softmax}(Q W_p),\\]$ where $\\( W_p \\in \\mathbb{R}^{D \\times 1} \\)$ is a *trainable* projection matrix. This learnable mapping helps align the textual representation with the spatiotemporal semantics of the spiking video features, thereby enabling cross-modal compatibility. In the paper, Figure 2b visualizes the learned saliency scores, showing higher values near target clip. \n\n**Weakness 2**\n> Although the SFG mechanism and other architectural innovations are well-motivated, the paper lacks thorough ablation studies to quantify the individual contributions of each component (e.g., normalization-free design, Cos-L2 transfer). This makes it difficult to assess which parts of the design are most critical to performance.\n\n**Response**\n\nWe thank the reviewer for this helpful suggestion. Below, we present additional ablations that quantify the contributions of (i) Cos-L2 Representation Matching (CLRM), (ii) Normalization-Free (NF) design, and (iii) 1-bit quantization on QVHighlights.\n\n(1) Effect of Cos-L2 Representation Matching (CLRM)\n\n| Method                 | @0.5      | @0.7      | mAP       | HIT\\@1    |\n| ---------------------- | --------- | --------- | --------- | --------- |\n| SpikingVTG w/o CLRM    | 60.12     | 39.68     | 36.23     | 62.49     |\n| **SpikingVTG w/ CLRM** | **67.58** | **50.82** | **40.81** | **68.64** |\n\nCLRM significantly improves performance by aligning the spiking model’s internal states and attention maps with a pretrained non-spiking teacher, enabling strong generalization without costly pretraining.\n\n(2) Effect of NF and 1-bit Quantization on Accuracy and Efficiency\n\n| Method                       | @0.5      | @0.7      | mAP       | HIT\\@1    | Energy (mJ) | Neural Activity |\n| ---------------------------- | --------- | --------- | --------- | --------- | ----------- | --------------- |\n| SpikingVTG w/ SFG       | 67.58 | 50.82 | 40.81     | 68.64 | 13.8        | 0.34           |\n| (NF)-SpikingVTG w/ SFG       | 66.59     | 48.31     | 40.61 | 67.73     | 10.1        | 0.25           |\n| 1-bit (NF)-SpikingVTG w/ SFG | 65.31     | 47.48     | 40.35     | 67.30     | 1.3     | 0.19       |\n\nThus, removing non-local operations and enabling extreme quantization results in reduced computational overhead with minimal reduction in model performance.\n\n**Weakness 3**\n> The experimental section primarily compares SpikingVTG to non-spiking transformer models. It does not benchmark against other lightweight, quantized, or pruned VTG architectures, which would be necessary to convincingly demonstrate the unique advantages of SNNs in this context.\n\n**Response**\n\nWe thank the reviewer for this valuable suggestion. Our primary objective is to present SpikingVTG as the first spiking neural architecture for Video Temporal Grounding (VTG). To the best of our knowledge, this is the first work to investigate VTG from a neuromorphic, spike-driven computation perspective. In the absence of prior spiking VTG frameworks, we benchmark SpikingVTG against strong non-spiking, transformer-based VTG models, which currently represent the state of the art on standard datasets such as QVHighlights, Charades-STA, and TACoS. In addition, we have considered and positioned our method in relation to recent efficient VTG approaches, as discussed below:\n\n**SpikeMba (Li et al., 2024)**: This hybrid model integrates spiking neurons into a Mamba-based architecture. However, its backbone is based on the Mamba architecture and remains fundamentally non-spiking and relies heavily on dense floating-point matrix multiplications. In contrast, 1-Bit NF SpikingVTG is a fully spiking architecture—including transformer, decoder, and feedback pathways—and avoids matrix multiplications altogether during inference by relying on integer accumulation (INT-ACC) operations, making it better suited for deployment on neuromorphic hardware (as discussed in the next segment).\n\n**$R^{2}$-Tuning (ECCV 2024)**: $R^{2}$-Tuning improves efficiency by exploring image-to-video transfer learning via learning  lightweight modules with reduced parameter count. While efficient in terms of model size, it still relies on full-precision dense computation during inference. SpikingVTG is fundamentally different in that it introduces a dynamic, event-driven computation model inspired by biological neurons and achieves significant energy savings without compromising competitive performance.\n\nWe will discuss this in more details in the final version of the paper.\n\n\n**Weakness 4,5**\n> The model is positioned as suitable for edge deployment, but the paper does not report real-time inference latency or throughput on actual hardware. Without these metrics, the practical viability for time-sensitive applications remains somewhat speculative. While the paper claims significant energy efficiency, the methodology for measuring energy consumption and the hardware context are not described in sufficient detail. This lack of transparency may limit reproducibility and make it difficult to directly compare with other efficient VTG models.\n\n**Response**\n\nWe thank the reviewer for pointing this out. In order to validate the feasibility of on-chip deployment, we have implemented our 1-bit (NF)-SpikingVTG to be compatible with Intel Loihi 2 using the Intel LAVA framework. The model is architecturally aligned with Loihi 2’s hardware capabilities, being fully matmul-free (replacing floating-point MACs with integer accumulation operations), removing all non-local normalization layers such as softmax and layer norm. We implement our ternary spiking neurons using **lava.proc.lif.models.PyTernLifModelFixed** and the 1-bit weights are implemented as a custom **ProcessModel** similar to the BitLinear implementation done by [1].\n\nLoihi 2 supports two execution modes: **pipelined** mode for full-input processing and **fall-through** mode for autoregressive token generation. For VTG tasks, we use pipelined mode for offline batch settings (e.g., full video moment retrieval) and fall-through mode for real-time highlight detection (clip-by-clip processing). While [1] implemented a 12-layer matmul-free transformer (370M parameters), our proposed model has only 4 layers with approximately $10\\times$ fewer parameters, further reducing memory and compute requirements.\n\nIn fall-through mode that is with realtime processing, reported Loihi 2 benchmarks for a matmul-free transformer akin 1-bit NF SpikingVTG, demonstrate a throughput of 41.5 tokens/sec compared to 14.3 tokens/sec for an iso-param conventional transformer on an NVIDIA Jetson GPU (around $3 \\times$ speedup). The energy cost per token when we process the entire sequence is 3.7 mJ/token on Loihi 2 versus 11.7 mJ/token on the Jetson GPU, i.e. $>3 \\times$ less. \n\nApart from leveraging computationally efficient accumulation based operations, these improvements stem from Loihi 2’s on-core storage of weights and recurrent states in local SRAM, which minimizes memory movement and reduces both latency and energy consumption.\n\nReferences:\n\n[1] Abreu, Steven, Sumit Bam Shrestha, Rui-Jie Zhu, and Jason Eshraghian. \"Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2.\" arXiv preprint arXiv:2503.18002 (2025).\n\n**We thank the reviewer for their thorough evaluation and valuable feedback. In light of the clarifications provided, we respectfully request the reviewer to reconsider their rating. We are happy to address any additional questions or comments.**"}, {"review_id": "vCiDNzj5Uc", "reviewer": "Reviewer_k5ce", "summary": "This paper proposes SpikingVTG, a novel SNN-based architecture designed for the Video Temporal Grounding (VTG) task. The method segments a video into fixed-length clips and predicts, for each clip, three values: a binary foreground indicator, a temporal offset pair, and a relevance score.\nA key contribution is the Saliency Feedback Gating (SFG) mechanism, which uses intermediate outputs to compute dynamic saliency scores and update video input features over discrete timesteps. The paper also formulates the convergence dynamics of the spiking transformer to steady-state ASR values and utilizes these in the training and inference pipeline.", "questions": "Could the authors clarify how convergence to equilibrium is defined and detected during training?\nHow do the authors ensure the compatibility of textual features and deep-layer video ASRs in the cosine similarity calculation within SFG?\nHas the computational cost of SFG (per-timestep cosine similarity) been evaluated on longer video sequences or real-time settings?\nIs the SFG module quantized in the 1-bit NF-SpikingVTG model? If not, could its floating-point operations impact the claimed energy efficiency?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\nThis is likely the first attempt to apply spiking neural networks to the VTG domain, introducing a new perspective on low-power temporal reasoning tasks.\nThe SFG mechanism is biologically inspired and dynamically adjusts input features based on feedback from average spiking rates (ASR), allowing for adaptive and sparse computation.\nWeakness\nAssumption of Equilibrium: While the paper claims that ASR values and saliency scores converge to equilibrium and provides partial visualizations, it lacks a rigorous empirical or theoretical analysis of the convergence process. A more precise treatment of convergence criteria, dynamics, generalizability, and other related aspects would strengthen the claims made in this section.\nFeature Space Compatibility: In the SFG module, the cosine similarity is computed between pooled text representations M and ASRs from deep layers a_i^{Nv[t]}. However, the compatibility between these feature types (from different depths and modalities) is not discussed.\nGating Mechanism Scalability: The computation of cosine similarity per clip and timestep may add overhead, and the potential latency of SFG as a feedback loop in larger-scale settings is not discussed.", "was_revised": true, "final_justification": "I appreciate the authors’ efforts in providing a thorough rebuttal and clarifications. Many of my earlier concerns have been addressed. My overall assessment remains unchanged at Borderline accept.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and insightful feedback. Below, we provide detailed responses to each of the questions raised.\n\n**Question 1**\n> Could the authors clarify how convergence to equilibrium is defined and detected during training?\n\n**Response**\nThank you for this question. In our framework, convergence to equilibrium is defined based on the discrete-time dynamics of per neuron average spiking rate (ASR). Below, we provide the overview of a theoretical proof of convergence following [10]. For simplicity we show the proof when all layers are linear and using Integrate-Fire neurons with binary spikes  \\([0, 1]\\) and define the average spiking rate (ASR) over \\(t\\) steps as:  $a_l[t] = \\frac{1}{t} \\sum_{\\tau=1}^{t} s_l[\\tau],$ which is bounded within \\([0, 1]\\) because there can be at most one spike per timestep.. For the $\\(l\\)$-th layer, the membrane potential update is given by:  \n\n$u_l[t + 1] = u_l[t] + W s_{l-1}[t] + b - V_{\\text{th}}\\ s_l[t + 1],$\nwhere $\\(s_{l-1}[t]\\)$ is the spike train from the previous layer and $\\(V_{\\text{th}}\\)$ is the firing threshold.  \n\n\nFollowing [10], we can decompose the membrane potential into negative and exceeded positive terms, $u\\_i[t] = u\\_i^{-}[t] + u\\_i^{+}[t]$. If the instantaneous input is negative, the neuron will not emit a spike, and this inhibitory contribution remains in the membrane potential as the **negative term** $u\\_i^{-}[t]$. The **positive term** $u\\_i^{+}[t]$ represents the excitatory drive, typically bounded between $0$ and $V\\_{\\text{th}}$. In our convergence analysis, we assume $u\\_i^{+}[t]$ is bounded by a constant. Thus, with a linear feedback ($F$) the  ASR of the first layer and subsequent layers becomes:  \n\n$a\\_1[t+1] = \\sigma\\\\left(\\frac{1}{V\\_{\\text{th}}} \\left(\\frac{t}{t+1} F a\\_N[t] + W\\_1 x[t] + b\\_1\\right)\\right) - \\frac{1}{V\\_{\\text{th}}} \\cdot \\frac{u\\_1^{+}[t+1]}{t+1}$,  \n$a\\_{l+1}[t+1] = \\sigma\\\\left(\\frac{1}{V\\_{\\text{th}}} \\left(W\\_{l+1} a\\_l[t+1] + b\\_{l+1}\\right)\\right) - \\frac{1}{V\\_{\\text{th}}} \\cdot \\frac{u\\_{l+1}^{+}[t+1]}{t+1}$, for $l = 1, \\dots, N-1$.\n\n\nwhere, $\\sigma(x) = 1$ if $x > 1$, $\\sigma(x) = x$ if $0 \\le x \\le 1$, and $\\sigma(x) = 0$ if $x < 0$.\n\n\nIf the average inputs converge $(x^{\\*})$, the exceeded terms are bounded $(\\(|u_l^{+}[t]| \\le c\\))$, and the product of spectral norms satisfies  $\\|F\\|\\_2 \\|W\\_1\\|\\_2 \\|W\\_2\\|\\_2 \\cdots \\|W\\_N\\|\\_2 \\le \\gamma V\\_{\\text{th}}^{N}$ with $\\gamma < 1$, then by iterative contraction we have:\n$\\[\n\\|a_1[t+1] - a_1[t]\\| \\le \\gamma \\|a_1[t] - a_1[t-1]\\| + \\epsilon (1-\\gamma)/2,\n\\]$\nfor sufficiently large \\(t\\). By Cauchy’s convergence test, $\\(a_1[t] \\to a_1^*\\)$, where:\n\n$a\\_1^\\* = f\\_1(f\\_N \\circ \\cdots \\circ f\\_2(a\\_1^\\*), x^\\*)$, and inductively $a\\_{l+1}^\\* = f\\_{l+1}(a\\_l^\\*)$ for all $l$. where $f\\_1(a, x) = \\sigma\\\\left(\\frac{1}{V\\_{\\text{th}}}(F\\ a + W\\_1 x + b\\_1)\\right)$ and  \n\n$f\\_{l+1}(a) = \\sigma\\left(\\frac{1}{V\\_{\\text{th}}}(W\\_{l+1} a + b\\_{l+1})\\right)$.  \n\nWe will add this proof to the appendix with more details.\n\n\n**Empirical Convergence**}: Empirically, we observe that the ASR across all layers converges within a small number of time steps, typically $T_c \\in [5,10]$. As shown in Fig.~3a of the main paper, the layer-wise ASR stabilizes rapidly. Consequently, we treat the ASR at the final simulation step, $a[T_{c}]$, as an approximate equilibrium value $a^*$. During training, we leverage this converged ASR to compute gradients via implicit differentiation at equilibrium (Section 4.5).\n\nWe hope this response clarifies both the theoretical foundations and empirical detection strategy for equilibrium convergence in our framework.\n\n**Question 2**\n\n> **How do the authors ensure the compatibility of textual features and deep-layer video ASRs in the cosine similarity calculation within SFG?**\n\n**Response**\n\nWe appreciate the reviewer’s thoughtful question. In the **Saliency Feedback Gating (SFG)** mechanism, the cosine similarity is computed between the **Average Spiking Rate (ASR)** of the final spiking transformer layer corresponding to each video segment\n  $\\( a^{N_v}_i[t] \\in {R}^D \\)$, and the **sentence-level textual embedding** $\\( M \\in {R}^D \\)$,\n\nas defined in Eq. (2) of the paper: $F^v_s[i,t] = \\cos(a^{N_v}_i[t], M) = \\frac{a^{N_v}_i[t] \\cdot M}{\\|a^{N_v}_i[t]\\|_2 \\|M\\|_2}.$\n\nTo ensure compatibility of these two modalities in the cosine similarity computation, we incorporate the following design choices:\nBoth $\\( a^{N_v}_i[t] \\)$ and $\\( M \\)$ reside in a latent space of dimensionality $\\( D \\)$. The query representation $\\( Q \\in \\mathbb{R}^{L_q \\times D} \\)$ is projected to a global sentence embedding ($\\mathbb{R}^{D}$) via attention-based pooling:$\\[M = Q^\\top \\text{Softmax}(Q W_p),\\]$ where $\\( W_p \\in \\mathbb{R}^{D \\times 1} \\)$ is a *trainable* projection matrix. This learnable mapping helps align the textual representation with the spatiotemporal semantics of the spiking video features, thereby enabling cross-modal compatibility.\n\n**Empirical validation:**  \nAs shown in **Fig. 2b** of the main paper, clips semantically related to the textual query exhibit higher saliency scores $\\( F^v_s \\)$, demonstrating that the representations are well-aligned. The effectiveness of this design is further reflected in **improved performance** and **reduced neural activity** with the SFG mechanism (see Table 5 and Fig. 3).\n\n**Question 3**\n> Has the computational cost of SFG (per-timestep cosine similarity) been evaluated on longer video sequences or real-time settings? Is the SFG module quantized in the 1-bit NF-SpikingVTG model? If not, could its floating-point operations impact the claimed energy efficiency?\n\n**Response**\n\n**Computational Cost of SFG:**  \n The SFG module computes per-clip saliency scores via cosine similarity between the output layer ASR ($a^{N_v}_i[t] \\in \\mathbb{R}^D$) and the global sentence embedding $\\( M \\in \\mathbb{R}^D \\)$.  This results in a total computational complexity of  $\\[\\mathcal{O}(L_v \\cdot D)\\]$  which is *linear* in the number of video segments $\\( L_v \\)$. In contrast, the spiking attention mechanism in the transformer core has quadratic complexity  $\\[\\mathcal{O}(L^2 \\cdot D + L \\cdot D^2)\\]$  where $\\( L = L_v + L_q \\)$. Thus, *even for long video sequences*, the computational cost of SFG is negligible compared to the spiking transformer core.  Moreover, the operations involved in SFG consist of dot products—element-wise in nature—and are computationally lightweight.\n\n**SFG in 1-bit (NF)-SpikingVTG:**  Yes, in the 1-bit (NF)-SpikingVTG model, the SFG module is fully compatible with the quantized architecture. No softmax operation is used in SFG. Instead, we apply ReLU followed by scaling as discussed in the paper avoiding non-local operations.  The projection matrix $W_p$ used to compute the sentence embedding is also quantized to 1-bit.\n\n**Energy Considerations of SFG**  While cosine similarity involves floating-point arithmetic (element-wise multiplication and summation), it does **not** require floating point matrix multiplications.  Therefore, its energy footprint is **significantly lower** than that of transformer attention layers.  For example, for the 1-bit NF SpikingVTG, considering cost of arithmetic operations on 45nm CMOS, while the transformer core consumes 1.3mJ the SFG mechanism only consumes ~0.01mJ only (Considering a video of 400 secs).\n\n**Thank you for your insightful review and comments. In light of our clarifications, we respectfully request the reviewer to reconsider their rating. We are happy to address any additional questions or comments.**"}]

Author rebuttal:

Meta-review: The paper proposes SpikingVTG, a spiking detection transformer for video temporal grounding (VTG) that introduces bio-inspired spiking neural networks for energy-efficient multimodal processing. Key contributions include the Saliency Feedback Gating (SFG) mechanism, Cos-L2 Representation Matching (CLRM) for knowledge transfer, and quantized normalization-free variants suitable for edge deployment. Strengths lie in its novelty, competitive performance on four benchmarks, and significant energy savings, particularly with the 1-bit NF-SpikingVTG. However, the model still underperforms compared to state-of-the-art full-precision VTG models, relies on heavy teacher models, and reports theoretical rather than real-device energy measurements. In rebuttal, the authors provided additional ablations, SFG visualizations, and deployment results on Intel Loihi 2 showing improved efficiency. Despite remaining performance gaps, the paper’s innovation, technical soundness, and potential impact support a borderline accept recommendation agreed by all the reviewers.
Decision: Accept (poster)