DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding
Authors: Thong Thanh Nguyen
Xiaobao Wu
Xinshuai Dong
Cong-Duy T Nguyen
See-Kiong Ng
Anh Tuan Luu
Venue: EMNLP 2023 Findings (2023)
Keywords: temporal language grounding
energy-based modeling
exponential-moving average
transformer

Abstract: Temporal Language Grounding seeks to localize video moments that semantically correspond to a natural language query. Recent advances employ the attention mechanism to learn the relations between video moments and the text query. However, naive attention might not be able to appropriately capture such relations, resulting in ineffective distributions where target video moments are difficult to separate from the remaining ones. To resolve the issue, we propose an energy-based model framework to explicitly learn moment-query distributions. Moreover, we propose DemaFormer, a novel Transformer-based architecture that utilizes exponential moving average with a learnable damping factor to effectively encode moment-query inputs. Comprehensive experiments on four public temporal language grounding datasets showcase the superiority of our methods over the state-of-the-art baselines.

DemaFormer:
Damped Exponential Moving Average Transformer with Energy-Baa -Based Modeling for Temporal Language Grounding
Thong Nguyen', Xiaobao Wu', Xinshuai Dong',
Cong-Duy Nguyen', See-Kiong Ng', Luu Anh Tuan?*
National University of Singapore, Singapore Nanyang Technological University, Singapore CCanegie Mellon University, USA
e0998147@u. nws. edu, anhtuan. luuêntu. edu.sg sg
Abstract
Temporal Language Grounding seeks to lo- calize video moments that semantically cor- respond to a natural language query. Recent advances employ attention mechanism to learn the relations between video moments and the text query. However, naive atten- tion might not able to appropriately capture such relations, resulting in ineffective distribu- tions where target video moments are difficult separate from the remaining ones. To re- solve the issue, we propose energy-based model framework to explicitly learn moment- query distributions. Moreover, propose De- maFormer, novel Transformer-based architec- ture that utilizes exponential moving average with a learnable damping factor effectively encode moment-query inputs. Comprehensive experiments on four public temporal language grounding datasets showcase the superiority of our methods over the state-of-the-ar base- lines. Our code and data are publicly available at https://github.com/. (the link is hidden now due the double-blind review).
(a) UMT
(b) DemaFormer
Figure 1: Visualization (t-SNE) moment-query rep- resentations of an input example by the previous best UMT baseline and our DemaFormer model. The target localizations are from the QVHighlights dataset label. Detailed input content is provided in Figure 2.
Tost quary: The someer wassing aurglassso crcaaso untal colrrid iu lniiige user the rier.
29
Figure A TLG example. To produce the output, we form the union of overlapped temporal boundaries in the groundtruth and models' localized moments. The UMT output about countryside scenes, which hardly align with the language query.
1 Introduction
2019b) or similarity functions (Anne Hendricks et al., 2017; Hendricks et al., 2018) to fuse tex- tual and visual features. To further enhance the localization performance, recent works divide the video into equal-length moments and employ the attention mechanism of the Transformer model to learn the relations between video moments and the language query. For example, Moment-DETR model (Lei et al., 2021) concatenates the visual mo- ments and the textual tokens, and then passes the concatenated sequence the Transformer encoder to capture the alignment. The UMT model (Liu et al., 2022) includes an additional audio channel into the Transformer encoder to construct unified architecture.
Temporal Language Grounding (TLG) is a task to determine temporal boundaries of video mo- ments that semantically correspond (relevant) to a language query (Hendricks et al., 2018; Gao et al., 2021a). TLG is complex and challenging task, since video processing demands understand- ing across multiple modalities, including image, text, and even audio. However, TLG has received increasing attention CV and NLP communities because it provides myriad usage for further down- stream tasks, c.g. VQA (Lei et al., 2018; Yc et al., 2017; Wang et ul., 2019a), relation extraction (Gao et al., 2021b), and information retrieval (Ghosh et al., 2019).
Early methods for TLG deploy concatenation with linear projection (Gao et al., 2017; Wang et al.,
However, for the TLGG task, previous works have shown that such attention-barn approach is still insufficient to capture the rich semantic interaction
Cooresponniig Author

---

between the text query and video moments (Xu al., 2023). As in Figure 2, the localized video mo- ments hardly align with the query statement. More- over, the attention mechanism the Transformer encoder does not assume any prior knowledge to- wards the input elements (Ma et al., 2022). For language localization, this design choice does not leverage the fact that video moments in temporal neighborhoods tend to exhibit closely related fea- tures. Therefore, this approach could lead inef- fective modeling of joint moment-queet inputs. As evidence, Figure demonstrates the distribution of joint mouent-query representations. Particularly, the representations of target moment-query local- izations and the remaining ones mingle together, making the grounding task more challenging.
moment-query inputs. We propose Demaloomer in which we equip the Damped Exponential Moving Average (DEMA) computation for the TransFormer architecture. Technically, the compu- tation applies exponentially decaying factors that consider the information from adjacent inputs. We further introduce learnable damping coefficients to enable the model to absorb adjacent information in sufficient manner that ensures distinction among inputs. Eventually, we combine the DEMA com- putation with the attention mechanism to construct Former encoder and decoder modules. To sum up, the contributions of our paper can be summarized as follows:
We propose DemalFomeer novel architec- ture for temporal language grounding. De- maFormer integrates exponential moving av- erage with learnable damping coefficients into the attention mechanism to appropriately capture dependency patterns among video- language inputs.
To address these limitations, we dive deeper into polishing the distribution of moment-query repre- sentations. In addition to supervised training to correctly localize the language in the video, we per- form unsupervised training to explicitly maximize the likelihood of moment-query localizations. This could help the multimodal model focus on captur- ing the distribution of target moment-query pairs and distinguishing them from others.
We propose a novel energy-based learning framework for temporal language grounding. The objective for the energy-based model can be formulated as contrastive divergence to assist a classical grounding loss for modeling moment-query representations.
As such, we propose to model the distribution moment-query representations under the frame- work of the Ehergy-Basee Model (EBM). In con- trast to other probabilistic models such as normal- izing flow (Rezende and Mohamed, 2015) or au- toencoder (Kingma and Welling, 2013), the EBM framework allows directly integrate the video moment's salience score into the density function, which results in accurate modeling of moment- query representations. Our implementation devel- ops into contrastive divergence objective which aims to minimize the energy of the relevant local- izations while maximizing the energy of the de- viating ones. Accordingly, the framework needs negative samples to represent high-energy regions. Therefore, we adapt the Langevin dynamics equa- tion to directly sample negative inputs from the EBM distribution. Such approach is appropriate because in the beginning the distribution will not match the true distribution, hence the generated samples are assured to be negative. As the train- ing progresses, the distribution will approximate the true distribution, consequently the Langevin equation is able to produce hard negative samples.
We conduct extensive experiments to demon- strate the superiority of our method over pre- vious state-of-the-ar baselines. Furthermore, we conduct comprehensive ablation studies evaluate our component proposals and deliver meaningful insights.
Related Work
Temporal Language Grounding (TLG), Intro- duced by Gao al. (2017); Anne Hendricks et al. (2017), TLG is locate relevant vidco moments given a language query. Early approaches use LSTM to encode language query and CNN for vi- sual clips, and then estimate cross-modal similarity scores (Anne Hendricks et al., 2017: Hendricks et al., 2018). Modern techniques leverage attention mechanism and structured graph network to learn the video-language relationship (Xiao et al., 2021; Gao et al., 2021a; Zhang al., 2020a; Yuan al., 2019). Recent works (Liu al., 2022: Lei et al., 2021) apply Transformer components eliminate hand-crafted pre- processcss and post-processing steps and make the model end-to-end trainable.
In addition, we incorporate the inductive bias that captures local dependencies among the

---

Vision-Language Representation Learning. Modeling the vision-language interaction is important for vision-language tasks (Gao et al., 2021a; Nguyen et al., 2022b,a, 2023; Wei et al., 2022, 2023, 2024). Previous works propose diverse techniques, including circular matrices (Wu and Han, 2018), dynamic filters (Zhang et al., 2019), Hadamard product (Zhang et al., 2020a), and contrastive learning (Nguyen and Luu, 2021). To better learn fine-grained token-level and movent-level cross-modal relations, several authors adapt graph neural networks with graph convolutional techniques (Gao et al., 202 Zhang et al., 2019).
where a E (0, 1)dd denotes the weighting coeffi- cients, 8 E (0, 1)d the damping coefficients, and @ the elementwise product. Both a and are randomly initialized and learnable during training. Subsequently, we project the hidden state I, back to the original input space:
x = Linear(1;).
(3)
DEMA Attention. the DEMA output in Eq. (3) and pass the output through non-linear layer:
Given the input X, we obtain
X' DEMA(X),
(4)
Z = SiLU(Linear(X"")
(5)
3 Methodology
where SiLU denotes the self-gated activation func- tion (Ramachandran et al., 2017; Elfwing et al., 2018). We experiment with other activation func- tions in Appendix D. Subsequently, we perform the attention operation and utilize Z which exhibits local dependencies as the value tensor:
Our task is to localize moments in videos from nat- ural language queries. Formally, given language query q of bn tokens and video composed of L equal-length input moments, where each moment is represented by visual frame sampled from the mo- ment, we aim to localize Lm time spans from o that is aligned with the query q, noted as {(4, ) where each moment spans from 4 to ry scaled by the video timelength and Imm < Lap.
Q Linear( (X),
(6)
K Linear (X)
(7)
V Linear( [Z);
(8)
Thus, we first describe our proposed damping exponential moving average attention for modeling video-language inputs in Section 3. the overall architecture in Section 3.2, and the training strategy empowered with energy- based modeling in Scction 3.3 and 3.4.
'KT Vdk
z' = softmax
V,
(9)
where dx denotes the dimension of K. Thereafter. aggregate the original input X and the attention output Z' in an adaptive manner:
3.1 Damping Exponential Moving Average (DEMA) Attention
X = sigmoid(Linear) (x)iieerrr))))))))))))))))))))))
(10)
P P= SiLLU(Linear(X)) (X') + Linear( / 7 Z')),
this section, we consider the input to our De- maFormer encoder Xe and decoder Xa in Section 3.2 as the general input X } of length Lx. We delineate the exponential moving average (EMA) with the damping influence applied on X follows.
(11)
= -
(12)
3.2 Overall Architecture
Figure illustrates our DemaFormer model for temporal language grounding. We explain the ar- chitecture in details in the following.
DEMA Computation. At first, we use a linear layer to map each input x; to an intermediate space:
Uni-modal Encoding, Given video consisting of L, moments and text query with Ly tokens, we employ pre-trained models to extract visual moment features F (f) textual features T = [t and audio features A = {at Audio-Dependent Video Encoding. For video- audio encoding, because audio signals pos- sess heavy noisy information (Liu et al., 2022: Badamdorj et al., 2021), we only perform 1-layer
g: Linear(x:),
Then, we estimate the current hidden state 1, as the sum of the previous hidden state 1 and the current intermediate input g: with the weighting coefficients that decrease exponentially and are re- laxed by damping coefficients:
1 = a G g: + (1 a
(2)

---

Salence
Center
Width
Offect
Liter
Iites
Linee
Linear
N,
Nzx
Norm
ReLU Noniinear
ReLU Horinear
SAL Alinios
Norm
Di MA Atjertion
DEMA Allertion
DEMA oedgen
beddinn
Videe Emriwasding
Teudl Embordiing
Momenii-Queyy Representations
Cross- Attenin
+D(D()
Apren
glanss..
Audio
Texd
Figure 3: Illustration proposed DemaFormer. Our archtiecture comprises an encoder of N₂ layers and a decoder of Na layers. We designate the first L, encoder outputs moment-query representations to become the input for the DemaFormer decoder.
attention to fuse the audio information into the vi- sual sequence. Particularly, the attention between the video and audio input becomes:
a non-lincar layer, and a residual connection:
H Norm (DEMA (x (x-1)) (xx)))
(18)
M 0 Norm (ReLU (H(10 + H
AFT Vd
= F softmax
F.
(13)
(19)
DemaFormer Encoder. Inspired by (Lei et al., 2021), concatenate the audio-dependent video and language tokens to form the input sequence:
Analogous to the encoder, we retrieve the Na-th layer output as the final output Og of the decoder:
O₂ 02=0% (N))
(20)
X, T].
(14)
Prediction Heads. For each output Od,i we des- ignate four separate linear layers to predict the salience score á, the center é₁ the center offset ço2, and the moment width dř;:
We push the input sequence X, to the DemaFormer encoder of N, encoder layers. Each encoder layer comprises DEMA attention layer, normaliza- tion layer, ReLU non-linear layer, and residual connection:
8, = Linear 04,), é Linear(o.,)) (21) co; = Linear Lin,ar(o.,,), e0 = Linear(o,,). (22)
(((+1) Norm (DEMA (x()))
(15)
Thus, each candidate moment's temporal bound becomes (a. + co; des + co, At test time, we extract the top-Lm moments whose salience scores are the largest.
((++1) = -000 = Norm [ReLU ((())) + (!)))
(16)
where (1) and !!)) denote the input and the output of the i-th encoder layer, respectively; !!! the
3.3 Energy-Based Models for Modeling Moment-Query Representations
intermediate output of the i-th encoder layer.
Given our joint vidco-language decoder outputs Oa \/ we designate the EBM to specify the density of Oa via the Boltzmann distribution:
We take the output of the Na-th encoder layer as the final output O₂ of the DemalFormer encoder.
O. = ((N).
(17)
po(oas) exp(-EE(Oz.)) Zo
(23)
DemaFormer Decoder. The input for the decoder is the first L, DemaFormer encoder outputs, i.e. Xa = !. The input sequence is forwarded to Nº decoder layers, each of which is composed DEMA attention layer, a normalization layer,
where Eg denotes the energy func- tion and Zn the normalizing constant Zp f exp (-Eo(O..)) do Inspired by (Du and Mordatch, 2019), we adopt Langevin

---

dynamies to conduct sampling from the above distribution:
while during testing Lm selected based on valida- tion. We define the matching loss Lmatch between predictions and groundtruth as:
d,s -dd Vo.; Eg (olt ") t*)))
(k) (k-1)
2
1 Lm
(24)
Lmarch
(8
A1llc Adll
Lm
(30)
(b) ~ r), idg = 0, d,s
(0)
(25)
A2l|ws r0:ll - \3 - - -000-1)ll1
where y is a hyperparameter to specify the vari- ance of the noise. We perform the Eq. (24) for K steps and take i'pi Our target is to better align the video-query rep- resentation Od. by minimizing the negative log likelihood of the muurnt-query representations, i.e. [NLL(() = Eoa.: [log [logro (o,0),)) This can be achieved by differentiating the (NLL(O) and opti- mize the resulting contrastive divergence for the gradient,
where (1122,3,4 denote the hyperparameter weights for the salience, center, width, and offsct losses, respectively. We jointly optimize the matching loss with the EBM negative log-likelihood (NLL) loss as follows:
(K) the sampling outcome.
== Lmach + ANLL
(31)
where ANLI denotes the weight to scale the NLL loss size.
Experiments
VoCNLL = Bo, [oEo(o B
(oEo(o
4.1 Datasets
(26)
whose detailed derivation can be found in Appendix
We evaluate our methods on four benchmark datasets for the temporal language grounding task: QVHighlights, Charades-STA, YouTube High- lights, and TVSum.
A. Because the samples generated by Eq. (24) do not approximate the true distribution in the begin- ning but will gradually converge to, we take these samples o,;; and assign their energy values de- caying weight with minimum value of Omin- We take the moment-query inputs whose groundtruth salience scores are larger than a threshold p as pos- itive samples o..
@Viighlights is collected by (Lei et al., 2021) to span diverse content on 3 major topics: daily vlog. travel vlog, and news. There are 10,148 videos with 18,367 moments associated with 10,310 queries. We follow (Lei et al., 2018; Liu et al., 2022) to split the dataset into 70% train, 15% val, and 15% test portions.
Moreover, because we maximize salience scores while minimizing the energy values of the posi- tive input (vice versa for the negative input), implement the negative salience-energy relation:
Charades-STA (Gao et al.. 2017) consists of videos about daily indoor activities. The dataset is split into 12,408 and 3,720 moment annotations for training and testing, respectively.
Eg(os,)) = -8
(27)
YouTube Highlights is prepared by (Sun et al., 2014) comprise six categories, i.c. dog, gymnas- tics, parkour, skating, skiing and surfing. each category, we inherit the original training-testing split as benchmark for the TL.G task.
As such, B becomes the DemaFormer's parameters and we obtain the final LNLL'S formulation:
1
a max
1+
2 Repoch
1
Ommin
(28)
TVSum (Hong et al., 2020) is a video summariza- tion dataset possessing 10 event categories. We employ the video title as the language query and the training/testing split of 0.8/0.2 for experiments.
CNLL = E EE(o) - a EE- Eo(Od,))
(29)
where Peepoch denotes the current training epoch.
4.2 Experimental Settings
Evaluation Metries. Our metrics include Rank k@@ μ. MAP@ μ. and Hir@ Rank k@ μ is the per- centage of the testing samples that have least one correct localization in the top-k choices, where lo- calization is correct if its IoU with the groundtruth
3.4 Training Objective
From a video-language input, we obtain Imm pre- dictions Ý = {6) é; co2, aū;) \ During train- ing Ly is the number of groundtruth localizations,

---

@
MAP@@
et al., 2022: Lei et al., 2021), for QVHighlights dataset, we also experiment with pre-training our ar- chitecture with noisy automatic speech recognition (ASR) captions before fine-tuning on the down- stream training samples. For all audio features, 55.60
Method
HIT@1
0.5
0.7
0.75
Avg.
CLIP
16.88
5.19
18.11
7.00
7.67
61.04
XML
41.83
30.35
44,63
31.73
32.14
55.25
XML+
46.69
33.46
47.89
34.67
34,90
$5,06
Moment-Dee
52.89
33.02 54.82
29.40
30.73
UMT
56.23 41.18 53.83 37.01
12
59.99
we use the PANN model pre-trained on AudioSet (Gemmeke et al., 2017). We provide detailed hy-
DemaFormer (Ours)
62.39 43.94 58.25
39.36 38.71
64.77
Moment-DETER w/ PT
59.78 40,33 60.51 35.36
36.14
60.17
perparameter settings in Appendix B.
UMT w/ PT
60.83
43.26 57.33 39.12 38.08
62.39
DemaFormer w/ PT
63,55 45.87 59.32 41.94 40.67
65.03
4.3 Baselines
Table I: Temporal language grounding results on the QVHighlights dataset. "w/ PT denotes pre-training with ASR-captions.
To evaluate the proposed methods, we compare our performance with a diversity of baselines:
RI@
RS@
Method
0.5
0.7
0.5
0.7
UMT (Liu et al., 2022): a multi-modal trans- former model to handle three modalitics, in- cluding audio, text, and video.
MAN
41.24
20.54
83.21
51.85
2D-TAN
39.70 23.31
80.32 51.26
DRN
42.90 23.68 87.80 54.87
Moment-DETR (Lei et al., 2021): a multi- modal transformer model that applies the original self-attention mechanism to encode no human prior and climinates manually- designed pre proppsscs and post-processing procedures.
RaNet
43.87 26.83 86.67 54.22
Moment-DETR
48.95 21.23 86,96 50.14
UMT
49.35
26.16 89.41 54.95
DemaFormer
52.63 32.15 91,94 60.13
Table 2: Temporal language grounding results on the Charades-STr dataset.
CLIP (Radford et al., 2021): framework of visual CNN and textual transformer models trained with contrastive objective.
is larger than the threshold μ. In similar manner, MAP@ μ is the mean average precision of localiza- tions whose IoU is larger than p. Hir@/ computes the hit ratio for the moment with the highest pre- dicted salience score in a video. We consider a moment is hit if its groundtruth salience larger than or equal to a threshold т. Following previ- ous works (Lei et al., 2021; Liu et al., 2022), we adopt Rank I @ with μ E (0.5, 0.75) and Hir@) I with = 4 for the QVHighlights dataset. For the Charades-STA dataset, we use Rank k@ μ with k E {1, 5} and μ E {0.5, 0.75). We apply mAP for both the TVSum and YouTube Highlights datasets. Implementation Details. For fair comparison with previous works (Liu et al., 2022: Lei et al., 2021), QVHighlights, use SlowFast (Feichtenhofer al.. 2019) and CLIP (Radford et al., 2021) to obtain features for the video moments and CLIP text encoder to obtain features for the language queries. For feature extraction of the Charades- STA dataset, we deploy VGG (Simonyan and Zis- serman, 2014) and optical flow features for video moments and GloVe embeddings (Pennington et al., 2014) for language tokens. On YouTube Highlights and TVSum, we utilize the I3D model (Carreira and Zisserman, 2017) pre-trained on Kinetics 400 (Kay et al., 2017) to extract moment-level visual representations, and CLIP text encoder to extract language representations. Furthermore, as (Liu
XML (Lei et al., 2020): framework of visual ResNet and textual RoBERTa models with a late fusion approach fuse the visual and tex- tual features. We include additional variant, XML+, which trained with the combination of our salience loss and the XML's loss.
RaNet (Gao et al., 2021a): : baseline with BiLSTM text encoder, CNN image encoder, and graph cross-modalithy interactor learn moment-query relations and select the target moments.
2D-TAN (Zhang et al., 2020b): method specify moment candidates as a 2D map where the row and column indices indicate the starting and ending points, respectively.
DRN (Zeng al., 2020): a dense regression network which treats all video moments as positive and seeks to predict its distance to the groundtruth starting and ending boundaries.
MAN (Zhang et al., 2019): a baseline with a structured graph network to model the moment wise temporal relationships.
TCG (Ye al., 2021): a multi-modal TLG architecture equipped with low-rank tensor fusion mechanism and hicrarchical temporal
context encoding scheme.

---

Dog
Gym.
Par.
Sur.
Avg.
LIM-S
$7.90
41.67
66.96
57.78
48.57
65. 08
56.38
DL-VHD
70.78
53.24
77.16
72.46
66. 14
76.19
69.26
MINI-Net
58.24
61.68
70.20
72.18
58.68
65.06
64.43
TCG
55.41
62.69
70.86
69.11
60.08
59.79
63.02
Joint- VA
64.48
71.92
80.78
61.99
73.23
78.26
71.77
UMT
65.87
75.17
81.59
71.78
72.26
82.66 62.66
74.92
DemaFormer
71.94
86.05
78.92
83.82
Table 3: Temporal language grounding results the YouTube Highlights dataset.
Joint-VA (Badamdorj al., 2021): ap- proach applying attention mechanism to fuse multi-modal features and a sentinel technique to discount noisy signals.
accomplishes the highest mAP scores across all categories and in overall as well. In detail, we
outperform the second-best UMT up to 19.34% at maximum on the BT portion. Analogous to the HViighlihhts experiments, this demonstrates that our framework can better model the video-language inputs in various contexts to polish the temporal language grounding performance. YouTubb Highlights. Equivalent to TVSum, our DemaFormer with the energy-based modeling approach outperforms prior competitive models across various subsets. Specifically, we gain mAP increases of 1.16% at minimum on the surfing por- tion and 6.07% at maximum the dog portion. We attribute such improvement to the more effec- tive modeling operation of the proposed DEMA computation attention, since can exhibit local dependencies of moment- query inputs for appro- priate modeling in various contexts.
MINI-Net (Hong et al., 2020): weakly su- pervised learning approach that trains pos- itive bag of query-relevant moments to pos- sess higher scores than negative bags of query- irrelevant moments.
LIM-S (Xiong al., 2019): TLG approach that leverages video duration weak super- vision signal.
DL-VHD (Xu et al., 2021): framework ap- plying dual learners to capture cross- cross-category concepts and video moment highlight notions.
4.4 Comparison with State-of-the-arts
We report results of our DemaFormer and the base- lines in Table 1. 2, 3, and 4 on the QVHighlights, Charades-STA, YouTube Highlights, and TVSum datasets, respectively. As can be seen, methods significantly outperform previous approaches. QVHighlights. Compared with the previous best method UMT, our DemaFormer achieves 2% abso- lute improvement least across all evaluation set- tings Rank 1 @ μ., particularly 4.54% for μ 0.5 and 2.01% for 0.7, respectively. When pre- trained with the ASR captions, our method outper- forms UMT with 2.59% of mAP average and 2.64 points of Hir@ I. These results demonstrate that our method can enhance the TLG operation in diverse settings, including daily vlog, travel vlog, and news.
4.5 Ablation Studies
In this section, we study the impact of (1) Damped Exponential Moving Average (DEMA), (2) Energy- Based Modeling (EBM), (3) Langevin Sampling Steps, and (4) Choice of Energy Functions.
With Without DEMA. From Table 5, remov- ing the damping factor results slight performance decrease, for example 1.04% and 0.26% in terms of Rank @ on QVHighlights and Charades- STA, respectively. The main reason is that with- out the damping coefficient, the model lacks the ability to adjust the information injected into ad- jacent input elements, such that the amount could be excessive to make it hard distinguish video moments. Moreover, we observe that completely eliminating the DEMA computation leads to signif- icant decrease, specifically up to 2.97% and 2.51% of Rank I 0.55 respectively on QVHighlights and Charades- since the model no longer specifies the moment-query distribution effectively.
Charades-STA. We increase the performance of UMT with 3.28% in terms of Rank / @@0.5 and 2.53% in terms of Rank 5@0.5. Upon tighter μ 0.7, we achieve larger degree of enhancement with 5.99% for Rank and 5.18% for Rank 5. We hypothesize that this is because modeling can focus on separating highly relevant localizations from other video moment candidates. TVSum. In Table 4, we compare our model with other competitive approaches. Our architecture
energy-basee
With vs Without EBM. Investigating the last rows of Table 5, we realize that adopting energy- based modeling can improve the temporal language

---

Method
VU
PK
FM
D8
LIM-S
55,92
42.92
61.21
53.99
60.36
47.50
43.24
66.31
69.14
62.63
56.32
DL-VHD
$6.51
68.74
74.93
86.22
79.02
63.21
58.92
72.63
78.93
64.04
73.32
MINI-Net
80.64
68.31
78.16
81.83
78.1
65.82
57.84
74.99
80.22
63.49
73.14
TCG
71.43
71.61
Voint-VA
83,73
57.32
78.54
86.08 08
80,10
69.23
70.03
73.03
97.44 44
67.51
76.30
UMT
57.54
81.51
88.21
78.83
81.42
87.04
75.98
$6.93
79.64
83.14
83.02
DemaFormer
88.73
85.15
92.11
N0.82
82.20
89.18
80.36
89.06
99.98
85.24
87.92
Table 4: Temporal language grounding results on the TVSum dataset.
R1
Dataset
Method
0.5
0.7
DemaFormer
62.39
w/o damping
60.52
42.90
OVHighiighss
63.5
w/o DEMA
59.42
42.06
60.0
EBM
DemuFormer
52,63
32.15
75.0
Charades STA
w/o damping
51.05
31.89
725
w/o FBM
49.96
30.77
Figure 4: Effect of the number of Langevin sampling steps upon localization performance on the VU and GA portions of the TVSum dataset and the skating portion of the YouTube Highlights dataset..
Table 5: Performance comparisons QVHighlights and Charades-STA datasets in ablative experiments of DEMA and EBM components of Demarormer.
RI@
RS@
Teas quary: The woman wearing su sunglesses crossis small colortul bridge over the river
Energy Function
0.5
0.7
0.5
0.7
Elementwise Cosine Similurity
31.08
90.74
Pooling-bused -hased Cosine Similarity
51.85
29.97
88.98
58.86
Salience score
52.63
32.15
91.94
60.13
Qualitative visualization of DemaFormer model. Green arrow line denotes the predicted localiza- tion and green normal line the predicted salience scores, Red arrow line denotes the groundtruth localization and red normal line the annotated salience scores.
Figure 5:
Table Performance comparison on Charades-STA dataset in ablative experiments of the energy function choices.
grounding performance. Particularly, adding the EBM training objective brings enhancement of 2.67% on Charades-STA and 3.16% on QVHigh- lights terms of Rank 1.5.5. This substantiates that the EBM can successfully capture the distri- bution of moment-query representations in which relevant localizations are separated from the irrele- vant ones. We provide more illustrative analysis in Section 4.6 and Appendix F.
mentwise and pooling-bant -based manner (we provide the formulations Appendix E), and evaluate the performance of the variants in Table 6. As the comparison shows, directly utilizing the salience score provides localizations with the most accuracy. This suggests that similarity functions do not fully implement the query-rele relevance concept.
4.6 Qualitative Analysis
Langevin Sampling Steps. We investigate the impact of the number sampling steps K in our Langevin equation (24) upon DemaFormer. Figure 4 shows that DemaFormer's performance increases as K grows. However, as long as K passes the threshold 100, the model performance converges with negligible fluctuation. We hypothesize that at K 100 the added noise is sufficient to segregate target localizations from elsewhere.
We illustrate prediction example from the QVHighlights dataset by our DemaFormer in Fig- ure 1, and 5, We observe that our model correctly localizes target moments with respect to the user query. Our predicted salience scores also align with the groundtruth scores, which are measured by av- eraging the three annotated scores in the dataset. In addition, we also utilize t-SNE to visualize the moment-query representations of the example in Figure 1. We realize that the representations of the target localizations stay separately from the remain- ing ones, whereas those from the UMT model do mingle together. This could explain the accurate localilzation of DemaFormer and verifies the effec-
Choice of Energy Functions. We experiment with different choices of the energy function. Inspired by the contrastive learning works (Chuang et al., 2020: Oord et al., 2018), we compute the negative cosine similarity of video clips 0" = (m and query tokens 0 = {or,) Lo+tl in the ele-

---

tive modeling of the proposed DEMA mechanism combined with the energy-based modeling. We provide more examples in Appendix
dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 0299-6308
Ching Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. 2020. Debi- ased contrastive learning. Advances in neural infor- mation processing systems, 33:8765-8775.
5 Conclusion
In this paper, we propose DemaFormer, a novel neu- ral architecture for the temporal language ground- ing (TLG) task. By leveraging the exponential mov- ing average approach with a damping factor, De- maFormer is capable of incorporating local depen- dencies among moment-query localizations. Ad- ditionally, we propose snergy-based strategy to explicitly model localization distribution. On four public benchmarks for the TLG task, our method is effective and outperforms state-of- ap- proaches with a significant margin.
Yilun Du and Igor Mordatch. 2019. Implicit generation and generalization in oneryy-based models. arXiv preprint arXiv: 1903.08689.
Shiv Ram Dubey, Satish Kumar Singh, and
Bidyut Baran Chaudhuri. 2022. Activation
functions in deep learning: A comprehensive survey
and benchmark. Neurocomputiig..
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018.
linear units for neural network function approximation reinforcement learning. Newral Networks, 107:3-11.
6 Limitations
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slowfast networks for video recognition. In Proceedings of the TEEE/CVF in- ternational conference on computer vision, pages 6202 6211.
Our framework requires negative sampling via the Langevin dynamics equation. This incurs addi- tional compute cost while training the language grounding model. Also, although we propose general methods to enhance the grounding perfor- mance, have not studied their impact in cross- domain scenarios, where the model is trained upon one domain (c.g. skiing videos) and tested upon another (e.g. skating videos). We leave these gaps as future work to optimize our framework more diverse contexts and use cases.
Jialin Gao, Xin Sun, Mengmeng Xu, Xi Zhou, and Bernard Ghanem. 2021a. Relation-aware video read- ing comprehension for temporal language grounding. arXiv preprint arXiv:2110.05717
Jiyang Gao, Chen Sun, Zhenheng Yang. and Ram Neva- tia. 2017. Tall: Temporal activity localization via language query. In Proceedings of the IEEE interna- tional conference on computer vision, pages 5267- 5275.
7 Acknowledgement
Kaifeng Gao, Long Chen, Yifeng Huang, and Jun Xiao. 202 Video relation detection via tracklet based visual transformer. In Proceedingy of the 29th ACM International Conference on Multimedia, pages 4833- 4837.
This research/project is supported by the National Research Foundation, Singapore under its AI Sin- gapore Programme (AISG Award No: AISG3-PhD 0023-08-05171 Thong Nguyen is supported by Google Ph.D. Fellowship in Natural Language Pro- cessing.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore. Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP). pages 776-780. IEEF.
References
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812.
Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann. 2019. Excl: Extractive clip localization using natural language descriptions. arXiv preprint arXiv: urXiv:1904.000 1904.02755
Taivanbat Badamdorj, Mrigank Rochan, Yang Wang. and Li Cheng. 2021. Joint visual and audio learning for video highlight detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8127-8137.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic. Trevor Darrell, and Bryan Russell. 2018. Localizing moments in video with temporal language. arXiv preprint 18X
Dan Hendrycks and Kevin Gimpel. 2016. Gaus- sian error linear units (gelus). arXiv preprint 100iv:1606.08475
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics

---

Fa-Ting Hong, Xuanteng Huang, Wei-Hong Li, and Wei-Shi Zheng. 2020. Mini-net: Multiple instance ranking network for video highlight detection. In Computer Vision-ECCV 2020: 16th European Con- ference, Glavgow, UK, August 23-28, 2020, Proceed- ings, Part XIII 16, pages 345-360. Springer.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03745
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP)) pages 1532-1543.
Will Kay. Joao Carreira, Karen Simonyan, Brian Zhang. Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence machine learning, pages 8748-8763 PMLR.
Diederik P Kingma and Max Welling. 2013. Auto- encoding variational bayes. arXiv preprint arXiv:
Jie Lei, Tamara L Berg, and Mohit Bansal. 2021. De- tecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 11846-11888 1858.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. 2017. Searching for activation functions. arXiv preprint urXiv:1710.0544
Danilo Rezende and Shakir Mohamed. 2015. Varia- tional inference with normalizing flows. In Interna- tional conference on machine learning, pages 1530- 1538. PMLR.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara I. Berg. 2018. Tvqa: Localized, compositional video ques- tion answering. arXiv preprint 1XXXv:88090016000
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. Tvr: A large-scale dataset for vidco-subtitle moment retrieval. In Computer Vision-ECCC ECCV 2020: 16th European Conference, Glasgow, UK, August 23- 28, 2020, Proceedings, Part XXI 16, pages 447-463, Springer.
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for scale image recognition. arXiv preprint arXiv: 401.4159..55
Min Sun, Ali Farhadi, and Steve Seitz. 2014. Rank- ing domain-specific highlights by analyzing edited videos. In Computer EscionEECCV 2014: 13th Eu- ropean Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 787-802. Springer.
Ye Liu, Siyuan Li, Yang Wu, Chang- Wen Chen, Ying Shan, and Xiaohu Qie. 2022. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Put- tern Recognition. pages 3042-3051.
Anran Wang. Anh Tuan Luu, Chuan-Sheng Foo, Hongyuan Zhu, Yi Tay, and Vijay Chandrasekhar, 2019a. Holistic multi-modal memory network for movie question answering. IEEE Transactions Image Processing, 29:489 499.
Xuezhe Ma. Chunting Zhou, Xiang Kong, Junxian He. Liangke Gui, Graham Neubig, Jonathan May. and Luke Zettlemoyer. 2022. Mega: moving av- erage equipped gated attention. arXiv preprint arXiv:2209. 105555.20..0065.
Weining Wang. Yan Huang, and Liang Wang. 2019b. Language-drivee temporal activity localization: A semantic matching reinforcement learning model. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 334-343.
Thong Nguyen and Anh Tuan Luu, 2021. Contrastive learning for neural topic model. Advances in neural information processing systems, 34:1 1974-1 1986.
Jie Wei, Guanyu Hu, Luu Anh Tuan, Xinyu Yang, and Wenjing Zhu. 2023. Multi-scale receptive field graph model for emotion recognition in conversa- tions. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Process- ing (ICASSP), pages 1-5, IEEE.
Thong Nguyen, Cong-Duy Nguyen, Xiaobao Wu, See-Kiong Ng. and Anh Tuan Luu. 2022a. Vision-and-language pretraining. arXiv preprint 071i12220..0172..
Thong Nguyen, Xiaobao Wu, Xinshuai Dong, Anh Tuan Luu, Cong-Duy Nguyen, Zhen Hai, and Lidong Bing. 2023. Gradient-boosted decision tree for listwise context model in multimodal review helpfulness pre- diction. arXiv preprint arXiv:2305.126: 12678.
Jie Wei, Guanyu Hu, Xinyu Yang. Anh Tuan Luu, and Yizhuo Dong. 2022. Audio-visual domain adaptation feature fusion for speech emotion recognition. In INTERSPEECH, pages 1988-1992.
Thong Nguyen, Xiaobao Wu, Anh-Tuan Luu, Cong- Day Nguyen, Zhen Hai, and Lidong Bing. 2022b. Adaptive contrastive learning multimodal trans- former for review helpfulness predictions. arXiv preprint arXiv:2211.0352.
Jie Wei, Guanyu Hu, Xinyu Yang. Anh Tuan Luu, and Yizhuo Dong. 2024. Learning facial expression and body gesture visual information for video emo- tion recognition. Expert Systems with Applications. 237:121419.

---

Aming Wu and Yahong Han. 2018. Multi-modal circu- lant fusion for addo-to-language and backward. In IJCAI, volume 3, page 8.
Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020b. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pages 12870-12877.
Shaoning Xiao, Long Chen, Jian Shao, Yueting Zhuang, and Jun Xiao. 2021. Natural language video localiza- tion with learnalle moment proposals. arXiv preprint arXiv:2109, 10678.
Bo Xiong, Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman. 2019. Less is more: Learning high- light detection from video duration. In Proceedings of the IEEE/CVF conference on computer vision and paliern recognition, pages 1258-1267.
Minghao Xu, Hang Wang, Bingbing Ni, Riheng Zhu, Zhenbang Sun, and Changhu Wang. 2021. Cross- category video highlight detection via set-based learn- ing. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 7970-7979.
Yifang Xu, Yunzhuo Sun, Yang Li, Yilei Shi, Xiaoxiang Zhu, and Sidan Du. 2023. Mh-detr: Video moment and highlight detection with modal transformer. arXiv preprint arXiv:2305.00355.
Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang. Qi Bi, Ping Li, and Guang Yang. 2021. Temporal cue guided video highlight detection with low-rank audio- visual fusion. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 7950-7959.
Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, and Yueting Zhuang, 2017. Video question answering via attribute-and attribute-augmented attention network learning. In Proceedings of the 40th International ACM SIGIR conference Research and Develop- ment in Information Retrieval, pages 829-832 -322.
Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu. 2019. Semantic conditioned dynamic modulation for temporal sentence grounding in videos. Advances in Neural Information Processing Systems, 32.
Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan. 2020. Dense regression network for video grounding. In Proceed- ings of the JEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 10287-10226.
Da Zhang. Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. 2019. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1247-1257.
Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. 2020a. Span-based localizing network for natural language video localization. arXiv preprint arXiv:2004.13931

---

R1 @
mAP@
Losses
HIT@1
0.5
0.7
0.5
0.75
Avg.
C, bi
57.29
40.97
54.29
36.61
35.98
60.84
L + Le + Lw + Loo
62.39
43.94
58.25
39.36
38.71
64.77
Table 7: Performance comparison on the QVHighlights dataset ablative experiments of the influence of localization loss terms our DemaFormer model.
A Gradient of the Energy-Based Negative Log-Likelihood Objective
We have the specification of the distribution of the moment-query representations:
po(o) exp(-Fog(o))
(32)
7
Zo /
(sp(-Fo(o))do.
(33)
We differentiate the negative log-likelihood of the representation:
8 log((p(o))
(34)
50
log
exp(-EE(())
(35)
88
Zn
(- log Zo - Eg(o))
(36)
50
log Sop(-Ealoe exp(-Es(o))dde Eg(o)
(37)
50
1
EEg do
(
cxp( Ea)
+
(38)
Zo
50
50
(EE(o')
(o)
(39)
50
80
o'-pp
VoEo(o*)) VoEo(o
(40)
We obtain the gradient formulation for the energy-based modeling component.
B Hyperparameter Settings
Our hidden dimension for the key tensor is dx = 256. Regarding our energy-based models (EBM), we adapt K- = 100 and y 0.1 for all datasets. For the training procedure, we adopt \ 3 A2 0.01, A3- 3, ANII. 0.1, and amin 0.1. To extract positive samples to train the EBMs, we use / 4,1, and 0.4 for the QVHighlights, Charades-STD YouTube Highlights, and TVSum datasets, respectively. During testing, adopt T.m. 10 based the validation performance. The number of DemaFormer encoder and decoder layers are as N: N₂ 2. For all training settings, we utilize the Adam optimizer with learning rate with batch size 32 for 200 epochs on the HVHighlights, batch size 8 for 100 epochs on the Charades-STr, batch size 4 for 100 epochs on the YouTube Highlights, and batch size for 500 epochs on the TVV Sum dataset, respectively.
and weight decay le We train our DemaFormer model
C Localization Losses
Following (Liu et al., 2022; Lei et al., 2021), we conduct experiments to study the effect of localization losses on our DemalFomerr architecture. We define L.
Lm
&; to be the salience loss term.

---

R1@
mAP@@
Activation Function
IIT@@
0.5
0.7
0.5
0.75
Avg.
Tanh
62.38
43.93
58.23
39.32
38.70
64.73
ReLU
62.36
43,90
58.22
39.27
38.68
64.71
GELU
62.34
43.89
58.18
39.26
38.66
64.67
43.94
58.25
39.36
38.71
64.77
Table 8: Performance comparison the QVHighlights dataset in ablative experiments of the activation functions.
Lo &gll to be the center loss term, L
Ce
Im E
||a
Σ
the width loss term, and lly wgyl
Tmm
Lco
co; co:ll the center offset loss term. Because the salience, center and width terms
mandatory, we justify the necessity of the center offset term. As can be seen from Table 7, with the center offset term the localization scores increase from 54.29% to 58.25% of mAP@0.. 0.5, and from 40.97% 13.91% of Rank / /.0.7. This demonstrates that the center offset term helps our DemaFormer architecture predict the localizations more precisely.
Choice of Activation Functions
In this appendix, we adopt different activation functions for our DEMA attention in Section 3.1 and compare their performances. In detail, we experiment with the Tanh (Dubey et al., 2022), ReLU (Dubey al., 2022), and GELU functions (Hendrycks and Gimpel, 2016). We report the temporal language grounding performance with these activation functions on the QVHighlights dataset in Table 8. We observe that DemaFormer exhibits negligible performance fluctuation. These results demonstrate the robustness of our proposed DemaFormer with respect the choice of activation functions.
Specification of Energy Functions
We provide the formulation of energy functions experiment with in Table 6.
Element-wise cosine similarity:
1 bi Σ
Ea(oss) = Lg
Oc,i
Ocg
(41)
Oc,i Oe.
j=
-.olling-base -ooling-based cosine similarity:
by = MaxPool
or. j= j= 11
(42)
0.
Eg(Oa,)
(43)
Oc.i 1
Salience score:
Eo(oa,)) = -
(44)
F
More prediction examples
this appendix, we present more predictions of our DemaFormer model in Figure 6, 7, and 8.

---

Text query: A man wearing a black sweater near a river, talking about bad weather and flooding.
(a) Input example
0.15
Larget arattaation
0.30
0.50
0.00
0.00
-0.00
-0.06
-5.15
lusilianiiss
0.10
0.00
0.00
1006
0.30
0.15
0.00
0.26
-0.16
-0.10
10
OL0S
0.30
0.30
0.15
Demaeormer
UMT
Figure 6: Prediction example| with the t-SNE visualizations of the DemaFormer model and the UMT model. Green arrow line denotes the predicted localization green normal line the predicted salience scores. Red arrow line denotes the groundtruth localization and red normal line the annotated salience scores.
Text query: Woman puts items plastic bags.
(a) Input example 2
largel kesaiiatioon
0.30
0.4
0.3
0.2
target
0.0
-0.06
11.1
-0.10
(b) DemaFormer
(c) UMT
Figure Prediction example 2 with the t-SNE visualizations of the DemaFormer model and the UMT model. Green arrow line denotes the predicted localization and green normal line the predicted salience scores. Red arrow line denotes the groundtruth localization and red normal line the annotated salience scores.

---

Text query: A man and a woman pose dressed like characters from video game.
(a) Input example 3
0.2
scuniiing mococs Iscest louizasiio
monart
tuggs
0,00
0.00
0.00
0.00
0.20
(b) DemaFormer
(c) UMT
Figure 8: Prediction example 3 with the t-SNE visualizations of the DemaFormer model and the UMT model. Green arrow line denotes the predicted localization and green normal line the predicted salience scores. Red arrow line denotes the groundtruth localization and normal line the annotated salience scores.


Reviews:
[{"review_id": "2OvlAX1z7G", "reviewer": "Reviewer_ZP2j", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We genuinely value your expertise and perspective, and we are committed to enhancing our work. We hope that our response can address your concerns. \n\n===========================================================================================\n\n**1. You and [1] use optical flow features. Why do not you cite and compare it?**\n\nThank you for mentioning this work. We respectfully note that [1] has multiple computational-heavy components including C3D, Faster-RCNN, and optical flow, and takes advantage of the use of extra video features, while the proposed method and all other baselines only use optical flow features. Therefore, directly comparing [1] with the proposed method and other baselines is not fair. For a fair comparison we follow the experimental setting of [2,3] where [1] is not included.\n\nWe agree that [1] should be cited and it would be even better if we can have an extra comparison where the proposed method and baselines all employ the same extra features as [1]. However, the authors of [1] have not released their code yet (though we have emailed them already), and thus we are not able to conduct such a comparison.\n\nWe thank the reviewer again for the comment and will cite [1] together with adding the above discussion to our paper to avoid further confusion.\n\n===========================================================================================\n\n**2. Missing some details about how to obtain $\\lambda_{\\text{NLL}}$?**\n\n$\\lambda_{\\text{NLL}}$ is a hyper-parameter to be tuned. In our implementation, we choose $\\lambda_{\\text{NLL}} = 0.1$ (mentioned in Appendix B). This is to ensure that the scale of $L_{\\text{NLL}}$ is approximately equal to $L_{\\text{match}}$ and empirically this choice of $\\lambda_{\\text{NLL}}$ works well. We have added this discussion to the paper to avoid further confusion.\n\n===========================================================================================\n\n**3. Parameter analysis experiments?**\n\nWe respectfully note that we do have conducted and included analysis of the choices of important parameters in our paper. Specifically, we conducted analysis upon the Langevin sampling steps (section 4.5, figure 4), the choice of energy functions (section 4.5, table 6), and the activation function of our DEMA attention (appendix D, table 8). Do you have any additional parameters that you expect us to analyze? Any further suggestions would be appreciated.\n\n===========================================================================================\n\n**4. Some typos, e.g., in line 385, it should be $\\mu \\in \\\\{0.5, 0.7\\\\}$ not $\\\\{0.5, 0.75\\\\}$. In Eq. (1), the comma should be replaced with period.**\n\nWe thank the reviewer for pointing them out and have revised accordingly. \n\n===========================================================================================\n\n**References:**\n\n[1] Exploring optical-flow-guided motion and detection-based appearance for temporal sentence grounding, IEEE Transactions on Multimedia 2023.\n\n[2] UniVTG: Towards Unified Video-Language Temporal Grounding, ICCV 2023.\n\n[3] Deco: Decomposition and reconstruction for compositional temporal grounding via coarse-to-fine contrastive ranking, CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method sounds reasonable although I do not check the reproducibility of provided codes carefully.\nConvincing ablation studies.\nThe motivation of this paper is clear and easy to understand.", "reasons_to_reject": "The performance is not state-of-the-art, e.g., in [1], on the Charades-STA dataset, its results are 68.98 (R1@0.5), 47.79 (R1@0.7), 96.82 (R5@0.5), 75.41 (R5@0.7). However, your results are 52.63 (R1@0.5), 32.15 (R1@0.7), 91.94 (R5@0.5), 60.13 (R5@0.7). You and [1] use optical flow features. Why donot you cite and compare it?\n\nMissing some details. e.g., how to obtain \\lambda_{NLL}? No parameter analysis experiments.\n\nSome typos, e.g., in line 385, it should be \\mu \\in {0.5, 0.7} not {0.5, 0.75}. In Eq. (1), the comma should be replaced with period.\n\n[1] Liu, Daizong, et al. \"Exploring optical-flow-guided motion and detection-based appearance for temporal sentence grounding.\" IEEE Transactions on Multimedia (2023).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes DemaFormer, a novel architecture for temporal language grounding. Since naive attention might not be able to appropriately capture the relations between video moments and  the text query, this paper introduces an energy-based model framework to explicitly learn moment-query distributions. A Transformer-based architecture utilizes exponential moving average with a learnable damping factor to effectively encode moment-query inputs.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AjSEkLNx2J", "reviewer": "Reviewer_goDS", "summary": "", "questions": "A. Why does the contrastive objective works better than the suprvised objective?\n\nB. You only do moving average on representaitons,  DEMA Attention may not be approriate.  Why do we need moving average, since self-attention may have captured global representaitons.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and feedback. We are happy to listen to your perspective and dedicated to polishing our work. We hope our response is able to address your questions.\n\n===========================================================================================\n\n**A. Why does the contrastive objective work better than the supervised objective?**\n\nOur final objective is a linear composition of $L_{\\text{match}}$ and $L_{\\text{NLL}}$ (Eq. 31, line 342), where $L_{\\text{match}}$ is a supervised loss that directly encourages grounding and $L_{\\text{NLL}}$ is a loss that maximizes the likelihood of “correct” video-text joint representations. As for the contrastive loss (as in Eq. 26), it is not to substitute the supervised $L_{\\text{match}}$ loss. Rather, the contrastive loss serves as a tool to make the maximization of $L_{\\text{NLL}}$ possible.\n\n**We do not want to claim that the contrastive objective works better than the supervised objective.** Rather, we believe that  $L_{\\text{match}}$ and $L_{\\text{NLL}}$ work together towards better grounding performance. If we only use the supervised $L_{\\text{match}}$ objective, we will be less able to distinguish the target moments from the remaining ones (figure 1), and thus achieve less satisfying predictions (figure 2; section 4.4). We thank the reviewer for pointing out the confusion and have added this discussion to the paper.\n\n===========================================================================================\n\n**B. Why do we need moving average, since self-attention may have captured global representations?**\n\nIn our framework, for each video frame we get a representation that is used to predict the center and the moment width, which is used as the prediction of grounding. Therefore, compared to global representations, we are more concerned with **local dependency** between temporally adjacent frames (as mentioned in lines 63-67). \n\nAs for self-attention, we agree that a self-attention layer has enough modeling capability and thus has the potential to capture local dependency. However, as it initially aims to model all the pair-wise relations, it necessitates a lot of high-quality data to capture the desired dependency, as mentioned in [1, 2]. In contrast, the proposed exponential moving average in itself can well capture the local dependency and thus serves as a kind of prior knowledge / inductive bias for better grounding performance.\n\nWe thank the reviewer for the question and have added this discussion to the paper to avoid further confusion.\n\n===========================================================================================\n\n**References:**\n\n[1] Co-Advise: Cross Inductive Bias Distillation, CVPR 2022.\n\n[2] How Do Vision Transformers Work?, ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper finds that implicit relation learning is ineffective, and proposes a effective solution to  adress the problem in expriments.\n\n2.  This paper is relatively complete, it has strong motivations, well-conducted model and training design, and satisfied experiments.", "reasons_to_reject": "The use of moving average and contrastive objective lacks understandings.  Refer to Questions for details.\n\n", "questions_to_authors": "A. Why does the contrastive objective works better than the suprvised objective?\n\nB. You only do moving average on representaitons,  DEMA Attention may not be approriate.  Why do we need moving average, since self-attention may have captured global representaitons.", "paper_topic_and_main_contributions": "This paper proposes a framework that helps match video frames and langauge.  The framework contains a moving-average mechanism that associate current time with previous time, a contrastive objective which aims to increase the divergence between the unrelated video and langauge and reduces the related ones . The framework achieves improvements over baselines on many temporal langauge grounding tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dddS9BlrGE", "reviewer": "Reviewer_ecVy", "summary": "", "questions": "1.\tIn Line 77-79, the authors proposed to use an unsupervised training model. How is it unsupervised? \n2.\tUnlike Charades-STA, in QVHighlighs, a query text will correspond to multiple video clips, can the learned moment-Query Representations handle multiple video clips? This paper seems only describe one target video clip case.\n3.\tThe author wrote in Line 313 that as training proceeds, the negative samples get closer to the positive samples, does this mean that the representations are gradually similar? What is the criterion for identifying the positive and negative samples during model training?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helpful comments. We appreciate your perspective and evaluation, and we are committed to polishing our work. We hope that our response can resolve your concerns and improve your ratings.\n\n===========================================================================================\n\n**CLARIFICATION**\n\n**_1. The motivation to introduce EBM._**\n\nThe motivation to introduce EBM is to model the distribution of target video representations (lines 79-85). By maximizing this distribution, we are able to have better separable representations that are useful for the grounding task (figure 1 and 2). In contrast, traditional Transformers fail to differentiate target video moments from the others well (as in lines 71-73; figure 1 left). It is possible to substitute EBM with other distribution modeling techniques. The reason why we choose EBM lies in that EBM provides a principled and flexible way for distribution modeling and likelihood maximization, and the negative energy score can be used as the saliency score in our model (lines 85-92; Eq. 27 on line 323).\n\n**_What is the meaning of energy here?_**\n\nIn the original EBM [3, 10], lower energy implies higher likelihood. In the proposed method, we additionally consider the negative energy score as a surrogate of the moment-query salience scores (in Eq. 27). This helps our learning of moment-query representations benefit from the modeling of the distribution of target video representations better.\n\n------------------------------------\n\n**_2. Why are the adjacent video features helpful to model joint moment–query representation (Line 64-68)? The temporally related video features should contain much redundancy._**\n\nWe agree that temporally neighboring frames should have redundant information. However, they also contain information that is crucial for each other. E.g., a previous frame may contain important context that is missing in the current frame, and borrowing this context information from adjacent frames to the current frame would facilitate better predictions. This is precisely why we propose the exponential moving average to capture such dependencies. As our weighting coefficient and damping coefficient are learnable during training (as in Eq 2 and line 202-203), it can be considered that we automatically learn the best trade-off between keeping useful information and abandoning redundant information from adjacent frames.\n\nWe thank the reviewer for the insightful question and have added this discussion to the paper to avoid further confusion.\n\n------------------------------------\n\n**_3. There is no description of EBM related works in Sec. 2 Related Work._**\n\nThank you for the valuable comments. [1] is about using EBM to learn the image distribution conditioned on another distribution generated by diffusion model, while [2] is about using EBM to resolve the VAE’s problem of generating out-of-distribution samples, all of which provide informative perspectives into our work. We thank the reviewer again and we have added a discussion about [1] and [2] to the related work section as you suggested.\n\n===========================================================================================\n\n**ANSWERS TO QUESTIONS**\n\n**_1. In Line 77-79, the authors proposed to use an unsupervised training model. How is it unsupervised?_**\n\nOur final objective is a linear composition of $L_{\\text{match}}$ and $L_{\\text{NLL}}$, where $L_{\\text{match}}$ is supervised and $L_{\\text{NLL}}$ is unsupervised in the sense of that $L_{\\text{NLL}}$ aims to maximize the likelihood of video moment inputs.\n\n------------------------------------\n\n**_2. Can the learned moment-query representations handle multiple video clips?_**\n\nYes, our learned moment-query representations can handle multiple video clips. During inference, we forward representations of multiple clip candidates to the prediction head and only select those whose predicted salience scores are larger than a threshold. We thank the reviewer for pointing out this confusion and we have added a sentence to the paper to avoid any further confusion.\n\n------------------------------------\n\n**_3. The author wrote in Line 313 that as training proceeds, the negative samples get closer to the positive samples, does this mean that the representations are gradually similar?_**\n\nYes, you are right. With the gradual convergence of negative samples towards positive ones, our EBM will learn to capture the underlying patterns of target moment-query localizations better. Such phenomenon is similar to the “hard negatives” discussed in [4,5].\n\n**_What is the criterion for identifying the positive and negative samples during model training?_**\n\nFollowing [6], we explicitly extract positive samples from the data and draw negative ones from the probabilistic model, i.e. our EBM (as mentioned in Eq. 24, line 297). By doing so, we are able to learn better moment-query representations as shown in Figure 1.\n\n===========================================================================================\n\n**References:**\n\n[1] Learning energy-based models by diffusion recovery likelihood, ICLR 2021.\n\n[2] Vaebm: A symbiosis between variational autoencoders and energy-based models, ICLR 2021.\n\n[3] Your classifier is secretly an energy based model and you should treat it like one, ICLR 2020.\n\n[4] Contrastive Learning with Hard Negative Samples, ICLR 2020.\n\n[5] Hard Negative Mixing for Contrastive Learning, NeurIPS 2020.\n\n[6] Generative adversarial nets, NeurIPS 2014.\n\n[7] Instance-Conditioned GAN, NeurIPS 2021.\n\n[8] A Progressive Conditional Generative Adversarial Network for Generating Dense and Colored 3D Point Clouds, International Conference on 3D Vision (3DV) 2020.\n\n[9] A survey on generative adversarial networks: Variants, applications, and training, ACM Computing Surveys 2021.\n\n[10] How to Train Your Energy-Based Models, arXiv 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper takes a novel perspective, i.e. introducing EBM, which distinguishes between target moments and other moments in the video by maximizing and minimizing the energy. \n2.\tThe designed DEMA module seems to be novel.\n3.\tThe state-of-the-art results are achieved on the four public datasets.", "reasons_to_reject": "1.\tThere is a lack of the motivation to introduce EBM. It is not clear to me why this module is helpful to model the distribution of moment-query representations. Besides, what is the meaning of energy here?\n2.\tIt is also not clear to me why the adjacent video features are helpful to model joint moment-query representation (Line 64-68). In my opinion, the temporally related video features should contain much redundancy, which not only increases computation cost, but also introduces noises.\n3.\tThere is no description of EBM related works in Sec. 2 Related Work. The authors might consider introducing some of the classic methods [1-2].\n\n[1] Learning energy-based models by diffusion recovery likelihood, ICLR 2021.\n\n[2] Vaebm: A symbiosis between variational autoencoders and energy-based models, ICLR 2021.", "questions_to_authors": "1.\tIn Line 77-79, the authors proposed to use an unsupervised training model. How is it unsupervised? \n2.\tUnlike Charades-STA, in QVHighlighs, a query text will correspond to multiple video clips, can the learned moment-Query Representations handle multiple video clips? This paper seems only describe one target video clip case.\n3.\tThe author wrote in Line 313 that as training proceeds, the negative samples get closer to the positive samples, does this mean that the representations are gradually similar? What is the criterion for identifying the positive and negative samples during model training?", "paper_topic_and_main_contributions": "In the Temporal Language Grounding task, naive attention is used in existing research to learn the relationship between video moments and textual queries, which results in ineffective distributions between features. To address this problem, this paper uses an energy-based model to achieve contrasting distributions of video representations. In addition, DemaFormer is proposed that uses the exponential moving average with a learnable damping factor to effectively encode moment-query inputs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Learning energy-based models by diffusion recovery likelihood, ICLR 2021.\n\n[2] Vaebm: A symbiosis between variational autoencoders and energy-based models, ICLR 2021.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}, {"review_id": "z6yQqSHs1G", "reviewer": "Reviewer_SbrT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your sincere note. If you have chances to review our future paper, we look forward to your insightful comments and feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "N/A", "reasons_to_reject": "N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "Dear Area Chair,\n\nI am not an expert in this field. If possible, could you please assign this paper to other reviewers? Thanks!", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces DemaFormer, a novel architecture designed for temporal language grounding. Since simple attention mechanisms may not effectively capture the relationships between video segments and textual queries, this paper introduces an energy-based model framework to explicitly learn distributions between moments and queries. The Transformer-based architecture employs an exponential moving average with a trainable damping factor to efficiently encode input from moments and queries. In summary, the proposed method is reasonable with clear and strong motivation. The experiments are well-conducted. In rebuttal period, the author addressed several concerns from reviewers. In particular, authors have performed an extra comparison where the proposed method and baselines all employ the same extra features as [1]. Although there was no clear answer from the reviewers, we can conclude that the results obtained from the experiment largely dispelled the reviewers' concerns about the performance. The clarification and discussion in the rebuttal should be added to the revised version. This paper has a high possibility of being accepted to findings.
Decision: Accept-Findings