Boundary Denoising for Video Activity Localization
Authors: Mengmeng Xu
Mattia Soldan
Jialin Gao
Shuming Liu
Juan-Manuel Perez-Rua
Bernard Ghanem
Venue: ICLR 2024 poster (2024)
Keywords: video activity localization
boundary denoising
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Video activity localization aims at understanding the semantic content in long, untrimmed videos and retrieving actions of interest. The retrieved action with its start and end locations can be used for highlight generation, temporal action detection, etc. Unfortunately, learning the exact boundary location of activities is highly challenging because temporal activities are continuous in time, and there are often no clear-cut transitions between actions. Moreover, the definition of the start and end of events is subjective, which may confuse the model. To alleviate the boundary ambiguity, we propose to study the video activity localization problem from a denoising perspective. Specifically, we propose an encoder-decoder model named DenosieLoc. During training, a set of temporal spans is randomly generated from the ground truth with a controlled noise scale. Then, we attempt to reverse this process by boundary denoising, allowing the localizer to predict activities with precise boundaries and resulting in faster convergence speed. Experiments show that DenosieLoc advances 
several video activity understanding tasks. For example, we observe a gain of +12.36% average mAP on the QV-Highlights dataset.
Moreover, DenosieLoc achieves state-of-the-art performance on the MAD dataset but with much fewer predictions than others.

Published as a conference paper at ICLR 2024
BOUNDARY DENOISING FOR VIDEO ACTIVITY LOCALIZATION
Mengmeng Xul Mattia Soldan'
Jialin Gao Bernard Ghanem'
Shuming Liu
Juan-Manuel Pérez-Rúa"
'King Abdullah University of Science and Technology (KAUST)
2 Natiooall University of Singapore
Meta Al
ABSTRACT
Video activity localization aims at understanding the semantic content in long, untrimmed videos and retrieving actions of interest. The retrieved action with its start and end locations can be used for highlight generation, temporal action detec- tion, etc. Unfortunately, learning the exact boundary location of activities is highly challenging because temporal activities are continuous in time, and there are often no clear-cut transitions between actions. Moreover, the definition of the start and end of events is subjective, which may confuse the model. To alleviate the bound- ary ambiguity, we propose to study the video activity localization problem from a denoising perspective. Specifically, we propose an encoder-decoder model named DenoiseLocc. During training. set of temporal spans is randomly generated from the ground truth with controlled noise scale. Then, we attempt to reverse this process by boundary denoising, allowing the localizer to predict activities with precise boundaries and resulting in faster convergence speed. Experiments show that DenoiseLoc advances several video activity understanding tasks. For exam- ple, we observe gain of +12.36% average mAP the QV-Highlights dataset. Moreover, Denoiseloc achieves state-of-the-ard performance on the MAD dataset but with much fewer predictions than others.
INTRODUCTION
The summation of human experience is being expanded at prodigious rate, making it imperative to devise efficient and effective information retrieval systems to aid the need for knowledge abstraction and dissemination. Recently, video data has emerged as the largest unstructured knowledge reposi- tory (Kay et al., 2017: Ran et al., 2018), making essential to develop algorithms that can understand and identify semantically relevant information within videos (Wu et al., 2019; 2017). Our research focuses on the video activity localization domain (Gao et al., 2017: Zhao et al., 2017), which enables users to identify, classify, and retrieve interesting video moments. Video activity localization tasks are defined to predict a set of temporal spans relevant to either a fixed class taxonomy (Caba Heil- bron et al., 2015) free-form natural language queries (He al., 2019). These algorithms have numerous applications, including action segmentation (Wang et al., 2020), highlight generation (Lei al., 20)1 product placement (Jiao et al., 2021), and video editing (Neimark et al., 2021).
Technical solutions for activity localization often draw inspiration from innovations in object de- tection, which is well-studied problem in the image domain. An analogy can, in fact, be drawn between spatially localizing objects and temporally localizing moments, with the key difference be- ing that temporal boundaries subjective and open interpretation (Wang al., 2020; Alwassel et al.. 2018). A prevailing approach for model design in object detection is to adopt an encoder- decoder design paired with a suitable training protocol (Carion et al., 2020; Lei et al.. 2021b). For video activity localization, the encoder processes the raw video frames and, optionally, a language query, with the goal of generating rich semantic representations that capture the interaction within the video and between video and language. These representations are referred to as "memory". The decoder leverages the encoder's memory to produce a list of temporal locations with corresponding Denotes equal contribution.

---

Published as a conference paper at ICLR 2024
confidence scores. The decoder achieves this by inputting candidate spans, which can be predefined based on ground truth activity locations statistics or learned during training.
The primary challenge in video localization tasks stems from the boundary ambiguity mentioned carlier. Unlike object boundaries, activities are continuous in time, and the saliency of a temporal event changes smoothly due to its non-zero momentum. Thus, transitions between activities not always clear or intuitive. Moreover, human perception of action boundaries is instinctive and sub- jective. This phenomenon is reflected in the existing video datasets, where we can identify multiple clues indicating that the variance of localization information is higher than for object locations. To support this thesis, DETAD (Alwassel et al.. 2018) conducted a campaign to re-annotate the Activ- ityNet (Caba Heilbron et al., 2015) dataset to estimate the annotator's 's agreement and quantify the severity of boundary uncertainty in humans. Additionally, a recent study (Lei et al., 202 found that over 10% of the queries in QV-Highlights showed disagreement on the boundary location with an Intersection over Union (loU) of 0.9. Thus, addressing the uncertainty of action boundaries is crucial for developing reliable and accurate video localization pipelines.
In this study, we aim to address the challenge of uncertain action boundaries in video activity local- ization. To this end, we propose DenoiseLoc, an encoder-decoder model which introduces novel noundary-denoisingg training paradigm. In DenoiseLoc, the transformer-based encoder captures the relations within and across modalities. The decoder is provided with learnable proposals and noisy ground truth spans and progressively refines them across multiple decoder layers. In detail, iteratively extract location-sensitive features and use them to update the proposal embeddings and spans in each decoder layer. Our boundary-denoise as an augmentation to guide the model on predicting meaningful boundaries under the uncertainty of initial noisy spans. Our denoising training method is easily generalizable to several video do- mains (i.e., YouTube videos, movies, egocentric videos) as it does not require hand-crafted proposal designs, and the inference process is the same as in generic mncoder-decoder design (Lei et al., 202 lb). Surprisingly, we find that with very few proposals (i.e., 30 per video), our model retrieves sufficient high-quality actions and performs on par with or better than computationally expensive methods evaluating thousands of proposals.
training jitters action proposals and serves
To demonstrate our model's effectiveness and generalization ability, we conducted extensive exper- iments on different datasets.
In Fig. 1, we probe our intermediate outputs on the
140
val split of QV-Hightlights (Lei et al., 2021a). The model gradually denoises the boundary to the ground- truth location through multiple layers and reaches 45.29% Average mAP, without a dedicated learning scheme. On the more recent and challenging MAD dataset (Soldan et al., 2022) for activity localization in long-form videos, we achieve significant improve- ments of 2.69% on Recall@ 1.
120-
end end time
100
+
GT Action layer4->layer8 layer2->layer4
80
+
60
layer1->layer2
Our contributions are: (1) A novel boundary-denoisin training approach for video activity localization tasks. The noise injected at training time leads to faster con- vergence with a much smaller number of queries. (2) DenoiseLoc, an encoder-decoder style multi-modality localization model, specifically designed to exploit the boundary-denoising training strategy. (3) Ex- tensive experiments to demonstrate the effectiveness of DenoiseLoc, achieving sfate-of-the-art performance on several datasets, including the large-scale MAD dataset. Additionally, we provide thorough ablation and analysis studies to investigate the design choices and modeling properties of our approach.
20
40
60 80 100 120 140 start time
Figure 1: Illustration of the boundary denoising process. Arrows show the pre- dicted noise from each of our decoder lay- ers, visualized on the st tttart-es red bound- ary map. The Average mAP of layerl, layer2. 1ayer4, and layer8 out- puts are increasing as 38.48%, 42.07%, 44.57%, and 45.29% evaluated on the val split of QV-Hightlights (Lei et al., 2021a).
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Video Activity Localization. Video Activity Localization is the task of predicting the temporal span in video corresponding to a numan-generatee natural language description. Many solutions have been proposed for this task; however, methods can be clustered into two groups: (i) proposal-basee
methods (Jiyang et al., 2017: Anne Hendricks et al., 2017; Soldan et al., 2021; Songyang et al., 2020; Escorcia et al., 2019; Xia et al., 2022), which produce confidence or alignment scores for predefined set of M temporal moments and (ii) proposal-free (Lei et al., 202 bb; Zeng et al., 2020; Mun et al., 2020; Shaoxiang & Yu-Gang, 2020; Rodriguez et al., 2020; Li et al., 2021), which di- rectly regress the temporal interval boundaries for a given vidco-query pair. Proposal proposal-based methods tend to outperform regression perression-based methods, but the latter have much faster inference times since they don't t require exhaustive matching to a set of proposals. Our proposed DenoiseLoe follows the proposal-free pipeline, and it is evaluated our model both short-form (QV-Highlights (Lei al., 2021a) and long-form (MAD (Soldan et al., 2022)) grounding datasets.
Denoising and Diffusion. The issue of boundary noise has been extensively studied in the field of weakly supervised temporal action localization, where temporal annotation information is un- available. To mitigate noise in generated pseudo labels, Yang. al. (Yang al., 2021: 2022) designed an uncertainty-aware learning module UGCT, while Li et al. et al., 2022b) proposed novel Denoised Cross-video Contrastive (DCC) algorithm reduce the negative impacts of noisy contrastive features. In fully supervised localization, Huang et al. (Huang et al., 2022) introduces Elastic Moment Bounding (EMB) to accommodate flexible and adaptive activity temporal bound- aries with tolerance to underlying temporal uncertainties. However, these studies mainly focus on noise modeling rather than denoising. Particularly, EMB predicts the action location and uncertainty and proposed new evaluation metric to measure the range of predicted boundaries. In contrast, our method directly predicts the denoised span and follows the standard evaluation.
More recently, studies detection have explored learning with noise. DN-DETR (Li et al., 2022a) discovers that including noisy object proposals can accelerate the training of transformer- de- tectors such DETR (Carion et al., 2020). DINO (Zhang et al., 2022) proposes to add noise to ground truth at different levels to generate positive and negative samples for training. Meanwhile, DiffusionDet (Chen et al., 2022), DifTTAD (Nag et al., 2023), and Diffusion (Zhao al., 2023) propose scheduled iterative denoising method to train model to refine the input proposals with dif- ferent noise levels. Although our work inspired by, or similar to these methods, we show diffusion process is not evidentally better than a single denoising step, as the detection output is in a way lower dimension manifold than generated images.
METHOD
3.1 PROBLEM FORMULATION
We aim to localize temporal instances that are relevant to pre-defined activity list natural lan- guage query. The input video is modeled as a sequence of n, snippets, V=(u; nt each comprising F consecutive frames. If available, the language query is tokenized in ny elements [.-(1.] )i Both inputs are mapped to feature vectors using pre-trained models and are identified as X. Xxxi, respectively. where o and q are the feature dimension of snippets and tokens.
To evaluate our model, we compare its predictions & with human annotations V. The model pre- diets M possible instances, denoted as (= ((")""""))))))))"""""""))")))))) M sorted by their predicted relevance. Each instance -f" (2)) represents the beginning (f:) and ending (f") times of the n-th relevant in- stance, and 8" is its corresponding confidence score. In this work, we regard coupled temporal endpoints (La, (n) as a "temporal span".
3.2 DENOISELOC ARCHITECTURE
As illustrated in Fig. 2, our model pipeline includes three main designs: encoder, decoder, and denoising training. First, feed the video features X, into the encoder list of computation blocks. The query embeddings X, are concatenated with the video feature after a projection layer when it is available. We choose multi-head attention our basic block unit due to its ability for
3

---

Published as a conference paper at ICLR 2024
Feature
Tranformer Encoder
xNac
DN DJ
DN
MLP
MLP
Positional
embedding
Projection
Projection
Decoder
XNoc
Learable Reature
[N
DI
Text Uncoder
Video
Votee
Temporal
Encoder
[2N D]
Noise Injection
[N
span
"A litll boy holding yellow bell walks by
X.
Learnable embedddings
SoT Ro7 RoT
Leansble
Ground truth spon
spans
Demoising Training
Figure 2: Architecture. Our proposed model consists of three main components: Encoder, De- coder, and Denoising Training. The input video and text are transformed into video and text em- beddings by means of pretrained networks, then concatenated and fed to the Encoder module (left). The output features from the Encoder, termed as memory, are subsequently routed to the Decoder. During training, the Decoder receives in input set of learnable embeddings, ground truth spans with additional artificial noise, and set of learnable spans, The objective the decoder is to en- rich instance-leve level features computed based the two sets of spans and refine (via denoising) the provided ground truth spans. Note that the GT spans are not used during inference.
long-range dependency modeling. Then, the output of the encoder (i.e., memory) is forwarded to a decoder, along with a set of learnable spans. The decoder is a stack of units that apply instance feature extraction and instance-instad can be directly used to obtain effective proposal features. The boundary-denoisee includes new set of noisy spans train the decoder, and it aims to help the decoder converge faster and better. We disable those extra spans during inference as the ground truth spans are not available anymore. Also, we do not need NMS because a loss was applied in training.
interaction, where we show that the boundary information
training process
3.2.1 ENCODER
Our encoder aims at modeling semantic information of the inputs, such as multi-modalipy interaction and non-local temporal dependency. It includes the feature projection layer(s) followed by Nene transformer encoder layers. Given the video snippet features X,,, and optionally the language token features X,, the feature project layer transforms its input into joint space. Then, we concatenate all the available features added with the sinusoidal positional embeddings to the video features and send them to the transformer encoder layers. Since the self-attention mechanism has the potential have message- -aaigggggagnnggggge over any two tokens, we assume the encoder output has rich semantic information over temporal relations and is conditioned by the given text prompt for video activity localization. Following the convention in DETR, we name this intermediate output memory.
3.2.2 DECODER
We design Naee denoising layers the decoder. Each of them works as a basic unit predict the noise of the input spans from the video feature. Specifically, each denoising layer takes the proposal start/end and corresponding proposal embeddings inputs and progressively refines the proposal boundaries, and classifies their categories. Such N₂ proposals serve as queries and interact with global video representation (memory) the decoder, which is introduced DETR. To avoid tedious hand-crafted designs such pre-defined anchors, the proposal start/end and proposal embeddings are set as learnable parameters and can be updated through backpropagation.
As illustrated in Fig. 3, in each decoder layer, the module follows the order of the self-attention layer, cross-attention layer, and feed-forward layer. To be specific, the self-attention layer is adopted on the proposal embeddings to model the proposal relations with each other. Next, the cross-altention layer simulates the interactions between the proposal embeddings with the encoder's output mem- ory, thereby the proposal embeddings can capture the rich semantic information from the video and refine the proposal boundaries. Different from DETR, we find that explicitly modeling proposal features is crucial for proposal embedding interaction. Specifically, we replace the standard cross-
4

---

Published as a conference paper at ICLR 2024
attention layer in the transformer decoder with the DynamicConv (He et al., 2017) layer, which is proposed in (Sun et al., 2021). This choice is ablated and validated Sec. 4.5, where we in investi- gate different architectural designs. As illustrated in Fig. temporal Rol alignment (AlignID) (Xu et al., 2020) is first adopted to extract the proposal features based on the start/end locations, then the proposal features and proposal embeddings are sent into the DynamicCony module accom- plish the interactions between implicit proposal embeddings with explicit proposal features, The DynamicConv layer oulputs the enhanced proposal features and the updated proposal embeddings. Subsequently, MLPs are used to predict the updated proposal start/end locations.
After each decoder layer, the proposal boundaries will be refined with predicted start/end offset. Together with the updated proposal embeddings, the refined proposals will be sent into the next decoder layer to regress more precise boundaries and more accurate classification probabilities.
3.2.3 BOUNDARY-
BOUNDARY-DENOISING DENOISING TRAINING
L
The Noundary-denoising training module in- cludes set of noisy spans to train the de- coder. Therefore, the input of the decoder is the union of the learnable spans and the new noisy spans. Differently, the new noisy spans in boundary-denoising training sampled around the ground truth activity loca- tion. These sampled spans able diversify the temporal inputs and accelerate model con- vergence. However, since the activity location is unknown in the test set, we only use learnable spans during model inference.
MLP
MLP
Decoder Layer
Dynamic Convolution
0300
Lls
randomly
SA
Align ID
Memory
Leamable embeddings
GT noisy Learnable
To be more specific, during training, given a set of ground-truth temporal activities, ((eg (2) Mag ) and a noise level we first sam-
spans
spans
Figure 3: Decoder. In each decoder layer, the self-attention is first adopted on the proposal em- beddings to model the proposal relations with each other. Then the proposal features are explic- itly extracted by Rol alignment, and they further interact with proposal embeddings to enrich the proposal representation through Dynamic Convo- lution. Last, the feed-forward MLPs update pro- posal embeddings and start/end gradually.
ple Nqn 2D vectors from Gaussian NN0,
Ngn denoted as {En J En E R². Then, we di- vide the Ngn noise vectors into two different groups. The first group of Mgt combines with ground truth temporal spans create positive set of proposals, e.g., one noisy proposal could The second group including Nyn Mas elements works as negative set of proposals so that we
be (L + cf.6 + ),)),, n E (1,.... Mge). (1.....M,,)
directly use (fheg of, peeg + <g), E (Mg + 1,..,, NNn) to make most of the proposals valid (i.e., inside of the video). Note that when there a boundary outside of (0, 1) for a proposal, will clamp it into the valid range. We have thep 0.25, gneg 0.75 so that the average center and width of those proposals are half of the video duration.
Once the noisy proposals are created, we feed them into the decoder to reduce the noise in each decoder layer. Note that we still have the learned proposal embedding and spans in the decoder, but the noisy proposal does not communicate with the learned proposal in case of annotation leakage. Morcover, the noisy proposals have their own assignment to the training target, and will not affect the matching algorithm applied in the learnable spans in the decoder. In inference, our predictions are from the learned proposal position, and the decoder has obtained the capability to denoise the learned proposal in each layer.
3.3 LOSS FUNCTIONS
We denote @ 23) ) as the set of predictions, and V= W2t as the set of ground-truth temporal activities, where " fn) and = tz). To compute the loss, we need to first find assignment between predictions and ground truth. We determine the assignment based on the
5

---

Published as a conference paper at ICLR 2024
matching cost match similar to DETR (Carion et al., 2020)
Mge
mmtech (V,, V)
Σ (m(m) 1=
+ Lspan (87(m) e.
(I)
With this matching cost, following previous works, we use the Hungarian algorithm (Kuhn, 1955) to find the optimal bipartite matching म. where == arg min, Lmatch- Based on this assignment f., our overall loss is defined as:
L Aooooooo + AoteCols + Asatiency zalienccy
where Xtae Acle- Asaliency R are hapareparameters balancing the three terms. Lioe combines an LI loss and generalized IoU loss (ID version as (Rezatofighi al., 2019)) between the prediction and ground-truth truth to measure the localization quality:
Lioc >r, Ci (1)7()) b²) + >gtot tot (1*(M)) 16").
and Lets is the cross entropy function to measure whether the action category is correctly classified or whether the prediction matches the query correctly. Leatiency directly borrows from Moment- DETR (Lei et al., 2021b) for query-dependent highlights detection. Hence, Asalieney is set to 0 for benchmarks without highlight detection.
EXPERIMENTS
This section presents our experimental setup, showcasing our pipeline's excellent performance. We first introduce datasets and metrics before reporting the comparison with the current state-of-the-ar methods. We conclude the chapter with our thorough ablation study, which validates all our design choices, including noise level, decoder design, and query number.
4.1 DATASETS
Unless otherwise stated, we adopt the original datasets' data splits and benchmark on the test set.
MAD (Soldan al., 2022). This recently released dataset comprises 384K natural language queries (train 280,183, validation 32,064, test 72,044) temporally grounded in 650 full-length movies for a total of over 1.2K hours of video, making it the largest dataset collected for the video language grounding task. Notably, it is the only dataset that allows the investigation of long-form grounding, thanks to the long videos contains.
QViiglighhts (Lei et al., 2021a). This is the only trimmed video dataset for the grounding task, constituted by 10,148 short videos with duration of 150s. Notably, this dataset is characterized by multiple moments associated with each query yielding total of 18,367 annotated moments and 10,310 queries (train 7,218, validation 1,550, test 1,542).
4.2 METRICS
Recall. Our metric of choice for the grounding task is Recall K for loU=0. K 1 if not specified.
Mean Average Precision (mAP). Following the literature, we compute the mAP metric with loU thresholds {0.5, 0.73. We also report the Average of mAP over multiple IoUs, denoted as AmAP.
4.3 IMPLEMENTATION DETAILS
Our training setting follows moment-DETR (Lei et al., 2021a). The input temporal spans get pro- cessed in the decoder blocks to predict their binary classes and the accurate boundaries. All the spans are used to train the binary classifier, but only the matched ones are applied to train the bound- ary predictor. Our algorithm compiled and tested using Python 3.8, PyTorch 1.13, and CUDA 11.6. Notably, have increased the number of encoder and decoder layers 8, beyond which observed saturation. We also use fixed number of 30 queries (proposals) during both training and inference. To train model, the AdamW (Loshchilov & Hutter, 2019) optimizer with

---

Published as a conference paper at ICLR 2024
Table 1: Benchmarking of grounding methods on the MAD dataset. We follow the methodology
presented in (Barrios et al., 2023) and adopt a two-stage approach. In our work, we re-use the first stage implemented in (Barrios al., 2023) and only work on the grounding (second stage) method. We report recall performance for baselines with (f) and without the first-stage model.
IoU=0.1
IoU-0.3
IaU=0.5
Model
R@1
R@5
10
R R00
R@55
R@
R
R@5
R@800
Zero-shor CLIP (Soldan et ul., 2022)
6.37
15.05
20.26
37.92
3.13
9.85
14.13
28.71
1.39
5.44
8.38
18.80
VI.G-Net (Soldan et al., 2021)
3.50
11.74
18.32
38.41
2.63
9.49
15.20
33.68
1.61
6.28
10.18
25.33
Moment-DETR (Lei 2021b)
0.31
1.52
2.79
11.08
0.24
1.14
2.06
7.97
0.16
0.68
1.20
4.71
DenoiseLoc (owrs)
1,06
4.07
6.75
20.07
0.86
3.31
5.41
15.67
0.57
2.17
3.50
9.73
1Zero-shot CLIP (Soldan et al., 2022)
9.30
18.96
24.30
39.79
4.65
13.06
17.73
32.23
2.16
7.40
11.09
23.21
fVLG-Net (Soldan et al., 2021)
5.60
16.07
23.61
15.35
4.28
13.14
19.86
39.77
2.18
8.78
13.72
30.22
iMoment- (lei et 2021b)
5.07
16.30
24.79
50.06
3.82
12.60
19.43
40.52
2.39
7.90
12.06
24.87
CONE (Hou et al., 2022)
8.90
20.51
27
20
43.36
6.87
16.
21.53
34.73
4.10
9.59
12.82
20.56
iDenoiseLoe (ours)
11.59
30.35
41
44
66.07
9.08
23.33
31.57
19.90
6.63
14.03
18.69
29.12
learning rate of le - 4 and weight decay of le 4. We train the model for 200 epochs and select the checkpoint with the best validation set performance for ablation. For evaluation, we compare our model with the state-of-the-arr (SOTA) on the test split. We do not use large-scale pre-training as in (Liu et al., 2022), but we train our models from random initialization. For detailed experimental settings each dataset, please refer to the Appendix. To extract video and text features, follow the implementations described in the existing literature. Specifically, we use the MAD feature from the official release in (Soldan et al., 2022), the QVHighlights feature from (Lei et al., 3021a), and the language feature from Glo (Pennington et al., 2014), as described in (Soldan et al., 2021).
4.4 COMPARISON WITH STATE-OF -HE
Comparative studies are reported in this section. The best result is highlighted in bold in each table, while the rumner-up is underlined.
MAD. Tab. I summarizes the performance of several baselines in the MAD dataset. For this eval- uation, we follow the two-stage methodology introduced in (Barrios et al., 2023) and combine our DenoiseLoc with a Guidance Model. This model serves as a first-stage method that conditions the scores of the second-stage grounding methods to boost their performance. We report the perfor- mance of the baselines with and without the Guidance Model, highlighting the former with the t symbol. Note that CONE (Hou et al., 2022) does not use the same first stage, yet it implements the same concept with different details.
First, let's analyze the second stage's performance (rows 1-4). We can observe that proposal-based methods such as Zero-Shot CLIP and VLG-Net (Soldan et al.. 2021) offer much stronger perfor- mance with respect the proposal- LoiseLoc. This observation is congruent with many other findings in the literature, even beyond the localization tasks video. In this case, the strongest challenge constituted by the long-form na- ture of MAD, which naturally yields many false positive predictions. As it turns out, proposal-free methods are particularly susceptible to this phenomenon. Nonetheless, the combination of guidance and grounding models is able to significantly boost these methods' recall by removing false-positee predictions. In fact, our DenoiseLoc obtains the highest metric for most configurations, with large margins (between 9.6% and 34.4% relative improvements) against the runner-up methods.
approaches Moment-DETR (Lei et al., 2021b) and De-
QVHighlights. We find that our method's good behavior translates to other datasets with very different characteristics, as in QVHighlights (Lei et al., 2021a). Recall and mAP performance presented in Tab. 2, where DenoiseLoc obtains the best performance for all metrics with relative improvements ranging from 3.4% and 14.0%. Note, that we strive for fair comparisons; therefore, we do not report Moment-DETR DETR and UMT performance when pretraining is used. Also, our method surpasses the concurrent work RiffusionVMR (Zhao et al., 2023), and we only run one denoising step. Additional details can be found in (Liu et al., 2022).
4.5 ABLATIONS AND ANALYSTS
We design several ablations and analysis studies to probe our method and devise meaningful take- aways that can help in identifying fruitful future directions.
7

---

Published as a conference paper at ICLR 2024
Table 2: Benchmarking of grounding methods on the test split of QVHighlights dataset. Note that our method surpasses the concurrent work Diffusion VMR (Zhao et al., 2023), and we do only one denoising step.
R@1
mAP
Model
@0.5
@0.7
@0.5
@0.75
Avg.
MCN (Hendricks et al. 2017)
11.41
2.72
24.94
8.22
10.67
CAL (Escorcia al., 2019)
25.49
11.54
23.40
7.65
9.89
XML (Lei et al., 2020)
41.83
30.35
44.63
31.73
32.14
XML+ (Lei et al., 2021a)
46.69
33.46
47.89
34.67
34.90
Moment-DETT (Lei ct al., 2021b)
52.89
33.02
54.82
29.40
30.73
UMT (Liu et al., 2022)
56.23
11.18
53.83
37.01
36.12
DiffusionVMRR (Zhao al., 2023)
61.61
11.49
61.55
10.17
39.92
QD-DETR (Moon et al., 2023)
62.40
14.98
62.52
39.88
39.86
DenoiseLoc (ours)
59.27
45.07
61.30
43.07
42.96
Table 3: Ablation study on the number of queries. When we have fewer queries, adding noise
helps the model quickly converge to a stable state, resulting in improved performance. However, increasing the query number will slow down the convergence rate because of the increased negatives.
Query Number
Denoise Training
Epoch Number
10
40
12,60
26.30
36.44
39.65
41.46
8
11.84
30.62
39.73
41.59
42.18
4.92
26.80
38.35
41.82
42.22
16
12.26
31.45
39.34
42.52
43.78
7,91
25.37
39.07
41.38
42.27
32
4.16
22.36
38.42
43.98
44.11
Table 4: Ablation study on noise ratio. A small noise leads the model underfit, while large noise reduces the prediction variance and performance. Note that dB (decibel) is expressed in loga- ritmic scale. NoiseRatio.as = 10 log
Pams P is the average of the squared values.
P Pazaal
Noise Ratio (dB)
(-20,-10)
(-10,0]
(0,10)
(10,20)
Avg. mAP (avg)
43.71
44.66
44.22
42.89
Avg. MAP (std)
1.64
0.86
0.56
0.40
Noise level. Tab. investigates the impact of varying noise levels (measured by noise-to-signal ratio). Specifically, we gradually increased the noise level from -20dB to 20dB to assess the model's performance. Note, dB refers to decibel. Our findings suggest that the model's performance follows parabolic pattern as the noise level increases, with the best results achieved 44.66% mAP in the (-10, 0] range. Additionally, it also implies that a noise level that is too small will risk the model to overfit, resulting in unstable performance and higher prediction variance. Conversely, a noise level that is too large will lead to underfitting, reducing prediction variance but causing a corresponding decrease model accuracy.
Decoder design. Tab. 5 illustrates the effect of various decoder designs with and without denoising training. Several observations are concluded: 1. Our model, built upon moment-DETR, assesses different ways to combine proposal and query features in Rows and 2. The first row shows that using the average of the feature to represent the action instance instead of Dynamic Conv. also improves performance. 2. Row 3 and Row 4 show when self-attention is applied, significant im- provements are observed, which suggests the proposal-proposse interaction plays an important role in global message passing. 3. Compared with the performances without denoising training, nearly all the metries improved once the noisy temporal spans participated model training. The improve- ments over non-self-attention models (Row 1, 2) are more significant, although we don't introduce any extra computation the decoder. We also find the best model performance is achieved by en- abling all the modules, especially Dynamic Cony. This is because the Dynamic Conv. operation is more sensitive to the temporal feature of the proposals, and our denoising training, by jittering the proposal boundaries, can bring more augmentations in the target data domain.
8

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study on our model architecture on the val split of the VQ-highlight dataset.
When self-attention is applied, we observe significant improvements, which suggests the proposal- proposal interaction plays an important role in global message passing. Also, nearly all the metrics improved once the noisy temporal spans participated in model training.
Dynamic Conv.
Self Atl
w.o.
Denoise Training
w: Denoise Training
IoU@0.5
I.U700.
AmAP
IoU@0.5
1oU@00.
AmAP
X
x
54.1
39.0
32.3
57.2
44.1
38.5
V
X
16.1
31.2
28.6
55.9
13.1
11.6
57.2
45.0
41.1
40.8
58.1
44.7
43.0
59.9
45.9
44.6
Query number. Tab. presents the performance comparison models trained on different epochs with varying numbers of queries. The first two blocks suggest that after adding noise, the model can quickly converge to a stable state, resulting in improved performance. This observation is consistent with the results shown in Fig. 4. Increasing the number of queries from 16 and then to 32, adding noise at the beginning helps the model to converge faster and perform better. However, as the number of queries exceeds a certain threshold, the convergence speed slows down, and performance deteriorates. This may be due to the increasing number of negative samples resulting from one-to- one assignments performed by the Hungarian matcher, which makes it harder for the model to learn how to denoise and causes decreased performance. This finding also highlights that our model can achieve excellent performance with small number of predictions, rendering NMS unnecessary.
4.6 FURTHER DISCUSSION
Table 6: Adapting our denoising model to diffusion framework. No benefit is found when transforming our denoising approach to diffusion one.
Steps
2
4
8
64
512
Avg. mAP
44.26
43.85
43.04
43.61
43.57
42.99
mAP@0.5
62.08
61.08
60.21
60.89
60.66
59.71
m.P@00.7
44.96
43.65
43.04
44.74
44.35
43.20
DenoiseLocr.s. Diffusiooif. Our model can be viewed as a conditional diffusion model when time embedding (parameter governing the diffusion process) is also included in the dynamic volutional layer. In this case, the input of our model is a temporal span of a certain level of noise, and the output the offset of the proposal the target activity localization. As an extension of Tab. 4, we design a DDPM (Ho et al., 2020)-like training protocol (Chen et al., 2022) to optimize our model, dubbed as DiffusionLoe. Specifically, in each training iteration, we random sample an integer t E {1, T), and decide the noise level a based on it. Then the following process is the same as DenoiseL.oc oc, we denoise the ground truth action location to train the network and prediction the denoised proposal in model inference. We investigate 7 E (1, 2, 4, 8, 64, 512) and report the model performance Tab. 6. More discussion can be found in Appendix E.
Surprisingly, we don't observe any evident performance gain of DiffusionLoc from DenoiseLoc, which achieves 44.66 + 0.86 on the same validation split. One potential reason is that knowing the noise level does not help the model to localize activities more precisely, and denoising training is already an effective method in temporal activity localization. This motivates why our method surpasses the concurrent work Diffusion VMR (Zhao et al., 2023) in one single denoising step.
CONCLUSION
We propose DenoiseLocc, an encoder-decoder model, which introduces a novel boundary-denoising training paradigm to address the challenge of uncertain action boundaries in video activity local- ization. DenoiseLoc captures the relations within and across modalities in the encoder and pro- gressively refines learnable proposals and noisy ground truth spans in multiple decoder layers. Our Soundary-denoistee training jitters action proposals and serves as an augmentation to guide the model on predicting meaningful boundaries under the uncertainty of initial noisy spans. Extensive experiments to demonstrate the effectiveness of DenoiseLoc, achieving state-of the-art art performance on several datasets, including the large-scale MAD dataset.

---

Published as a conference paper at ICLR 2024
REFERENCES
Humam Alwassel, Fabian Caba Heilbron, Victor Escorcia, and Bernard Ghanem. Diagnosing error in temporal action detectors. In Proceedings of the European conference on computer vision (ECCV), pp. 256-272, 2018.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing Moments in Video With Natural Language. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Wayner Barrios, Mattia Soldan, Fabian Caba Heilbron, Alberto Mario Ceballos-Arroyo, and Bernard Ghanem. Localizing moments long video via multimodal guidance. arXiv preprint 23311:23022.3332, 2023.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.0978% 2022.
Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan C. Russell. Temporal localization of moments video collections with natural language. CoRR, abs/1907.127633, 2019. URL wwt.:http//.. / rxiv, org/ abs 1907. 12763.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via
language query. In Proceedings of the IEEE international conference computer vision, pp.
5267-5275, 2017.
Dongliang He. Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen. Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 8393 8400, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Zheng Shou, and Nan Duan. Cone: An efficient coarse-to-fime alignment framework for long video temporal grounding. arXiv preprint urXiv:2209. 10918, 2022.
Chan, Chong- Wah Ngo,
Jiabo Huang, Hailin Jin, Shaogang Gong, and Yang Liu. Video activity localisation with uncertain- ties in temporal boundary. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXIV, pp. 724-740. Springer, 2022.
Xu Jia. Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. Advances in neural information processing systems, 29, 2016.
Licheng Jiao, Ruohan Zhang, Fang Liu, Shuyuan Yang, Biao Hou, Lingling Li, and Xu Tang. New generation deep learning for video object detection: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2021.
10

---

Published as a conference paper at ICLR 2024
Gao Jiyang, Sun Chen, Yang Zhenheng, and Ram Nevatia. TALL: Temporal Activity Localization via Language Query. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06998 2017.
Harold W Kuhn. The hungarian method for the assignment problem. Naval Research Logisties Quarterly, 2(1-2):83- 97, 1955.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. TVR: Large-Scale Dataset for Video- Subtitle Moment Retrieval. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
Jie Lei, Tamara L Berg. and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Nearal Information Processing Systems, 34:1 1846-11858, 2021a.
Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natu- ral language queries. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 11846- 11858. Curran Associates, Inc., 2021b. URL httptt:///pocceeeeeeeeeeee http///pooeeeenneeeeeeeeeeeessee wtw.ooeeeiinnn.eeriiiee.eeeeeeee neurips.co. paper/2021/Eilee
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13619- 13627, 2022a.
Jingjing Li, Tianyu Yang, Wei Ji, Jue Wang, and Li Cheng. Exploring denoised cross-video contrast for weakkiy-uueervied temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19914,19924, 2022b.
Kun Li. Dan Guo, and Meng Wang. Proposal-free video grounding with contextual pyramid network. Proceedings of the AAAI Conference Artificial Intelligence, 35(3):1902-1910 May 2021. doi: 10.1609/aaai.v355/ 16285. URL hwtptt:/t/////ttt www..arg.a.a.....aaa....... /index.....////// phhp AAAI BAAT/article/viev// 12285.
Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3042-3051, 2022.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: /owporoeeevieeeeeee net /oot//orum?? id= BKg6RiCqY7.
WonJun Moon, Sangeek Hyun, SangUk Park. Dongchan Park, and Jae-Pil Heo. Query-dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. -3022323033, June 2023.
Jonghwan Mun, Minsu Cho, and Bohyung Han. Local-Global Video-Text Interactions for Tempo- ral Grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Sauradip Nag, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, and Tao Xiang. Difftad: Temporal action detection with proposal denoising diffusion. arXiv preprint arXiv:2303.14880 2023.
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3163-3172, 2021.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
11

---

Published as a conference paper at ICLR 2024
Xukan Ran, Haolianz Chen, Xiaodan Zhu, Zhenming Liu, and Jiasi Chen. Deepdecision: A mobile deep learning framework for edge video analytics. In IEEE INFOCOM 2018-IEEE conference computer communications. pp. 1421-1429. IEEE, 2018
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 658 666,, 2019.
Cristian Rodriguez, Edison Marrese- Taylor, Fatemeh Sadat Saleh, Hongdong Li, and Stephen Gould. Rodriguez, Cristian and Marrese- Marrese-Taylor, Edison and Salch, Fatemeh Sadat and Li, Hong- dong and Gould, Stephen. In Proceedings of the IEEE Winter Conference Applications of Computer Vision (WACV), 2020.
Chen Shaoxiang and Jiang Yu-Gang. Hierarchical Visual-Textual Graph for Temporal Activity Localization via Language. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem. Vig-net: Video- language graph matching network for video grounding. In Proceedings of the IEEE/CVF Inter- national Conference Computer Vision, pp.
2021.
Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5026-5035, June 2022.
Zhang Songyang, Peng Houwen, Fu Jianlong, and Luo Jiebo. Learning 2D Temporal Adjacent Net- works for Moment Localization with Natural Language. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 144 14454-14463, 2021.
Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu. Relaxed transformer decoders for direct action proposal generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13526-13535. October 2021.
Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part 1 16, pp. 382-298. Springer, 2020.
Zhenzhi Wang. Ziteng Gao. Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary-awart cascade networks for temporal action segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 51. Springer, 2020.
Chao- Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 284-293, 2019.
Zuxuan Wu, Ting Yao, Yanwei Fu, and Yu-Gang Jiang. Deep learning for video classification and captioning. In Frontiers of multimedia research, pp. 3-29, 2017.
Kun Xia, Le Wang, Samping Zhou, Nanning Zheng, and Wei Tang. Learning to refactor action and co-occurrence features for temporal action localization. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pp. 13884-13893, 2022.
Mengmeng Xu, Chen Zhao, David S. Rojas, Ali Thabet, and Bemard Ghanem. G-tad: Sub-graph localization for temporal action detection. In Proceedings of the IEEE/CVF Conference on Com- pufer Vision and Pattern Recognition (CVPR), June 2020.
12

---

Published as a conference paper at ICLR 2024
Wenfei Yang, Tianzhu Zhang, Xiaoyuan Yu, Tian Qi, Yongdong Zhang, and Feng Wu. Uncertainty guided collaborative training for weakly supervised temporal action detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 53-63, 2021.
Wenfei Yang, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Uncertainty guided collaborative training for weakly supervised and unsupervised temporal action localization. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022.
Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan. Dense Regression Network for Video Grounding. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition (CVPR), 2020.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In The Eleventh International Conference on Learning Representations, 2022.
Henghao Zhao, Kevin Qinghong Lin, Rui Yan, and Zechao Li. Diffusionvvmr: Diffusion model for video moment retrieval, 2023.
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In Proceedings of the IEEE international conference on computer vision, pp. 2914-2923, 2017.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 LIMITATIONS AND ETHICS CONCERNS
Limitations: common limitation of deep learning-based computer vision solutions for video understanding is the potential for overfitting. This is driven by the adoption of very large models and sometimes small datasets. this work, we address this concern by evaluating the benefit of our approach across multiple datasets, some of which are of massive scales, like MAD. Yet, our work presents itself with other limitations. In fact, throughout the experimental section, we mentioned how our performance for high recall (i.e., R@5) might not achieve Mate-of-the-art results. We hypothesized that this phenomenon due to the Hungarian matcher that is adopted in this work and several others (Tan et al., 2021). Such algorithm assigns the ground truth to the best prediction, allowing us to compute the loss only for R@ and not optimize for multiple predictions. We believe researchers should foeus on improving this aspect of the current set-prediction approaches. We leave the solution to this limitation to future works.
Ethics Concerns: As the performance of video understanding algorithms gradually improves, such techniques, publicly available for research purposes, might be adopted for illicit surveillance pur- poses, raising concerns about privacy and potential misuse. Our work is released the public in accordance with the limitation of the MIT license. Any misuse of the information provided in this document is to be condemned and apprehended.
A.2 EXPERIMENT SETTINGS
This section presents the detailed experimental setup for each dataset evaluated in the main paper.
MAD (Soldan al., 2022). We used this dataset to evaluate our method under the long-form video setting. To have fair comparison with the literature, we follow coarse-to-fine two-stage method. We only use the DenoiseLoc prediction as a finer prediction in a short window that is predicted by (Barrios et al., 2023). Also, reduce the number of queries from 30 to 14 and do not find any performance drop. In addition, since MAD does not have snippet-wise annotation that could be used saliency loss in Eq. (2), we randomly sample 3 snippets inside the temporal span annotation as positives to help the encoder's convergence.
QVHighlights (Lei et al., 2021a). Our training process follows precisely the description of Sec. 4.3. To receive the test set performance, we submit our prediction to the official server hosted on CodaLab.
A.3 TRAINING EFFICIENCY
DenoiseLoc is an encoder-decoder model that incorporates a temporal span denoising task during training. Fig. 4, we demonstrate that our denoising training yields a fast converging model.
A.4 VISUALIZZATIONS
We visualize the predictions from decoder layers 1, 2, 4, and of video example in Fig. 5. We use colored arrows to denote predicted temporal spans (the darker the color, the higher the model confidence). We also filtered the predictions if the confidence is less than 0.2. As the decoder goes deeper (bottom to up), the proposal boundaries become more precise, and the ranking confidence scores become more accurate. In the meantime, the redundant proposals are also suppressed.
A.5 DENOISE TRAINING ON DECODER (DENOISER)
Training as a denoiser. The proposed denoising training can be viewed in two ways. (1) Augmented training set. Besides the learned proposal span, we generate random spans around the ground-truth location. Those new spans can accelerate the decoder's convergence and make our training more robust. (2) Individual module. Since the generated spans and learned spans do not interact in the transformer decoder layers, they can be studied in two different modules independently, while the two modules share the same model architecture and weights. Therefore, the individual module, e.g. denoiser, works similarly regularizer.
14

---

Published as a conference paper at ICLR 2024
Average mAP (5)
20
Moment-DEII -ETT
10
UMT
DenoiseLoc
20
40
60
80
100
Epoch
Figure Our method has faster convergence and better performance compared to other state-of- the-art methods on the QV-Highlights dataset.
Query: Man wears bandana around his face while out and about.
Os
2s
52s
74s
84s
92s
layer 8
layer 4
layer 2
layer
Ordered prediction
Ground-truth
(blueness correlated confidence)
Figure Qualitative Results on QV-Highlights. A deeper color means the proposal's confidence score is higher. Our method progressively denoises the action boundaries and removes the redun- dancies. See Sec, A.4 for more details. Best viewed in color.
Spans in inference. Although the denoiser module disabled during inference as the ground-truth information is not available anymore, we discover that replacing the learnable spans with random spans also gives competitive performance, setting) stays almost the same, but using random spans gives more advantages under more strict metric loU=0.7 and is also good at middle/short activities, shown in the last two columns. Therefore, we can deduce the learned temporal spans are good at long activities over lenient constraints, while random spans have higher chance of predicting precise action boundaries over short activities.
Tab. 7. The average performance (e.g. AmAP in all
Uncertainty in inference. Following the random span setting in inference, another good property of DenoiseLoc that the uncertainty of model predictions can be measured by generating random spans with different random seeds. To explore this, we change the seed from 0 to 9, run inference, and visualize the predictions on the same video from all runs in Fig. 6 and Fig. 7. The ground spans in red are in the top subplot, and the predictions from ten different seeds are in the bottom. In each subplot of the predictions, the one with higher confidence higher and of darker color. Predictions with confidence scores less than 0.2 removed. The unit the x-axis is per second.
15

---

Published as a conference paper at ICLR 2024
Query: A cop is sitting for interview in uniform.
Video: Gwdizgh16jy 210.0_360.0
ground-truth ground truth
Seed 00
01
02
03
Seed 00
01
02
03
04
0.
seed 2
01
02
03
04
01
02
03
04
05
0.
01
02
03
04
05
01
02
0.
04
05
0.
03
04
05
06
01
02
03
04
seed
04
05
06
07
08
01
02
02
05
0.
20
40
60
80
100
120
140
Figure 6: Visualization with ten random seeds. Almost all subplots have correct predictions, although orders are different. Link.
16

---

Published as a conference paper at ICLR 2024
Query: Man shows the video of police from his phone.
Video: yutpAXhjMII 60.0 210.0
ground-truth ground-truth
1.0
seed 0
01
04
seed_1 1 00
03
0.8
04
seed_2
02
03
seed_
02
02
04
0.6
Seed 4
01
02
Seed 00
01
02
seed
01
02
03
01
02
0.2
seed 00
01
seed_9 00
01
0.0
40
100
120
Figure 7: Visualization with ten difference random seeds. The model is more sensitive to the random seed in this case. Link.
Table 7:
Comparison of different spans during inference. Replacing the learnable spans with random ones also give competitive performance.
Top @0.5
Recall
Average mAP
span
@0.7
all
long
middle
short
learned
60.52
45.23
44.41
52.67
45.10
11.68
random
59.29
46.84
44.51
50.82
45.75
12.10
Fig. visualizes our results to the query "A cop is sitting for an interview uniform." " We observe that almost all subplots have correct top-3 predictions, although their order could differ. Experiment with seed misses a target probability because all the random spans are far away from this location. Thus don't find any more predictions after timestamp 110 (x-axis) for seed 8.
Fig. 7 is more challenging according to our predictions because the model more sensitive to the random seed. We observe both the central location and the duration of the top-1 prediction vary lot over the 10 runs, while seems that only the seed-9 experiment gives a correct answer. However, take closer look at the video, there is camera- camera-change from 2:13-2:15 (equivalent to 73-75 in our x-axis). Therefore, our model predicts as two separate activities, but both are relevant to the query, "Man shows the video of police from his phone."
17

---

Published as a conference paper at ICLR 2024
A.6 SUPPLEMENTARY OUR DIFFUSION VARIANT
DiffusionLoc A time embedding can be incorporated into the dynamic convolutional layer in the decoder. We name this variant our model as DiffusionLoc. To be more concrete, given an integer we project its sinusoidal position embeddings by multi-layer perception to produce scale and shift vectors, denoted as $1 and $2- Given the original output X after the FC layer, our time-conditioned output is ($1 1)X + 82. Note that we have used &i as spans, the decoder can be formulated as
Vpred Dec(y noiser
M Memory, +)
where Memory is the output from the encoder and θ is the learnable parameters of the model.
The training phase of DiffusionLoe is pretty much the same as DenoiseLoc, while the difference is that the noise level is controlled by t, rrandomly sampled in {1,2, . in each iteration and each video, i.e.
(1) Vnoise Voellgt + v1 ārē.
where 1000, the choice of 0e follows DDPM (Ho et al., 2020), and č is the noise term as defined in our DenoiseLoc.
The ideal inference of DiffusionLor is to take from T,T 1, until In each step, we estimate the noisy spans at the noise level at t, and the decoder will predict the activity intervals from the estimated spans. Mathematically, each step can be shown as,
Dec( ('noisse Memory, t)
pred =
notes = Wa (t++) pred +
Oi..E.
In practice, observe the predicted negative spans degenerate quickly and cannot be recovered to a proper random distribution, while the predicted positive spans also have the risk of being negative after the disfasion process. Therefore, it is best for us have only step in inference. However, given a converged DenosieLoc model, we can always progressively boost the model performance by using different sets of initial spans and ensemble the predictions from each run. We would like to gain attention from the community and leave this study and follow-up works from DenoiseLoc.
A.7 COMPUTATIONAL BLOCKS
In this section, we provide additional information and visualizations regarding the Rol alignment (A) (Xu et al., 2020) and DynamicConv (Sun et al., 2021) blocks.
Spans [N. 2J
Rol
Features
AlignID
Features [N
IC, T.I
Time
Spans
[N,2]
[C,
Interpolation
[C, D]
Stack
Rol Fearures
Figure 8: Rol Alignment (1D). See text for details.
18

---

Published as a conference paper at ICLR 2024
Rol Align Figure 8 illustrates the workflow of the Rol Align block in dimension case. This block, termed A or Rol Align ID is borrowed directly from (Xu et al., 2020) and builds on the original Rol Align 2D introduced in (He et al., 2017).
This block accepts in input a set of visual features (shape [C,T,] where the feature channel and 7n is the number of features with In our implementation, the features are the memory from the transformer encoder (see Section 3 and Fig. 3). For the i-th temporal span, first identify those elements of the features that fall within the span's ss temporal endpoints and group them together. Each group undergoes an interpolation operation produce fixed number (D) of features regardless of the input feature number, resulting in new tensor with shape [C, D), as shown in the figure. The outputs from all groups are finally stacked together produce the output feature of dimension [N, C, DJ. The use this block allows us to obtain a fixed dimensional representation feature for each temporal span.
(1....: N), and list of temporal spans (shape [N, 2)).
Proposal Features [N.C]
Updated proposal features
DynamicConv
Rol Features D.
D.
[N,
Matrix multiplication
Norm
ReLU
Matrix multiplication
Norm
ReLU
Flatten
FC
[NCC
Split + Reshapo
[N, C14)
[N,C4.C]
Figure 9: DynamieConv. See text for details.
DynamicCony Figure 9 illustrates the workflow of the DynamiConv block. This block is borrowed directly from (Sun et al., 2021), which was used in the object instance segmentation task.
DynamicConv accepts in input sets of features. In our implementation, feed the Rol features, the output of the Rol Align block, and the proposal features.
DynamicConv is special type Dynamic Filter (Jia al., 2016) and Conditional Operation (Tian et al., 2020) proposed for the detection problem (Sun et al., 2021). It improves Rol features from a given proposal features and outputs an enhanced feature of all the temporal spans. In our implemen- tation, the Rol features are extracted from the temporal spans and memory by Align Besides, the proposal features are originally from learnable embeddings the first decoder block and from the output of previous DynamicConv layers in the following decoder blocks.
To be more specific, DynamicConv projects the proposal feature (NN, CT) into two parameter groups with increased dimensionality, [N, C, C//4], [N,C//4CC, Then, the Rol feature shaped in [N, D, C] multiplied by them sequentially with non-linear functions to gain instance-level in- formation. The result shaped in [N, D, C'] is then flattened and projected as new [N, C*] feature. This feature used for boundary denoising and improving the Rol feature of the next decoder block.
19


Reviews:
[{"review_id": "T1qRy9Ldsf", "reviewer": "Reviewer_UMQX", "summary": "This paper tackles the problem of video activity localization, specifically given language descriptions. The main challenge of this task is boundary ambiguity caused by the annotator subjectivity and the smoothness of temporal events. To this end, the authors design a novel framework, named denoiseloc, aiming to progressively refine the moment predictions. To facilitate the model training, boundary-denoising training scheme is adopted, which encourages the decoder to reconstruct ground truths from the noisy moments. In the experiments on two benchmarks, MAD and QVHighlights, the effectiveness of the proposed method is validated.", "questions": "Please refer to the Weakness section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "+ The paper is well-written and easy to follow with good readability.\n+ The figures well represent the proposed method, helping the understanding.\n+ The proposed approach surpasses the strong competitors on both benchmarks.\n+ The comparison between Denoiseloc and Diffuseloc is interesting, and brings valuable insights.", "weaknesses": "- Some important details of the method are missing. In its current form, the information about the model is insufficient in the manuscript.\n\n(1) DETR-like approaches conventionally adopt the moment representation of (center, width). In contrast, the authors stated that they express a moment as start and end. In this case, the start position can be predicted as a larger value than the end position. How do the authors handle this?\n\n(2) The details of temporal ROI alignment and DynamicConv layer are missing. I would like to suggest the authors to include graphical illustrations of these operations at least in the appendix for help the understanding.\n\n(3) In the boundary-denoising process, the model generates two types of noise-injected proposals, i.e., positive and negative sets. To my understanding, the proposals in the positive set have their corresponding ground truths by design, so the model learns to recover the ground-truth moments from them. However, there is a lack of explanations about the role of the proposals in the negative set. Are they also used to recover ground truths? Or do they serve as negative samples for classification? If the former is the case, how is the matching performed? In addition, what happens if they overlap with ground truths? Will it disturb the training?\n\n- The comparisons with existing DETR-approaches seem not fair. To my knowledge, the DETR-based approaches (e.g., Moment-DETR and UMT) leverage four encoder layers and two decoder layers with a total of 10 queries on QVHighlights. On the other hand, the proposed architecture utilizes (at most) four times more encoder/decoder layers than those of the competitors, and three times more moment queries than those of the competitors. This makes it unclear whether the performance gains come from increased parameters or the proposed algorithm, and it is highly encouraged to perform comparisons under the same setting. In addition, comparisons on the computational cost and the memory consumption will be beneficial. Meanwhile, one of the state-of-the-art method, QD-DETR [1], is missing in the comparison table. If included, the proposed method shows inferior performances even with more layers and more queries.\n\n[1] Moon et al., “Query-Dependent Video Representation for Moment Retrieval and Highlight Detection”, CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JCgigvMzWM", "reviewer": "Reviewer_uvci", "summary": "This paper tackles an important and common challenge of boundary ambiguity in the video action localization task. The authors adopt the encoder-decoder framework as DETR for embedding video/caption features and predicting the boundary locations. The proposed denoiseloc aims at regress more precise boundaries by noise injection. Extensive experiments to demonstrate the effectiveness of denoiseloc.", "questions": "- What is the definition of an action span or temporal span?\n- What do 0.25 and 0.75 mean in the Sec. 3.2.3? Negative proposal set is from the inside or outside of the ground truth?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The inspiration of boundary-denoising training approach has good novelty.\n+ This paper is well-organized and the proposed method achieves good performance.", "weaknesses": "- This paper uses a complex symbol system, which makes it difficult to read. \\epsilon presents the number of fixed consecutive frames and then it presents a vector of a span. n, which represents a quantitative index, is sometimes a subscript and sometimes a superscript.\n- The process of denoising is unclear. Which loss function is used for boundary denoising? The core technology is a proposal augmentation strategy to obtain more candidate proposals for training?\n- Missing related works of boundary ambiguity and temporal grounding.\n\nWang Z, Gao Z, Wang L, et al. Boundary-aware cascade networks for temporal action segmentation[C]. ECCV2020.\n\nXia K, Wang L, Zhou S, et al. Learning to refactor action and co-occurrence features for temporal action localization[C]. CVPR2022.\n- Typo. L5 of Sec. 3.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bgCHdp7bsJ", "reviewer": "Reviewer_6dfr", "summary": "This paper proposed an encoder-decoder model, namely DenoiseLoc, for video activity localization. DenoiseLoc introduces a boundary-denoising paradigm to address the challenge of uncertain action boundaries. DenoiseLoc leverages across modalities in the encoder and progressively refines learnable proposals and noisy ground truth spans in decoder layers. Extensive experiments on standard benchmarks demonstrate the effectiveness of the proposed DenoiseLoc.", "questions": "Please refer to Weaknesses for more details.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- A novel boundary-denoising paradigm is proposed to address the challenge of uncertain action boundaries in video activity localization task.\n- Extensive experiments on standard benchmarks demonstrate the effectiveness of the proposed DenoiseLoc.\n- It is interesting to find that satisfactory performance can be achieved with very few proposals and very few denoising steps.", "weaknesses": "-  Lack of visual analysis. It would be helpful to understand the properties of the proposed method if some cases can be visually analyzed.\n- \"DenoiseLoc\" and \"denoiseloc\" are used interchangeably, which confuses readers. It is recommended that all be changed to \"DenoiseLoc\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LE5TFQUpP6", "reviewer": "Reviewer_61UD", "summary": "The authors propose a method for boundary denoising to tackle video activity localization. A model architecture DenoiseLoc is proposed, together with a boundary denoising training method. The authors argue that a single step denoising is better than the diffusion process with multiple steps. Experiments show some improvement over the previous state-of-the-art.", "questions": "Apart from the above concerns, some other specific questions are below.\n\n\n1)\tCould the authors provide the time gains from using a single denoising step against multiple?\n2)\tCould the authors provide the model size and time gains as compared to previous state-of-the-art methods?\n3)\tIn table 4, when the dB measure for noise is used, what exactly does it mean in this context?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The experimental results show some improvements over the previous state-of-the-art.", "weaknesses": "Most importantly, the method part is not clear. It is rather hard to follow through most of the written parts. Figure 2’s caption also does not provide a clear overview of the proposed method and the novel aspects.\n\n\nIn Figure 2, it is rather confusing what the pipeline actually is. For instance, the ground truth span/noise injection part should definitely not be part of the inference pipeline. So, it is not clear what is done during the inference process.\n\n\nSuddenly, in 3.2.2, the dynamic convolution is used without much explanation. Why is it important to the proposed design? What is the dynamic convolution exactly doing, and why no other design can be used? It is not well motivated.\n\n\nThe boundary denoising training part in 3.2.3 is not clear at all. How the method works, what loss is used, where the loss is applied, and why it is designed this way is not clear. Why do we need to divide into two different groups? How does the model use both of them during training?\n\nImportantly, since boundary denoising has been widely explored, what are the further insights that make the proposed method more effective than previous works? This has not been clearly expressed.\n\n\n\n\nExperimentally, there are also some parts not well established.\n\nMost importantly, it is very strange to me why adding diffusion will lead to performance drops. Furthermore, the more steps used, the worse the performance seems to get. This is totally different from what is usually observed in many diffusion-based works (for generation and for prediction tasks). Usually, the benefit of using a single step is only for efficiency purposes. Furthermore, the given reason is also not convincing. It would be good if the authors provide a lot more details about how diffusion is used, and more qualitative/quantitative evidence to substantiate this claim, since it is quite a strong and counterintuitive claim.\n\n\nIt seems that more ablations are required for various designs, for example the various designs in denoising training. But, currently the method is too unclear for me to suggest concrete settings.\n\n\n\n\n\n\n\nNote that there are some mistakes with the spelling/formatting. This does not affect the score. Some examples are:\n\nPg 2 bottom: citation formats are mixed up, and all the authors of the papers are listed in the citation.\nPg 1 and Pg 2: “QV-Hightlights”\nPg 9: “prediction the denoised proposal”, “more discussion”\n\nThroughout, please standardize as DenoiseLoc (instead of denoiseloc at several parts).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)