MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction
Year: 2023

Abstract:
Given a query, the task of Natural Language Video Localization (NLVL) is to localize a temporal moment in an untrimmed video that semantically matches the query. In this paper, we adopt a proposal-based solution that generates proposals (i.e., candidate moments) and then select the best matching proposal. On top of modeling the cross-modal interaction between candidate moments and the query, our proposed Moment Sampling DETR (MS-DETR) enables efficient moment-moment relation modeling. The core idea is to sample a subset of moments guided by the learnable templates with an adopted DETR (DEtection TRansformer) framework. To achieve this, we design a multiscale visual-linguistic encoder, and an anchorguided moment decoder paired with a set of learnable templates. Experimental results on three public datasets demonstrate the superior performance of MS-DETR.1

heading: 1 Introduction
text: Natural language video localization (NLVL) aims to retrieve a temporal moment from an untrimmed video that semantically corresponds to a given language query, see Fig. 1 for an example. This task is also known as temporal sentence grounding in video, and video moment retrieval. As a fundamental video-language task, it has a wide range of applications, such as video question answering (Fan et al., 2019; Yu et al., 2018; Li et al., 2019), video retrieval (Gabeur et al., 2020; Liu et al., 2019; Chen et al., 2020), and video grounded dialogue (Le et al., 2019; Kim et al., 2021).
Generally speaking, in NLVL models, a video is first split to a sequence of many small fixed-length segments. Video features are then extracted from these segments to interact with the text query. Conceptually, each video segment can be viewed as a
∗Corresponding Authors 1Code is released in https://github.com/K-Nick/MS-DETR
form of “video token”. There are mainly two genres of approaches to NLVL. Proposal-free methods directly model the interaction between video tokens and text, and aim to identify start/end boundaries along the video token sequence. Proposal-based methods generate candidate moments as proposals and then select the best matching proposal2 as the answer. Each proposal is a continuous span of video tokens.
To generate proposals, some methods enumerate all possible moment candidates via pre-defined anchors. Anchors are reference start/end positions along the video. Fig. 2 shows three 2D-Map examples. Each cell in a 2D-Map corresponds to a candidate moment defined by its start/end time along the two axes. Some other methods produce moment candidates with a proposal generator guided by text query and then refine them independently. The interaction between text and video is mainly modeled between text and video moments; each moment is characterized by the video segments that compose it. Very few studies have considered moment-moment interaction. Consequently, it is challenging to discriminate among moments if there are multiple moments that all demonstrate high level of semantic matching with the text query. For instance, the two candidate moments in Fig. 1 have very similar video content and share similar semantic correspondence with the query.
In this paper, we adopt the proposal-based ap2We use the terms proposal and candidate moment interchangeably, or even simply moment when the context is clear.
1387
proach for its capability of cross-modal interaction at both segment level and moment level. We propose MS-DETR to facilitate effective text-moment alignment and efficient moment-moment interaction. For text-moment alignment, we devise a multi-scale vision-language transformer backbone to conduct segment-word and segment-segment interactions at different segment scales. For momentmoment interaction, our main focus is on which moments should be sampled for interaction, due to the large number of possible pairs. Recall that a moment is a span of segments. Let O(N) be the magnitude of segment space; the magnitude of moments is O(N2). Then moment-moment interaction has a space of O(N4).
In practice, not every pair of moments are relevant to each other, and are needed to be discriminated for a given query. Existing methods (Zhang et al., 2020b, 2021b; Wang et al., 2021a) mainly rely on a strong assumption that only the overlapping or adjacent moments are more likely to be relevant, i.e., moment locality. An example of moment locality is shown in Fig. 1, where two adjacent candidate moments share high level of visual similarity. The local interaction strategy is illustrated in Fig. 2, where the reference moment only interacts with the surrounding moments in the 2D-Map. However, not all relevant moments are overlapping or located close to each other. Following the example in Fig. 1, if the person plays saxophone again in the later part of the video (not showing for the sake of space), and the query becomes “He plays saxophone again”, then there will be at least two highly relevant moments for playing saxophone, separated by his action of talking in between. To correctly locate the answer, the model needs to understand that “again” refers to the second moment of playing saxophone. This calls for a better way of sampling moments for efficient moment-moment interaction, to avoid the full global interaction as
shown in Fig. 2. The proposed MS-DETR samples moments for interaction using learnable templates and anchors, illustrated in the third 2D-Map in Fig. 2. We design an anchor-guided moment decoder to interact and aggregate moment features from the encoder in an adaptive and progressive manner. A fixed number of learnable templates paired with dynamic anchors are used to match the moment content and its location. Here, the templates are used to match video content in a moment, and anchors specify the reference start/end positions of the moment because multiple moments may share similar visual features. We then revise the anchors based on the predictions from the last decoder block in an iterative manner. We remark that our method has no assumption on moment locality: the moments can be scattered in diverse locations of the video.
Our key contributions are threefold. First, we propose a novel multi-scale visual-linguistic encoder (Section 4.1) to align textual and video features as well as to aggregate language-enhanced semantics of video frames, in a hierarchical manner. Second, we introduce a new anchor-guided moment decoder (Section 4.2) to decode learnable templates into moment candidates, in which we propose an anchor highlight mechanism to guide the decoding. Third, we conduct extensive experiments (Section 5) on three benchmark datasets: ActivityNet Captions, TACoS, and Charades-STA. Our results demonstrate the effectiveness of the proposed MS-DETR.

heading: 2 Related Work
text: We first briefly review existing NLVL approaches and highlight the differences between our work and other proposal-based solutions. Next, we briefly introduce object detection to provide background for the concept of learnable templates.
Natural Language Video Localization. NLVL was first introduced in Hendricks et al. (2017), and since then a good number of solutions have been proposed (Zhang et al., 2022c). As aforementioned, existing methods can be largely grouped into proposal-based and proposal-free methods. Proposals, or candidate moments, can be either predefined (Gao et al., 2017; Hendricks et al., 2017) or computed by proposal generator (Xiao et al., 2021a,b; Liu et al., 2021a). Proposal-free methods output time span (Zhang et al., 2020a, 2022b, 2021a; Liu et al., 2021b) or timestamps (Yuan et al.,
2019; Ghosh et al., 2019; Li et al., 2021; Zhou et al., 2021) directly on top of video tokens, without considering the notion of candidate moments.
Most proposal-based methods conduct multimodal interaction between video segments and text, then encode moments from the segment features. Typically there is no further interactions among moments. 2D-TAN (Zhang et al., 2020b) is the first to demonstrate the effectiveness of moment-level interaction. However, 2D-TAN assumes moment locality and only enables local interactions among moments as shown in Fig. 2. However, similar moments requiring careful discrimination may be scattered all over the video. This motivates us to go beyond the moment locality assumption and propose moment sampling for interaction, which is a key difference and also a contribution of our work.
In this paper, we adapt the concept of learnable templates from DETR framework to achieve dynamic moment sampling. DETR was originally introduced for object detection in computer vision (CV), to be briefed shortly. Most similar to our work is Xiao et al. (2021a), which also uses learnable templates. However, their work directly adopts learnable templates without any adaption to the specific requirements of NLVL. For instance, the answer moment in NLVL needs to match the given text query, whereas in object detection, there is no such requirement. We bridge the gap between NLVL and object detection by introducing a hierarchical encoder and a decoder with an anchor highlight mechanism. These designs greatly improve performance and unveil the potential of DETR for NLVL. At the same time, these designs also make our model much different from the original DETR.
Transformer-based Object Detection. Object detection is a fundamental CV task. Transformerbased methods now set a new paradigm that uses learnable templates to sparsely localize objects in images. The core idea is to aggregate encoder features globally, by using (randomly initialized) learnable templates. To achieve end-to-end detection, object detection is reformulated as a set prediction problem, e.g., certain template combinations can be used to identify some specific image objects. Early solutions match predictions with ground-truth one by one using bipartite matching, leading to unstable matching and slow convergence. Recent work alleviates this issue by designing many-to-one assignment (Chen et al., 2022; Jia et al., 2022) or the self-supervision task specifically for learnable
templates (Li et al., 2022; Zhang et al., 2022a). Introducing learnable templates to NLVL poses two challenges: supervision sparsity and scale mismatching. An image typically contains multiple objects and these co-occurred objects all serve as detection objects for supervision. In NLVL, given a good number of candidate moments in a video, there is only one ground-truth. We refer to this phenomenon as supervision sparsity. The scale extremity in NLVL is more severe than that in object detection. The ground truth moments in videos, analogous to objects in images, vary from 3% to 90% in terms of video length. The diverse scales bring the issue of scale mismatching when the learned templates are decoded to cover all encoder features, i.e., the entire video. Hence in MS-DETR, we adapt learnable templates mainly for the purpose of sparsely sampling moments for interaction, rather than as the main backbone.

heading: 3 Problem Formulation
text: We first present how to map video and text into features, and then define NLVL in feature space.
Let V = [ft]t=T−1t=0 be an untrimmed video with T frames; L = [wj ] j=M−1 j=0 be a natural language query with M words. We uniformly split the video V into N segments (i.e., video tokens) and employ a pre-trained video feature extractor to encode these segments into visual features V = [vi]i=N−1i=0 . The M words are encoded with pre-trained word embeddings as L = [wj ] j=M−1 j=0 .
Given the video and text query in their encoded features (V,L), the task of NLVL is to localize the timestamp pair (ts, te), the start and end timestamp, of the video moment that matches the query. Note that, due to the uniform split to segments, there is a correspondence between ts and te of the original video and the segment Ids in the segment sequence.

heading: 4 Method
text: The main architecture of the proposed MS-DETR is depicted in Fig. 3. Illustrated in the feature extraction part, given visual features V ∈ Rdv×N and language query features L ∈ Rdw×M , we first project them into a unified dimension d using single layer FFN and decorate them by adding positional encoding, respectively. The linearly projected visual features {v0i }i=N−1i=0 and language query features {w0j}j=M−1j=0 are then concatenated and fed into multi-scale vision-language transformer. Next, we mainly detail two main components: multi-scale
visual-language encoder, and anchor-guided moment decoder.

heading: 4.1 Multi-scale Visual-Language Encoder
text: Many transformer-based methods for cross-modal interaction treat video and language tokens identically, in a unified sequence. However, video and text have completely different syntactic and semantic structures. It is more reasonable to use separate projections for the two modalities, similar to the idea of modality-specific expert Peng et al. (2022). In MS-DETR, we separate the projections by using specifically designed attention modules.
Before we further modify the multi-modal attention modules to handle different video resolutions (i.e., multi-scale), we present our attention designs in their base form. We design two sets of attentions: visual cross-modal attention and linguistic crossmodal attention, see the middle part of Fig. 3. The two sets are highly similar. For conciseness, we only introduce visual cross-modal attention, which contains language to video (L→V), and video to video (V→V) attentions. The visual cross-modal attention aggregates visual embeddings Vl ∈ RN×d and language embeddings Ll ∈ RM×d into new visual features as Vl+1:
Al+1LV = FFN(Vl)FFN(Ll)√
dh (1)
Al+1V V = FFN(Vl)FFN(Vl)√
dh (2)
Al+1 = Al+1LV ⊕Al+1V V (3) Vl+1 = Softmax(Al+1)
× ( FFN(Ll)⊕ FFN(Vl) )
(4)
The linguistic cross-modal attention uses a similar set of equations to model language to language
(L→L) and video to language (V→L) attentions, to get new language features as Ll+1.
Sequence-reduced Multi-modal Attention. Recall that relative lengths of ground truth range from 3% to 90% to their source videos. A fixed resolution for all moments becomes sub-optimal. To this end, we extend the aforementioned multi-modal attention and build a transformer that is capable of providing hierarchical text-enhanced video features, from high to low temporal resolutions. Our encoder design is motivated by the Pyramid Vision Transformer (PVT) (Wang et al., 2021b), which is a successful application of deploying transformer in segmentation problem.
Handling high temporal resolution is a challenge. Directly applying multi-modal attention on high temporal resolution video features suffers from its quadratic complexity as in Eq. 2. Recall that the sequence lengths of key, query, and value in multihead attention (Vaswani et al., 2017) do not have to be the same. Its output has the same length as the query, and the key-value pair keep the same length. Thus, reducing sequence lengths of the key and value simultaneously is an effective way to save computation. Accordingly, we modify V→V attention in the visual cross-modal attention module to a sequence-reduced version as follows:
Vlr = Conv1D(V l)
Al+1V V = FFN(Vl)FFN(Vl)√
dh (5)
Vl+1 = Softmax(Al+1)
× ( FFN(Ll)⊕ FFN(Vlr) )
(6)
Here, Conv1D is a non-overlapping 1D convolution with stride and kernel size set to R. Eq. 2 and Eq. 4 are respectively modified to their new versions in Eq. 5 and Eq. 6. Time complexity is reduced from O(N2) to O(N2R ). We also apply
sequence reduction to V→L attention in the linguistic cross-modal attention. Conceptually, this sequence reduction technique can be explained as decomposing the local and global interaction. The local interaction is achieved by convolution and the global interaction by attention. Next, we focus on how to merge high to low temporal resolutions.
Temporal Merging To form a hierarchical architecture, a crucial step is a pooling-like step to shrink the temporal scale. We utilize an 1D convolution with overlapping to shrink representations from high to low temporal resolutions. The overlapped convolution allows information flow among convolutional windows, so that the interaction is not constrained locally within windows. With both sequence-reduced multi-modal attention and temporal merging, we form a hierarchical architecture. For the deeper layers in the encoder, which already have a low resolution, we turn off these two components and use the vanilla multi-modal attention.
Auxiliary Supervision Losses We design two auxiliary losses: span loss and masked word loss. Span loss is to enhance the language-conditioned video representations from encoder. We use the video features V(Lenc−1) from the last layer of encoder to predict whether each video segment falls within the ground truth. This auxiliary supervision facilitates the model to distinguish relevant video segments from irrelevant ones. We predict span logits Ssp = FFN(VLenc−1) by passing forward encoder output V(Lenc−1) after a two-layer FFN. Span scores Psp are then calculated from Ssp with a sigmoid function. Then the span loss is computed in Eq. 7, where Ysp ∈ {0, 1}.
Lspan = (logPsp ×Ysp)× (Psp −Ysp)2 (7)
Considering ground-truth can be a small portion of the source video, focal loss (Lin et al., 2020) is adopted here to alleviate the label imbalance issue.
The masked word loss aims to better align text features and video features. We dynamically replace 15% words from language query during training with randomly initialized mask embedding. The model is then compelled to learn from both textual and video contexts to recover the missing tokens. Text features W(Lenc−1) from last layer of encoder are used to predict the original words before masking. Masked word score is predicted by Pwm = Softmax(Swm), where
Swm = FFN(W (Lenc−1)). We use the cross entropy loss for masked word prediction.
Lmask = CrossEntropy(Pwm,Ywm) (8)
Multi-scale Text-enhanced Features After Lenc layers of encoder, we select C text-enhanced video features of different scales from intermediate layer outputs. We re-index the selected outputs {Vi0 · · ·ViC−1} into {V0s · · ·VC−1s } for future reference.

heading: 4.2 Anchor-guided Moment Decoder
text: After obtaining the multi-scale text-enhanced video features Vs = {Vcs}c=C−1c=0 , our focus now is to decode the learnable templates with their corresponding anchors into moment timestamps. Recall that templates aim to match moment content and anchors are the reference start/end positions. Initially, the anchors are uniformly distributed along the video to guarantee at least one anchor falls within the range of ground truth.
The moment decoder contains two parts: (i) Moment-moment Interaction, which is achieved by self-attention, and (ii) Anchor Highlighting, which aims to not only highlight the area that is relevant to the current moment but also be aware of the global context. The highlighting, or searching for relevant moments, is achieved through an Anchor Highlight Attention, an modification of the cross attention in DETR with RoI features, shown in Fig. 4. All attentions mentioned above follow the specification of multi-head scaled dot-product defined in Vaswani et al. (2017).
Learnable Templates and Anchors In the original DETR (Carion et al., 2020) paper, the learnable templates can be seen as special positional embeddings, to provide a spatial prior of objects. However, the recent success of advanced DETRs (Liu et al., 2022; Meng et al., 2021) motivates us to separately model a moment anchor according to which the attention is constrained. Let k denote the index
of templates among the total Nq templates. We define qk as the kth learnable template and (c0k, w 0 k) as its initial anchor. Here c and w stand for the center and width of the corresponding moment, which can be easily mapped to the start/end boundary. Anchors will be refined in the decoder, layer by layer. We use (clk, w l k) to denote the anchor after refinement of the lth decoder layer.
Anchor Highlight Attention. One of our motivations is to discriminate the best matching moment among all candidate moments that share good matching to the text query. To highlight the areas that are similar to the current moment, we modify the attention query to adjust attention weight.
Suppose the current anchor is (ck, wk), we can easily locate the corresponding area in the nth multi-scale feature from the encoder output. We use rc,k to denote the features in this area that are taken from the cth multi-scale video features Vcs. Let Rk be the collection of features from all scales. We then construct a function f to map Rk to a single vector Hk ∈ Rd to guide the highlight in attention mechanism, illustrated in Fig. 4. Let HNq×d ∈ R be the collection of Hk and M be the moment features after self-attention module in decoder layer, we adjust the attention as follows:
AAH = FFN(M+H)FFN(VC−1s ) T
√ dh
MAH = AAH × FFN(VC−1s ) (9)
Here, AAH refers to the adjusted attention weight, and MAH is the output of the adjusted cross attention. Since H is sampled and transformed from the corresponding anchor areas in encoder outputs Vs, it is essentially the representation of moment content. Therefore, the term H(VC−1s )
T will be more responsive when a specific area from VC−1s is similar to the moment content. Consequently, the attention above will highlight the areas similar
to the current moment. We then refine the anchors based on these highlighted areas, through an offset prediction head as shown in Fig. 4.
Anchor Refinement. Based on the predictions from the last decoder block, we revise the anchor with the predicted offsets. This is analogous to the eye skimming process of humans: focuses on a local area in the video and then decides where to move her sight at the next step. The anchors are refined iteratively as shown in Fig. 5. Specifically, we first project the center clk and scale s l k of the kth anchor at the lth decoder level into logit space, using an inverse sigmoid operation. The offset (∆clk,∆w l k) is added to their logits, then the modified logits are projected back using sigmoid. The whole process is described in Eq. 10.
cm+1k = σ ( ∆cmk + σ −1(cmk ) )
wm+1k = σ ( ∆wmk + σ −1(wmk ) ) (10)
Here σ stands for sigmoid function, and σ−1 for inverse sigmoid function.
Boundary Modeling. After encoding moment candidate features, we pass them through two separate FFNs to predict anchor offset and scores, respectively. Depending on anchor positions, only a small portion of anchors may match with ground truth moments. Among them, we simply select the candidate moment with the largest IoU (intersection over union) with ground truth as our positive sample. A similar label assignment strategy has been used in early studies (Carion et al., 2020).
After labeling predictions as positive or negative, we refer to the index of positive prediction as ip. Then we model the boundary with two losses: (i) IoU prediction loss, and (ii) L1 regression loss. Note that, L1 regression loss is only applied to the positive prediction. Let (tks , t k e) be the timestamps predicted by kth anchor and (tgs, t g e) be the groundtruth timestamps, we calculate L1 regression loss and IoU prediction loss as follows:
LL1 = 1
2
( |tips − tgs|+ |t ip e − tge| )
LIoU = 1
Nq
∑
k∈Nq Focal(TrIoUk, ok)
(11)
Here TrIoU truncates IoU between (tks , t k e) and (tgs, t g e) below a threshold θ and set IoU of the assigned positive prediction to 1. Different from Carion et al. (2020), by using TrIoU , we not only
calculate IoU loss for the positive prediction but also consider the hard negative predictions which have large overlapping with ground-truth. Note that, IoU prediction loss and L1 regression loss are calculated for all decoder layer outputs.

heading: 4.3 Training and Inference
text: The overall training loss of MS-DETR contains three losses:
L = λspanLspan + λmaskLmask + ∑
m∈Ldec (λIoULIoU + λL1LL1) (12)
To stabilize training, we introduce an extra denoising group of templates and pass them through the decoder, motivated by (Chen et al., 2022). The overall loss is averaged over losses calculated from two groups independently. During inference, we deprecate the denoising group and use the main group only. All moments are sorted by their scores and their anchors are converted from (c, w) to start/end format. We apply truncation to start/end timestamps to deal with out-of-range values, since no constraint is attached to (c, w) during training.

heading: 5 Experiments
text: We evaluate MS-DETR against baselines on three public benchmarks: ActivityNet Captions (Krishna et al., 2017), TACoS (Regneri et al., 2013), and Charades-STA (Gao et al., 2017). The three datasets cover videos from different domains and lengths (see Appendix A.1 for video distributions and train/dev/test splits).
Following prior work (Zhang et al., 2021a), we adopt “R@n, IoU = µ” and “mIoU” as evaluation metrics. R@n, IoU = µ is the percentage of testing samples that have at least one of top-n results hitting ground truth, where “hitting” means an overlapping with IoU ≥ µ. mIoU denotes the average IoU over all test samples. We set n = 1 and µ = {0.3, 0.5, 0.7}. In our comparison and discussion, we mainly focus on µ = 0.7 as large IoU means high-quality matching.

heading: 5.1 Comparison with the State-of-the-Arts
text: Results on the three datasets are compared in Tables 1, 2, and 3, respectively. Baseline results are mostly cited from (Zhang et al., 2021a). We also include GTR (Cao et al., 2021), LP-Net (Xiao et al., 2021a) and MMN (Wang et al., 2022) for a complete comparison.
MS-DETR achieves the best R@1, µ = 0.7 and mIoU on ActivityNet and TACos, and the second best on Charades-STA. Our model achieves reasonably good results on smaller µ’s. However, large µ ensures high-quality matching. A possible reason for the results on Charades-STA is that the videos in this dataset are very short (30 seconds on average), making moment-level interaction less necessary.

heading: 5.2 Ablation Study
text: We perform ablation studies on ActivityNet Captions for the effectiveness MS-DETR.
Multi-scale Encoder. We evaluate four variants to study the effectiveness of multi-scale design in our transformer encoder. First, to evaluate whether
hierarchical design benefits cross-modal interaction, the ‘uni-scale’ variant replaces all sequencereduced layers with normal layers without resolution shrinkage, and set the number of clips to 32. The multi-scale transformer now degrades to a uniscale cross-modal transformer. To study the contribution of encoding moment contents R for anchor highlighting in multiple scales, the ‘single-scale’ variant selects the output of the last encoder layer only and fuses it to attention query, while keeping encoder’s hierarchical structure. Then, we study the effect of arranging sequence-reduced layers in different positions in the 5 encoder layers. We compare two arrangements “BBBRR” and “RBBBR” against MS-DETR’s “RRBBB”. Here ‘R’ means sequence-reduced and ‘B’ means the base version.
Results in Table 4 suggest the effectiveness of multi-scale hierarchical encoder. Performance drops with the removal of the multi-scale mechanism, or the other arrangement of sequencereduced layers. Placing sequence-reduced version at shallow layers serves the purpose of reducing computational cost while benefiting performance.
Anchor Highlight Attention is a variant of standard cross attention Vaswani et al. (2017). It is used to highlight similar content with correspond-
ing moments across the video. We compare its design with the standard cross attention. Table 5 shows that anchor highlight attention outperforms standard cross attention, by a large margin. This justifies the advantage of using anchor highlight attention and dynamic anchor jointly, to narrow the range of attention to anchor areas.
The Auxiliary Loss. We use two auxiliary supervision losses, span loss and word mask loss, in our encoder (see Section 4.1). Table 6 reports the results of removing either one or both auxiliary losses. Results suggest that both auxiliary losses benefit MS-DETR, and span loss contributes more to the effectiveness of MS-DETR. That is, supervising encoder to discriminate whether segments fall within the ground-truth area is important for vision-language alignment.
Hyper-parameter Study. Results of the choices of the number of encoder/decoder blocks, and number of denoising groups for training stabilization are in Appendix A.3.

heading: 6 Conclusion
text: In this paper, we adapt DETR framework from object detection to NLVL. With the proposed MSDETR, we are able to model moment-moment interaction in a dynamic manner. Specifically, we design a multi-scale visual-linguistic encoder to learn hierarchical text-enhanced video features, and an anchor-guided moment decoder to guide the attention with dynamic anchors for iterative anchor refinement. The promising results on three benchmarks suggest that moment-moment interaction for NLVL can be achieved in an efficient and effective manner.

heading: 7 Limitation
text: The limitation of this paper are twofold. First, our method does not provide a recipe for data imbalancement in NLVL task. Thus, our method does not guarantee the effectiveness on edge cases. Second, the choice of feature extractor is considered relatively outdated. Our model does not benefit from the recent development of pre-trained visionlanguage models. On the other hand, using pretrained vision-language models remains in its early stage in NLVL tasks. Not using pre-trained features makes a fair comparison between our model with existing baselines. As a part of future work, we will explore the potential of using more powerful feature extractors in our model.

heading: 8 Acknowledgement
text: This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

heading: A Appendix
text: A.1 Dataset Details
ActivityNet Captions (Krishna et al., 2017) contains over 20K videos paired with 100K queries with an average duration of 2 minutes. We use the dataset split “val_1” as our validation set and “val_2” as our testing set. In our setting, 37, 417, 17, 505, and 17, 031 moment-sentence pairs are used for training, validation, and testing, respectively. TACoS (Regneri et al., 2013) includes 127 videos about cooking activities. The average duration of videos in TACoS is 7 minutes. We follow the standard split which includes 10, 146, 4, 589, and 4, 083 moment-sentence pairs for training, validation, and testing. Charades-STA (Gao et al., 2017) is built on Charades and contains 6, 672 videos of daily indoor activities. Charades-STA has 16, 128 sentence-moment pairs in total, where 12, 408 pairs are for training and 3, 720 pairs for testing. The average duration of the videos is 30s.
A.2 Implementation Details
We use AdamW with learning rate of 3 × 10−4 and batch size of 32 for optimization. We follow (Zhang et al., 2020b) and use pretrained 3D Inception network to extract features from videos. The number of sampled video frames is set to 512 for ActivityNet Caption and TACoS and 1024 for Charades-STA. For MS-DETR architecture, we use a 5-layers encoder and a 5-layers decoder with all hidden sizes set to 512. For inference, we select the proposal with highest score from the last decoder layer as our prediction. As for the specific choice of f mentioned in Section 4.2, we use RoIAlign to extract multi-scale feature R, then concatenate them and pass them through an FFN. One extra denoising group is used for stabilizing training. The loss is then averaged over two groups. During inference, No extra operation like Non-Maximum Suppression (NMS) is required. All experiments are run on a single A100 GPU. The reported versions take roughly 8-10 GPU hours for training.
A.3 Hyper-parameter Study Number of Encoder/Decoder Blocks We study the impact of the number of encoder and decoder blocks. We evaluate one of them from 2 to 6, while keeping the other fixed to 5. The performance across various numbers of encoder and decoder blocks are listed in Table 7. Best performance is achieved by Lenc = 5 and Ldec = 5. Though the setting of Ldec = 6 has slightly larger “R@1, IoU = 0.7”, poorer “mIoU” is observed. We speculate that the cause is overfitting on some overly confident examples.
Number of Denoising Groups. We study the effectiveness of using different numbers of denoising groups in training stabilization. The results are evaluated with the number of denoising groups ranging from 0 to 3, in Table 9. We observe the performance increases after using one denoising group, then gradually decreases. We suspect there is a trade-off between training stability and the ability to escape local minima.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
7 A2. Did you discuss any potential risks of your work? Our paper uses public benchmarks and does not have any risk of ethics or infringement.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 5 and A.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use three benchmarks, which are all open to public
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No, the benchmarks we use are all open to the public and do not contain any private information.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section A.1
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section A.2
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: As a part of future work, we will explore the potential of using more powerful feature extractors in our model