CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding
Year: 2023

Abstract:
This paper tackles an emerging and challenging problem of long video temporal grounding (VTG) that localizes video moments related to a natural language (NL) query. Compared with short videos, long videos are also highlydemanded but less explored, which brings new challenges in higher inference computation cost and weaker multi-modal alignment. To address these challenges, we propose CONE, an efficient COarse-to-fiNE alignment framework. CONE is a plug-and-play framework on top of existing VTG models to handle long videos through a sliding window mechanism. Specifically, CONE (1) introduces a query-guided window selection strategy to speed up inference, and (2) proposes a coarse-to-fine mechanism via a novel incorporation of contrastive learning to enhance multi-modal alignment for long videos. Extensive experiments on two large-scale long VTG benchmarks consistently show both substantial performance gains (e.g., 3.13% +119% −−−→6.87% on MAD) and state-of-theart results. Analyses also reveal higher efficiency as the query-guided window selection mechanism accelerates inference time by 2x on Ego4D-NLQ and 15x on MAD while keeping SOTA results. Codes have been released at https://github.com/houzhijian/CONE.

heading: 1 Introduction
text: Video temporal grounding (Anne Hendricks et al., 2017; Gao et al., 2017) aims to locate specific moments in an untrimmed video relevant to a textual user query. This is a crucial task in multi-modal video understanding and can be applied to many practical applications such as video retrieval (Xu et al., 2016), video editing, and video question answering (Lei et al., 2018, 2020a).
Temporal grounding for long videos is especially highly-demanded and emerging due to the
∗ Indicates equal contribution. This work was done during the first and second authors’ internships in MSR Asia.
flourishing growth of online videos in quantity and length. However, limited by previously available datasets (Gao et al., 2017; Krishna et al., 2017; Lei et al., 2021), existing works (Zhang et al., 2020a,b) mainly deal with relatively short videos ranging from 0.5 to 2.5 minutes on average. Recently, Ego4D (Grauman et al., 2022) and MAD (Soldan et al., 2022) datasets have been created and attempted to deal with long video, which spans from several minutes to hours.
Early attempts for long video grounding extends existing VTG models for short videos through sparse sampling (Grauman et al., 2022) or a sliding window adaptation (Soldan et al., 2022). On the one hand, existing models for short videos typically downsample videos to a fixed-length frame sequence, which leads to temporal information loss for long videos (i.e., fewer visible frames via sparse sampling). Besides, the window-based adaptation methods divide the long video into candidate windows and inference on every window, which leads to high inference computational cost. On the other hand, massive moment candidates from long videos make their precise multi-modal alignment with the NL query more challenging, which leads to contextual information loss (i.e., weaker alignment to fine-grained contents, like objects, actions, and
8013
scenes). As the motivating example shown in Fig. 1, accurate grounding requires a coarse-grained localization of relevant video segments (e.g. “in the room" v.s. “outdoor"), and fine-grained alignment to detailed object and action in frames (e.g., “women" and “holding"). In conclusion, long video length poses two new challenges: (1) higher computational cost during inference on the numerous windows of the entire long video; (2) weaker multimodal alignment due to the abundance of moment candidates.
To address these challenges, we propose CONE, a COarse-to-fiNE alignment framework for long video temporal grounding. First, we slice the arbitrary long video into candidate windows. Then, we employ a query-guided window selection strategy for efficiency and further introduce a coarse-tofine mechanism for effectiveness. Specifically, the query-guided window selection strategy efficiently reduces the sizeable window number of the long video to a modest amount via efficient alignment score computation. Moreover, the coarse-to-fine alignment mechanism consists of three modules to gradually align the multi-modal inputs via multiscale granularity: (1) window (coarse granularity) selection to reliably select semantically relevant candidate windows; (2) window (coarse)-proposal (fine) joint contrastive learning to generate candidate proposals considering both inter-window and intra-window relations; (3) proposal (fine granularity) ranking to further accurately sort out the perfect proposal. Notably, we incorporate contrastive learning into this mechanism. On the one hand, we utilize the powerful multi-modal alignment ability of contrastive vision-text pre-trained models (e.g., CLIP), which computes matching scores between video frames and textual query for both window selection and proposal ranking stages. On the other hand, we select a contrasting negative window and design an inter-window contrastive loss during training for the joint learning stage.
With this coarse-to-fine design, CONE has the following advantages: (1) higher efficiency in handling long video inputs with less temporal information loss; (2) more accurate multi-modal alignment with less contextual information loss. We evaluate CONE on two large-scale long video grounding benchmarks and achieve both the state-ofthe-art results and significant performance boosts (3.13%→6.87% on MAD, and 10.46%→13.46% on Ego4D-NLQ in terms of R1@IoU=0.3). Further
analysis shows that the window selection strategy shortens the inference speed by 2x for Ego4D-NLQ and 15x for MAD compared to inference on every window, without sacrificing its performance.
Contributions Our work presents two major contributions to the long video temporal grounding field. (1) We propose a novel coarse-to-fine alignment framework that utilizes a pipeline of {window slicing and selection, proposal generation and ranking}, resulting in state-of-the-art performance and high efficiency on two representative benchmarks. (2) We introduce a novel approach for incorporating contrastive learning into multi-modal alignment.

heading: 2 Related Work
text: Video Temporal Grounding. Current models for this task typically fall into two categories based on the usage of proposals. Proposal-free methods predict start/end timestamps directly, bypassing the generation of proposals (Ghosh et al., 2019; Zeng et al., 2020; Zhang et al., 2021c). Conversely, Proposal-based methods merge proposal generation and ranking within an end-to-end framework (Zhang et al., 2020b; Lei et al., 2021; Cao et al., 2021). An excellent survey paper provides further details on these models (Zhang et al., 2022). However, these models are predominantly designed for relatively short videos, which leads to substantial information loss when adapted directly to long-form settings. Regarding long-form video grounding, VSLNet-L (Zhang et al., 2021b) extends VSLNet (Zhang et al., 2020a) with a multiscale split-and-concat mechanism to address performance degradation. Still, the multi-scale mechanism adds computation cost during inference, and the sparse sampling approach suffers from significant temporal information loss for hour-long videos. Authors of Ego4D and MAD adapt 2DTAN (Zhang et al., 2020b) and VLG-Net (Soldan et al., 2021) into simple window-based baselines, yet these models lack the ability for coarse-to-fine alignment, affecting the final results.
Furthermore, other studies have focused on different issues. For example, NaQ (Ramakrishnan et al., 2023) addresses data scarcity with an effective data augmentation strategy, while DeNet (Zhou et al., 2021) tackles the ground-truth bias problem with a de-coupling and de-bias strategy. There are also other related tasks involving language grounding with video corpus input (Lei et al., 2020b; Hou
et al., 2021) or spatial-temporal output (Yang et al., 2022), but they fall outside the scope of this paper. Long-form Video Modeling. Long-form video modeling is an emerging topic recently investigated in (multi-modal) video understanding, including classification, segmentation, localization, and retrieval. The common challenges of long videos are modeling long-range temporal dependency, efficiency issues, and accurate multi-modal alignment (if language is involved). To ease long-range dependency issues, some works explore either feature memory banks (Wu et al., 2019; Feng Cheng, 2022), tracked object-level representations (Wu and Krahenbuhl, 2021), bayesian non-parametric model (Qiu et al., 2023), or structured state-space sequence layers (Islam and Bertasius, 2022). To alleviate efficiency issues, other works (Lin et al., 2022b) exploit dense audio sampling as additional information to enable sparse video frame sampling. In contrast, CONE mitigates both efficiency issues via query-guided window selection strategy and multi-modal alignment issues via novel contrastive learning incorporation. Contrastive Learning. Contrastive learning has been widely studied in vision (Misra and Maaten, 2020; He et al., 2020), language (Gao et al., 2021b; Meng et al., 2021), and multi-modal fields (Liang et al., 2020; Yan et al., 2021; Wang et al., 2022). Some recent VTG works also attempt to adopt contrastive learning. They mainly focus on frame-level contrastive loss via hard negative mining (Zheng et al., 2022) or contrasting ground-truth frame with non-ground-truth frame (Nan et al., 2021; Zhang et al., 2021a). In contrast, we further propose window-level contrastive learning to repel the negative windows. Moreover, CONE also jointly incorporates contrastive vision-text pre-trained models for accurate alignment, because those contrastive models show strong multi-modal alignment ability (Luo et al., 2021; Xu et al., 2021) derived from pre-training large-scale vision-text pairs.

heading: 3 Task Definition
text: Since never-ending video streams in real applications have a higher demand for long videos, in this paper, we study the task of video temporal grounding (VTG) in a more challenging setting with long video inputs. Taking a natural language (NL) query Q and a long video V as the inputs, the task of VTG requires the system to locate the matched moment M from the video V relevant to the query
Q. Formally, video V = [v1, v2, . . . , vLv ] is a sequence of uniformly sampled video frames, and Lv denotes the length of the sampled frames. The NL query Q = [q1, q2, . . . , qLq ] is a sequence of tokens with sentence length Lq. The moment M is a subsequence of V that is relevant to Q. The long-form video setting is more challenging because of the natural demands for more computation and time to process the entire video. Moreover, the accurate multi-modal alignment between each vi and Q is also harder when Lv increases.

heading: 4 Approach
text: We present the proposed CONE for long video temporal grounding. As shown in Fig. 2, we first slice the long video into several fixed-length video windows via a sliding window mechanism (§ 4.1). Then, we propose a coarse-to-fine mechanism for efficient and effective multi-modal alignment. At the coarse-grained window level, we introduce a query-guided window selection strategy (§ 4.2) to accelerate inference and select semantic relevant windows. At the window-proposal joint learning level, we rely on the existing VTG work for short videos to generate reliable candidate proposals and conduct both intra-window and proposed interwindow contrastive learning (§ 4.3) to assign each proposal score. At the fine-grained proposal level, we further accurately rank candidate proposals with a fine-grained ranking (§ 4.4) mechanism.

heading: 4.1 Window-based Video Slicing
text: To flexibly handle long videos without decreasing the sample rate and alleviate temporal information loss, we first slice the entire video V into several video windows W . A sliding window with window length Lw is used to be slid on the entire video to derive a set of Nw fixed-length video windows Wi = [vwbi+1 , vwbi+2 , ..., vwbi+Lw
], where wbi is the start index of window i. Concretely, we slide the window by increasing wb with window stride Lw/2. Intuitively, not every window is correlated with the NL query, so we refer the positive window to the window overlapping with the ground-truth moment, and the negative window otherwise.

heading: 4.2 Coarse-grained Window Selection
text: Lengthy video input is sliced into a sequence of windows during inference. If the number of windows is Nw, the model needs to conduct the whole encoding-prediction process for Nw times, which
will become computationally costly with increased video length, especially when the model has enormous parameters. Therefore, it is necessary to reduce the inference computation cost by reliably filtering windows irrelevant to the natural language descriptions. We propose a query-guided window selection strategy via contrastive vision-text pretrained models.
Vision-Text Contrastive Model. CLIP (Radford et al., 2021) and EgoVLP (Lin et al., 2022a) are pre-trained with large-scale vision-text pairs via contrastive learning, aiming to align the visual representation with its related text representation. So they excel at multi-modal alignment, and efficient dot-product score computation provides higher matching efficiency. We utilize the pre-trained model to compute the video features V and the text features Q beforehand.
V = [v1,v2, . . . ,vLv ] Q = [q[CLS], q1, q2, . . . , qLq ] (1)
where [CLS] is a special token at the beginning of text tokens. The multi-modal matching score aj = vj · q[CLS] is computed via the efficient dot product between jth video feature and the text feature. And the window-level matching score Ai is the maximum score of all the frames in window i:
Ai = max([awbi+1 , awbi+2 , ..., awbi+Lw ]) (2) We rank all windows with Ai and select the top-k windows for inference. Thus, we reduce the number of candidate windows from Nw to a constant k to guarantee a controllable computation cost in an efficient and reliable manner.

heading: 4.3 Window (Coarse)-Proposal (Fine)
text: Joint Contrastive Learning
Since there are many excellent works in the VTG literature for short video input, we target CONE as a flexible plug-and-play framework on top of the existing proposed-based model. Existing models function as the base model to generate reliable candidate moment proposals for further processing. In our scenario, the base model takes window W and the NL query Q as inputs and outputs several moment proposal candidates. Each of them has a moment proposal (pi) and corresponding score (si), respectively.
Nevertheless, the training of the base model typically focuses only on the positive window and conducts intra-window relation learning but neglects plenty of negative windows for long videos. In real practice, negative windows are the majority during inference, which results in a discrepancy between training and inference. To mitigate the discrepancy, we further design inter-window contrastive learning to distinguish the positive and negative windows. As a result, the overall training loss consists of two parts: (1) the intra-window loss of the base model, and (2) the proposed inter-window contrastive loss.
Concretely, each training instance has both a positive window and a random contrasting negative window from the long video. We discriminate the negative window from the positive window through proposal-level comparison. Proposals in the negative window should be assigned with minimized scores compared with positive proposals (e.g., the IoU with ground truth is large than 0.7) in the posi-
tive window, as follows,
Lcon = − ∑
p+i ∈W+ log(si)−
∑
p−i ∈W− log(1− si)
(3) where Lcon is the proposed contrastive loss, p+i is each positive proposal from the positive window W+ and p−i is each proposal from the negative window W−, si is the corresponding proposal score.
This mechanism can be generalized to different existing proposal-based models with different intrawindow losses. To show the generalization ability of CONE, we test on two kinds of base models: 2DTAN (Zhang et al., 2020b) and Moment-DETR(Lei et al., 2021) due to their available codes and superior performances. Please refer to Appendix A.2 for comprehensive training details.

heading: 4.4 Fine-grained Proposal Ranking
text: With the increased length of video inputs, the finegrained attention between each video frame and the text query will be weakened by many other perturbed frames, resulting in contextual information loss. To remedy this issue, we propose a finegrained ranking strategy to conduct accurate proposal ranking utilizing proposal matching scores computed by contrastive vision-text pre-trained models (described in § 4.2). Note that we simply re-use the pre-computed video frames and text query features as in Eq. (1).
Visual Adapter. With a lightweight visual adapter on top of CLIP, we exploit adapter-based tuning to adapt the representations from the general contrastive model to the data distribution of the current downstream task. Inspired by Gao et al. (2021a), our main idea is to add an additional bottleneck layer to learn the task-adaptive visual features and conduct residual-style blending with the original pre-trained features. The lightweight adapter complies with a 2-layer FFN followed by ReLU. The ith adapted visual feature is: v̂i = Adapter(vi) + vi. Then, the proposal feature for the jth proposal is computed with the mean pooling of all the adapted video features in it: hj = Mean([v̂bj , . . . , v̂ej ])
For adapter training, we denote the positive proposal (with feature hpos) as the ground-truth one, and the negative proposals are the other in the same batch. We follow the standard contrastive learning and use the NCE loss:
Ladapt = − ∑
pos
(log exp(hpos · q[CLS])∑ j exp(hj · q[CLS]) ) (4)
Note that we also use the adapted visual feature to compute the window-level matching score in § 4.2 to improve window selection quality.
Ranking Score Computation. Finally, we aim to conduct a fine-grained ranking for proposals. For the jth proposal, the final ranking score is fused with two components: (1) proposal scores sj generated from the previous module and (2) fine-grained matching scores mj computed by adapted CLIPbased proposal feature: mj = hj · q[CLS]. The former captures the correlation between proposals via the sophisticated contextual model design, while the latter focuses on fine-grained content matching between frames in the proposal and the textual query. We perform min-max normalization for these two types of scores for a more stable score fusion. The final ranking score rj is the sum of two normalized scores:
s̃j = MinMax([s1, s2, ..., sNp ]),
m̃j = MinMax([m1,m2, ...,mNp ])
rj = s̃j + m̃j
(5)
where Np is the total candidate proposal number.

heading: 5 Experiments
text: We conduct experiments to explore the effectiveness of CONE from the following aspects: (1) model comparison with SOTA methods (§ 5.3); (2) ablation study to analyze the impact of each component and different variants (§ 5.4); (3) efficiency analysis of acceleration with window selection (§ 5.5) and (4) qualitative analysis (§ 5.6). Implementation details are given in Appendix A.1.

heading: 5.1 Dataset
text: We conduct comprehensive experiments on two representative large-scale benchmarks on long video
temporal grounding: Ego4D-NLQ (Grauman et al., 2022) and MAD (Soldan et al., 2022). The data statistics are summarized in Table 1.
Ego4D-NLQ is a subtask of the Ego4D dataset. Ego4D is a large-scale egocentric video understanding benchmark, where 931 camera wearers worldwide record their daily activities in hundreds of scenarios. The unedited videos involved have variant lengths ranging from 3.5 min. to 20 min. The NL query is designed to retrieve the relevant moment from the episodic memory of camera wearers, and involves 13 question types for locating different types of information.
MAD is a large-scale long video temporal grounding benchmark on full-length movie videos. The video duration is magnificent longer than previous benchmarks at an average length of 110.8 min (ranging from 47 min. to 202 min). The textual queries in the training set are derived from translated movie audio descriptions from professional narrators. The annotations in the evaluation set are of higher quality derived from the LSMDC
dataset (Rohrbach et al., 2017) with more strict manual refinement. The NL queries are cleaner and the ground-truth moments have preciser temporal boundaries.

heading: 5.2 Experimental Settings

heading: 5.2.1 Evaluation Metric.
text: For consistent comparison, we follow the previous baseline evaluation setting and use the metric Recall@k at IoU = θ (R@k ). It is the percentage of queries, having at least one prediction among the top-k predictions, whose temporal IoU with ground-truth is larger than the threshold θ (0.3 or 0.5). Note that there is only one ground-truth answer for each query in both datasets.

heading: 5.2.2 Visual and Textual Features.
text: For MAD dataset, we use CLIP (Radford et al., 2021) to extract the visual and textual features, which are then utilized in the inputs of the window selection, proposal generation, and fine-grained ranking stages. The MAD authors extract the visual features every 0.2s (5fps). For Ego4D-NLQ dataset, we adopt the egocentric contrastive pre-trained model, i.e., EgoVLP (Lin et al., 2022a), to extract visual and textual features, because CLIP is trained with third-person image-text pairs and has the domain adaptation gap for first-person videos. We extract the visual feature every 16 frames (1.875fps).

heading: 5.3 Model Comparison

heading: 5.3.1 Baselines.
text: We compare CONE to the following methods: (1) the window-based adaptation of proposalbased models, i.e., 2D-TAN (Zhang et al., 2020b), VLG-Net (Soldan et al., 2021) and MomentDETR (Lei et al., 2021); (2) the sparse samplingbased proposal-free model VSLNet (Zhang et al., 2020a); (3) the two-stage model CLIP (Radford et al., 2021), which first generates offline proposals
and then ranks those proposals with CLIP matching scores.
Results on Ego4D-NLQ. Table 3 reports the performance comparison on the validation and the blind test set of Ego4D-NLQ. Regarding all the metrics, CONE outperforms these baselines by a large margin. Take R1@IoU=0.3 and R5@IoU=0.3 as examples, the absolute performance gains are +3.31% and +11.49% on the val. set, and +3% and +6.92% on the blind test set, respectively. CONE also achieves consistent performance gains using different features and base models, showing a better generalization ability.
Results on MAD. The main results of MAD are shown in Table 2, and demonstrate that CONE achieves state-of-the-art performances with obvious boost (e.g., 3.13% +119%−−−−→6.87% and 9.85% +63.5%−−−−→16.11% improvement on R1@IoU=0.3 and R5@IoU=0.3).
From the two tables, the SOTA performance on two benchmarks demonstrates the effectiveness of CONE. We speculate the outstanding results are brought by the following reasons: (1) CONE processes the entire video without decreasing sample rate, which alleviates temporal information loss; (2) the coarse-to-fine mechanism enables the better alignment ability of relevant proposals to the NL query and reduces contextual information loss. The lower results on the MAD benchmark also verify that the grounding task indeed becomes more challenging with longer videos.
Regarding base model selection, we observe that, compared with Moment-DETR, the performance of 2D-TAN is worse in Ego4D-NLQ but comparable in the MAD dataset. This gives the clues that the base model might not be influential to the final results given more training samples for hourlevel videos. Because of the lower parameter and GLOPs number of Moment-DETR, from now on, CONE adopts it as the default base model.

heading: 5.4 Model Analysis
text: Ablation studies are shown in table 4 to unveil the effectiveness of each component in CONE. We consider three components: (a) contrastive loss (§ 4.3), which is eliminated by only training on positive windows without our contrastive loss; (b) finegrained ranking fusion (§ 4.4) can be removed by taking only the proposal score from the proposal generation module for ranking; (c) visual adapter
(§ 4.4) is removed by using general CLIP-based features for matching score computation. Note that the full CONE model refers to the first row, and the baseline model (i.e., the window-based adaptation of Moment-DETR) refers to the last row.
From the table 4, we highlight the following findings for Ego4D-NLQ dataset (MAD shows similar trends): (1) Ablating visual adapter leads to a performance drop from 14.15% to 12.62%, which indicates that domain-adaptation of visual features is essential in modelling task-specific semantic variance. (2) Eliminating fine-grained ranking also harms the performance (row 2 vs. row 3), showing that capturing fine-grained semantic alignment benefits accurate grounding. (3) Further removing contrastive loss (row 3 vs. row 4) leads to a significant performance drop of 3.36%, which reveals that identifying the inter-window semantic variance is critical for reliable grounding.
Influence of window length. Figure 3 exhibits how different window lengths affect the overall performance of CONE. We observe that increasing the window length indeed brings performance variance. Longer window length (more visible frames in a single window) can model the interaction between more frames, but can also weaken the multi-modal attention between the NL query and every single frame as the performance drops significantly with the largest window size. We find a better tradeoff between window length and performance, i.e., nearly 48 seconds ( 90 video features) for Ego4DNLQ and nearly 25 seconds (125 video features) for MAD through this analysis.
5.5 Efficiency Analysis
Figure 4 shows inference speed with respect to different window numbers after the window selection stage on the val. sets of Ego4D-NLQ and MAD. We observe that the overall inference time is approximately linearly reduced with smaller window numbers, and the performance of CONE is relatively stable when the filtered window number becomes 10 for both datasets. Surprisingly, we also observe that CONE achieves the optimal performance (shown as the box with value in Fig. 4) at a modest size of window number rather than the full size. Those observations show that our queryguided window selection strategy can reliably filter the irrelevant windows and largely improve the efficiency of long video temporal grounding.
Concretely, the average window numbers for full videos (before filtering) are 23.3 and 588 for Ego4D-NLQ and MAD, respectively. If we set the filtered window number to 10 for Ego4D-NLQ and 30 for MAD (better trade-off values obtained from Fig. 4), the total numbers of windows for inference are reduced by 2.3x and 19.6x, with a marginal performance variance (R@1) by -0.1% and +0.2% for Ego4D-NLQ and MAD benchmarks, respectively. In real practice, we consider the effect of implementation methods and window selection time. When CONE is inferred using one P100 GPU, its running time (w/o feature extraction and post-processing time, because both time costs are the same for all methods) is largely reduced (74 s -2x−−→36 s and 276 min. -15x−−→18 min.) on the val. set of Ego4D-NLQ and MAD.
Furthermore, we also conduct an inference comparison between VSLNet and CONE on Ego4DNLQ. Using the official Ego4D released code, VSLNet takes approximately 12 seconds to infer the Ego4D validation split and achieves a Recall@1 score of 10.84 at IoU=0.3. In contrast, CONE, with
the extreme setting of selecting only the top 1 window, takes about 10.5 seconds to infer the Ego4D validation split and achieves a Recall@1 score of 12.4 at IoU=0.3. It demonstrates that CONE strikes an optimal tradeoff between accuracy and speed compared to sparse sampling-based baselines.

heading: 5.6 Qualitative Analysis
text: Fig. 5 shows two success examples to analyze the effect of contrastive learning and fine-grained ranking. We observe that baseline Moment-DETR (a) is not capable of repelling negative windows and tends to give an equally high score to the proposal in the negative window, thus it wrongly ranks the correct prediction to a much lower position (e.g., 54th in Example A). Further adding inter-window contrastive learning mechanism (b), the rank position of the ground truth moment is somewhat improved, but it still lacks fine-grained matching to detailed contents to push the perfect proposal into the first place. For example, in Query-A, the most essential object is the vegetable rather than the bag; In Query-B, the query requires the precise scene “in the park" and the objects “two women" and “ribbons". Finally, adding fine-grained ranking (c), the full CONE successfully locates the groundtruth moment relevant to the textual query. More qualitative examples with both success and failure cases are shown in Appendix B.

heading: 6 Conclusions
text: We present CONE , a COarse-to-fiNE alignment framework for long video temporal grounding. CONE jointly achieves state-of-the-art performances on two benchmarks and high efficiency while keeping high frame sample rate. Through our experiments, we show the proposed coarse-tofine mechanism via contrastive learning plays an important role in boosting performance. Regarding efficiency, the introduced query-guided window selection strategy largely accelerates inference by 2x and 15x on Ego4D-NLQ and MAD benchmarks, respectively, with even slight performance gains. Since CONE can be generalized with different existing proposal-based models, we hope it can be used to improve the model efficiency and performance for temporal grounding on long videos.
Limitations
Currently, CONE is mainly implemented for proposal-based models as they can generate ex-
plicit moment proposals for the introduced interwindow contrastive learning. In contrast, proposalfree methods (e.g., VSLNet (Zhang et al., 2020a)) directly predict the start/end timestamps without explicit proposals. In the future, it is worthwhile to explore how to incorporate the coarse-to-fine alignment mechanism with proposal-free methods to enhance the generalization ability of CONE.
Furthermore, CONE falls short on the groundtruth moment case whose duration is longer than the adopted video window duration. To ease this issue, future work can explore adaptive-duration window slicing to ensure complete containment of scenes and events within windows or some rulebased proposal merging techniques.
Ethics Statement
The present study was conducted in accordance with ethical principles. This study involved the analysis using publicly available data and did not involve any human participants, and potential risks about credentials or privacy. Therefore, no ethical clearance was required and there were no potential risks associated with the conduct of this research.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful feedback. This research was partially supported by CityU MF_EXT project number 9678180.

heading: A Model Details
text: A.1 Implementation Details
During training, we perform parameter optimization via AdamX and set the learning rate to 1e-4 for the base model and 1e-5 for the visual adapter. We set the batch size to 32 and adopt the early stopping strategy. During inference, we use NonMaximum Suppression (NMS) with a threshold of 0.5 as post-processing.
Moment-DETR Base Model. The experiments are conducted on one P100 GPU. For the hyperparameters of network architecture, we set the hidden size d to 256, the transformer layer number in the encoder/decoder to 2, and the moment query number to 5. As a result, the total parameters of CONE are 4.35M (4.22M Moment-DETR + 0.13M visual adapter). We set the window length to 90 video features (48 seconds) and 125 video features (25 seconds) for the Ego4D-NLQ and MAD datasets, respectively. We train Ego4D-NLQ for 150 epochs and MAD for 30 epochs, and the training time is about 3 hours for the Ego4D-NLQ and 18 hours for the MAD dataset. During inference, we set the filtered window number to 10 and 30 for the Ego4D-NLQ and MAD datasets, respectively.
2D-TAN Base Model. The experiments are conducted on two V100 GPUs. For the hyperparameters of network architecture, we set the hidden size d to 256, the convolution network layer to 4, and the kernel size to 9. As a result, the total parameters of CONE are 23.99M (23.86M 2D-TAN + 0.13M visual adapter). We set the window length to 64 video features (34.1 seconds) and 128 video features (25.6 seconds) for the Ego4D-NLQ and MAD datasets, respectively. We train Ego4D-NLQ for 90 epochs and MAD for 6 epochs, and because of 6x more model parameters than Moment-DETR, the training time is quite larger, i.e., about 18 hours for the Ego4D-NLQ and 3 days for the MAD dataset. During inference, we set the filtered window number to 5 and 15 for the Ego4D-NLQ and MAD datasets, respectively.
A.2 Training Loss Details
Moment-DETR Base Model. The original loss of Moment-DETR consists of three parts: moment localization, classification, and saliency losses. Localization loss requires ground truth moment to measure the discrepancy with the predictions and
can not be used in contrastive loss. Thus, we design two-level contrastive losses based on two other losses: (1) proposal-level loss and (2) frame-level loss with a randomly sampled negative window.
For the proposal-level contrastive loss (Lp), proposals in the negative window are assigned with minimized scores compared to positive proposal (i.e., the optimal proposal selected from the Hungarian algorithm) from the positive window, as follows,
Lp = − ∑
p+i ∈W+ log(si)−
∑
p−i ∈W− log(1− si) (6)
where p+i is the positive proposal from the positive window W+ and p−i is each proposal from the negative window W−, si is the corresponding proposal score.
For the frame-level loss Lf , we set the average saliency scores for frames located in the positive window is larger than the maximum saliency score of frames in the negative window over a margin δ:
Lf = max(0, δ +max(S(W−, Q))− mean(S(W+, Q)) (7) where S() is its saliency scoring function. So the overall contrastive loss is Lcon = Lp + Lf . 2D-TAN Base Model. The original loss of 2DTAN consists of the binary cross-entropy loss, which learns to align each proposal score with its scaled IoU value. Thus, we assign each proposal in the negative window with minimized scores compared to the positive proposals (i.e., the IoU with ground truth is large than 0.7) from the positive window, as follows,
Lp = − ∑
p+i ∈W+ log(si)−
∑
p−i ∈W− log(1− si) (8)
where p+i is the positive proposal from the positive window W+ and p−i is each proposal from the negative window W−, si is the corresponding proposal score. And the overall contrastive loss is Lcon = Lp. Overall Loss. In total, our training loss (L) consists of three parts: (1) original training loss of the base model; (2) contrastive loss to discriminate the negative window versus the positive window; (3) adapter loss (shown in Eq. 4) to tune visual representations from general pre-training to the current downstream task:
L = Lori + λcon × Lcon + λadapt × Ladapt (9)
where λcon and λadapt are loss weight hyperparameters to control the loss value. During training, we set λcon to 1 and λadapt to 0.2.

heading: B More Qualitative Examples
text: We show four success cases in Figure 6 and four failure cases of our model CONE in Figure 7.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section 6 page 9
3 A2. Did you discuss any potential risks of your work? section 6 page 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? experiments section 5
3 B1. Did you cite the creators of artifacts you used? experiments section 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Statement
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. table 1
C 3 Did you run computational experiments? experiments section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? experiments section 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, it is worthwhile to explore how to incorporate the coarse-to-fine alignment mechanism with proposal-free methods to enhance the generalization ability of CONE. Furthermore, CONE falls short on the groundtruth moment case whose duration is longer than the adopted video window duration. To ease this issue, future work can explore adaptive-duration window slicing to ensure complete containment of scenes and events within windows or some rulebased proposal merging techniques