End-to-end Multi-modal Video Temporal Grounding
Authors: Yi-Wen Chen
Yi-Hsuan Tsai
Ming-Hsuan Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computer Vision
Vision and Language
Video Temporal Grounding
Multi-modal Learning
Transformer
Contrastive Learning

Abstract: We address the problem of text-guided video temporal grounding, which aims to identify the time interval of a certain event based on a natural language description. Different from most existing methods that only consider RGB images as visual features, we propose a multi-modal framework to extract complementary information from videos. Specifically, we adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. While RGB images provide abundant visual cues of certain events, the performance may be affected by background clutters. Therefore, we use optical flow to focus on large motion and depth maps to infer the scene configuration when the action is related to objects recognizable with their shapes. To integrate the three modalities more effectively and enable inter-modal learning, we design a dynamic fusion scheme with transformers to model the interactions between modalities. Furthermore, we apply intra-modal self-supervised learning to enhance feature representations across videos for each modality, which also facilitates multi-modal learning. We conduct extensive experiments on the Charades-STA and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.

End-to-end Multi-modal Video Temporal Grounding
Yi-Wen Chen' Universityy of California, Merced
Yi-Hsuan Tsai
Ming-
Ming-Hsuan Hsuan Yang!.3.
Phiar
3 Yonsei University
*Google Research
Abstract
We address the problem text-guided video temporal grounding, which aims to identify the time interval of certain event based on natural language description. Different from most existing methods that only consider RGB images as visual fea- tures, propose multi-modal framework to extract complementary information from videos. Specifically, we adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. While RGB images provide abundant visual cues of certain events, the performance may be affected by background clutters. Therefore, we use optical flow to focus on large motion and depth maps to infer the scene configuration when the action is related to objects recognizable with their shapes. To integrate the three modalities more effectively and enable inter-modal learning, we design a dynamic fusion scheme with transformers to model the interactions between modalities. Furthermore, we apply intra-m modal self- seff-supervised learning to enhance feature representations across videos for each modality, which also facilitates multi-mode learning. We conduct extensive exper- iments on the Charades-STi and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.
Introduction
With the rapid growth of video data in our daily lives, video understanding has become an ever increasingly important task computer vision. Research involving other modalities such as text and speech has also drawn much attention in recent years, e.g. video captioning [17, 23], and video question answering [18, 16]. In this paper, we focus on text-guided vidco temporal grounding, which aims to localize the starting and ending time of segment corresponding to text query. It one the most effective approaches to understand video contents, and applicable to numerous tasks, such video retrieval, video editing and human-con computc interaction. This problem is considerably challenging as requires accurate recognition of objects, scenes and actions, as well as joint comprehension of video and language.
Existing methods [34, 26, 33, 22] usually consider only RGB images as visual cues, which are less effective for recognizing objects and actions in videos with complex backgrounds. To understand the video contents more holistically, we propose multi-modal framework to learn complementary visual features from RGB images, optical flow and depth maps. RGB images provide abundant visual information, which is essential for visual recognition. However, existing methods based on appearance alone are likely to be less effective for complex scenes with cluttered backgrounds. For example, since the query text descriptions usually involve moving objects such as "Closing a door" or "Throwing pillow". using optical flow input is able to identify such actions with large motion. On the other hand, depth is another cue that is invariant to color and lighting, and often used complement the RGB input in object detection and semantic segmentation. In our task, depth information helps the proposed model recognize actions involving objects with distinct shapes as the context. For example, actions such "Sitting in bed" or "Working at table" are not easily recognized by optical flow due to small motion, but depth can provide structural information to assist the learning process. We also note that, our goal design end-to-end multi-modal framework
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

for video grounding by directly utilizing low-level cues such as optical flow and depth, while other alternatives based on object detector or semantic segmentation is out of the scope of this work.
To leverage multi-modal cues, one straightforward way is to construct a multi-stream model that takes individual modality the input in each stream, and then averages the multi-stream output predictions to obtain final results. However, we find that this scheme is less effective due to the lack of communication across different modalities, e.g., using depth cues alone without considering RGB features is not sufficient to learn the semantic information as the appearance cue does. To tackle this issue, we propose a multi-modal framework with 1) an inter-modal module that learns cross- moddl features, and 2) an intra-modal module to sear- feature representations across videos.
For inter-modal learning, we design fusion scheme with co-attentional transformers [20] to dynami- cally fuse features from different modalities. One motivation is that, different videos may require to adopt a different combination of modalities, e-g. "Working a table" would require more appearance and depth information, while optical flow is more important for "Throwing pillow". To enhance feature representations for each modality and thereby improve multi-modal learning, we introduce an intra-modal module via self-supervised contrastive learning [7. 15]. The goal is to ensure the feature consistency across video clips when they contain the same action. For example, with the same action "Eating", it may happen at different locations with completely different backgrounds and contexts, or with different text descriptions that "eats" different food. With our intra- antra-modal learning, it enforces features close to each other when they describe the same action and learn features that are invariant to other distracted factors across videos, and thus it can improve our multi- modal learning paradigm.
We conduct extensive experiments on the Charades-STAX [10] and ActivityNet Captions [17] datasets to demonstrate the effectiveness of our multi-modal learning framework for video temporal grounding using eD),,hh, (R)GB, and optical (F)low with the (T))xt as the query, and name our method as DRFT. First, we present the complementary property of multi-modality and the improved performance over the single-modality models. Second, we validate the individual contributions of our proposed components, i.e., inter- and intra-modal modules, that facilitate multi-modal learning. Finally, we show state-of- oatte-of-thearrr the-art art performance for video temporal grounding against existing methods.
The main contributions of this work are summarized as follows: 1) We propose
framework for text-guided video temporal grounding by extracting complementary information from RGB, optical flow and depth features. 2) We design dynamic fusion mechanism across modalities via co-attentional transformer to effectively learn moder-modal features. 3) We apply self-supervised contrastive learning across videos for each modality to enhance intra-modal feature representations that are invariant to distracted factors with respect to actions.
2 Related Work
Text-Guided Video Temporal Grounding. Given a video and a natural language query, text- guided video temporal grounding aims to predict the starting and ending time of the video clip that best matches the query sentence. Existing methods for this task can be categorized into two groups, i.e., two-stage and one-stage schemes (see Figure (aa)())). Most two-stage approaches adopt a propose-and-rand pipeline, where they first generate clip proposals and then rank the proposals based on their similarities with the query sentence. Early two-stage methods [10, 14] obtain pro- posals by scanning the whole video with sliding windows. Since the sliding window mechanism is computationally expensive and usually produces many redundant proposals, numerous methods are subsequently proposed to improve the efficiency and effectiveness of proposal generation. The TGN model [2] performs frame-by-we word interactions and localize the proposals in one single pass. Other approaches focus on reducing redundant proposals by generating query-guidee proposals [31] or semantic activity proposals [3]. The MAN method [34] models the temporal relationships between proposals using a graph architecture to improve the quality of proposals. To alleviate the computation of observing the whole video, reinforcement learning [13, 30] is utilized to guide the intelligent agent to glance over the video in a discontinuous way. While the two-stage methods achieve promising results, the computational cost is high for comparing all proposal-que -uerry pairs, and the performance is largely limited by the quality of proposal generation.
To overcome the issues of two-stage methods, some recent approaches adopt a one-stage pipeline to directly predict the temporal segment from the fusion of video and text features. Most of the one-stage approaches focus on the attention mechanisms interaction between modalities. For
2

---

Proposals
Inter-Modal Feature Learning
Intra-Modal Feature Learning
D
Transformer
close
R
(a) Two-stage approach
Transformer
F
RGB
Depth
Fused feature
Intra-modal module
T
Text
Flow
Prediction
(b) One One-stage approach
(c) DRFT
Figure I: Overview of the proposed algorithm. Previous methods focus (a) two-stage schemes relying on proposals, which is computational expensive and sensitive to the quality of proposals; (b) one-stage schemes predicting the temporal segment from the fusion of RGB and text features. In contrast, our DRFT method adopts the end-to-end one-stage scheme with multi-modal learning using RGB, depth, and optical flow, such that the model can better capture action-related features via our proposed inter-modal and intra-modal learning modules. Specifically, the inter-modal module uses co-attentional transformer to extract features across RGB and the other modality (either depth or optical flow), while the intra-modal module aims to perform cross- video feature learning, in which features within each modal are closer when they share the same action category.
example, the ABLR method [32] predicts the temporal coordinates using co-attention based location regression algorithm. The ExCL mechanism [11] exploits the cross-modal interactions between video and text, and the PITML-GA model [26] improves the performance by introducing the query- guided dynamic filter. Moreover, the DRN scheme [33] leverages dense supervision from the sparse annotations facilitate the training process. Recently, the LGI model [22] decomposes the query sentence into multiple semantic phrases and conducts local and global interactions between the video and text features. our framework, we adopt LGI as the baseline that uses the hierarchical video-text interaction. However, different from LGI that only considers RGB frames as input, we take RGB, optical flow and depth as input, and design the inter- she-moodlltty learning technique to learn complementary information from the video. Furthermore, apply contrastive learning across videos to enhance the feature representations each modality, which helps the learning of the whole model (see Figure I(c)).
Multi-Modal Learning. As typical event or actions can be described by signals from multiple modalities, understanding the correlation between different modalities is crucial solve problems more comprehensively. Research on joint vision and language learning [6. 23, 16, 7] has gained much attention recent years since natural language an intuitive way for human communication. Recent studies [24, 8] based on the transformer [29] have shown great success in self-supervised learning and transfer learning for natural language tasks. The transformer based BERT model [8] has also been widely used to learn joint representations for vision and language. These methods [20, 21, 35, 19, 5, 9] aim to learn generic representations from large amount of image-text pairs in a self-supervised manner, and then fine-tune the model for downstream vision and language tasks. The ViLBERT scheme [20] extracts features from image and text using two parallel BERT-style models, and then connects the two streams with the co-attentional transformer layers. In this work, we focus on the video temporal grounding task guided by texts, while introducing multi-modality to improve model learning, which is not studied before. For fusing the co-attentional transformer layers [20] in our framework and design an approach by fusing the RGB features with optical flow and depth features respectively.
information, leverage the
3 Proposed Framework
In this work, we address the problem of text-guided video temporal grounding using multi-modal framework, The pipeline of the proposed framework is illustrated in Figure 2. Given an input video V {V6) with I frames and query sentence Q (Q with N words, aim to localize

---

Inter-Modal
Intra-Modal Feature Learning
Feature Learning
FC
close
Video
Co-TRM
M
Depth
Fitt
LGI
FC
far
Co-TRM
RGB
E,-
LGI
Share we ghts
REG
P F,1
FC
Co-TRM
Flow
By
LGI
FC
Co-TRM
Query:
"A person pours cup of coffee"
E
Figure 2: Pipeline of the proposed DRFT framework, Given an input video, our goal is to predict the video segment te] that contains the action in the guided query text. To this end, we first feed multi-modal information, i.e., depth, RGB, and optical flow to individual encoders, Ed, Er, Ey. and extract their features. Then we use the local-global interaction modules (LGI) to attend the text features via the text encoder F, on individual modalities, and obtain the multi-modal features Md. M, and yy for depth, RGB and flow respectively. To fuse multi-modal features effectively, we propose an inter-modal module that uses the co-attentional transformers (Co-TRM) to learn features between RGB and the other modality (either depth or optical flow), and then dynamically fuses the multi-modal features, i.e., the @ and @ operations with FC layers to predict the weights. To enhance feature representations within each module, we incorporate an intra-modal module that enforces cross-video features in each modality be close each other when they contain the same action, otherwise they should be apart from each other.
the starting and ending time [/ te] of the event corresponding to the query. To this end, design multi-modal framework to learn complementary visual information from RGB images, optical flow and depth maps. From the input video, we first compute the depth map of each frame and the optical flow of each pair of consecutive frames. We then apply the visual encoders Ed. E,, Ey to extract features from the depth, RGB and flow inputs. A textual encoder F, utilized to extract the feature of the query sentence Q. The local-global interaction modules (LGI) then incorporate the textual feature into each visual modality, and generate the multi-modal features Ma. M, and M, for depth, RGB and flow respectively.
To effectively integrate the features from different modalities and enable inter-modal feature learning, we propose dynamic fusion scheme with transformers to model the interaction between modalities. The feature after integration is then fed into a regression module (REG) to predict the starting and ending time [t., te] of the target video segment. To enhance the feature representations in each modality, we introduce an intra-modal learning module that conducts self-supervised contrastive learning across videos. The intra-modal learning is applied on the multi-modal features Ma. M, and My separately to enforce features of video segments containing the same action to be close to each other, and those from different action categories to be far apart.
3.1 Inter-Modal Feature Learning
Videos contain rich information in both spatial and temporal dimensions. To learn information more comprehensively, in addition to the RGB modality, we also consider optical flow that captures motion, and depth feature that represents image structure. An intuitive way to combine the three modalities is to utilize a multi-stream model and directly average the outputs of individual streams. However, since the importance of each modality is not the same in different situations, directly averaging them may downweigh the importance of a specific modality and degrade the performance. In Table 1, we present the results of two-stream (RGB and flow) and three-stream (RGB, flow and depth) baseline models, where the outputs from different modalities are averaged before the final output layer. Compared to the single-stream (RGB) baseline model, the multi-stream models do not improve the performance, which shows that it is not intuitive to learn complementary information from multi- modll-lmddl features.

---

Table I: Performance of baseline methods. For the single modality, we use RGB only. while using RGB and flow as the two modalities. More baselines results are provided in the supplementary material.
Charades-STA
Activity Captions
Method
@.0.3
R@0.5
R@0.7
mloU
R@0.3
R@0.5
R@0.7
mIoU
Single-stream baseline
72.14
59.16
35,31
50.94
58.47
41.35
23.60
41.38
Two-stream baseline
71.57
58.35
35.72
50.34
57.78
40.51
22.66
40.70
Three-stream baseline
71.13
57.39
34.69
48.21
56.45
38.63
22.05
39.86
Such cases may happen frequently in certain actions. For example, flow features would not help much for "Sitting in a bed" but would help more for "Closing a door". Therefore, having a dynamic mechanism is critical for multi-modal fusion.
Co-attentional Feature Fusion. The ensuing question becomes how to learn effective features across modalities and also fuse them dynamically. First, we observe that, although depth and flow modalities are effective some situations, they alone are not able capture the semantic information, which is crucial for video-text understanding. Thereby, we design a co-attentional scheme to allow joint feature learning between RGB and another modality (either depth or flow).
Inspired by the co-attentional transformer layer [20] that consist of multi-headed attention blocks, where it takes paired feature as the input (e.g. Ma and M,) and forms three matrices, Q, K, and V that represent queries, keys, and values (also see Figure 2). In this way, the multi-headed attention block for each modality takes the keys and values from the other modality as the input, and thus outputs the attention-poole features conditioned on the other modality. For instance, we consider the pair of Ma- = {m² .... m and M,- {m-.... .... mm") features for depth and RGB, where T is the number of frames in video clip. The co-attentional transformer layer performs depth-conditioned RGB feature attention, as well as RGB-conditioned depth feature attention. Similarly, for flow and RGB features, M, = {m},. {m....., m²7)) } and M, {m+ ..., m }. we obtain another set of flow- conditioned RGB feature attention and RGB-conditioned flow feature attention. Note that, since RGB feature generally contains the most abundant information in the video and is used for both depth and flow attention, we adopt shared multi-headed attention block for RGB as shown in Figure 2.
Dynamic Feature Fusion. To effectively combine these output features from co-attentional trans- formers and perform the final prediction, we dynamically learn the weights for each multi-modal feature and linearly combine the four features using the weights (see Figure 2). Since the importance of each modality depends on the input data, we generate the weights by feeding each feature into a fully-com connecced (FC) layer, and normalize the weights make the sum equal to 1. By dynamically generating weights from the features, we are able to adapt the multi-modke fusion process according to the input video and the query text. The fused feature is then served as input of the regression module (REG) to predict the starting and ending time.
3.2 Intra-Modal Feature Learning
To facilitate the multi-modal training, we introduce an intra- modal feature learning module, which enhances feature representations within each modality by applying self-supervised contrastive learn ing. Our motivation is that features in the same action category should be similar even if they are from different videos. this end, for each input video V. we randomly sample positive videos V. that contain the same action category, and negative videos V_ with different action categories. We perform contrastive learning on the multi-modal features Ma. M, and M, E Rexx separately for each modality, where is the feature dimension and 7 is the number of frames. Since the multi-modal feature contains information from the whole vidco, we only consider features that contain the action by extracting the corresponding video segment. We then conduct average pooling in the temporal dimension and obtain a feature vector M E g. The contrastive loss La is formulated as:
Σ
MMYTN(M4)/+ M(M4//T
M+ER+
La = log Σ M M+EQ+
(I)
ch(M) M(M4)T
+ Σ
Ch(M) R(M
M M_EQ
5

---

where Q- and Q- are the sets of positive and negative samples, and T is the temperature parameter. Following the SimCLR approach [4], we use linear layer h(-) to project the feature M another embedding space where we apply contrastive loss. We accumulate the loss from each modality to be the final contrastive loss, namely Iad = (Ang (M.)) + 1.2 (ha( (eol((Aeooo((Ma))) + L' (by (AugPool(My))) for RGB, depth, and flow, respectively.
3.3 Model Training and Implementation Details
Overall Objective. The overall objective of the proposed method is composed of the supervised loss Lyrn for predicting temporal grounding that localizes the video segment and the self- sepe-sueeeeeeed contrastive loss Le for intra-mode learning in (I): L = Lyrn + Lal The supervised loss Lgrn the same as the loss defined in the LGI method [22], which includes:
1) Location regression loss Liveg = smoothr. (? (*) + smoothr. (8 - (1)) that calculates the L1 distance between the normalized ground truth time interval (f=,t% {²)) E [0, 1] and the predicted time
interval (t", t where sm is defined as 0.5x² if [x]] < 1 and |ar] - 0.5 otherwise.
log(O.)
2) Temporal attention guidance loss Liag =
for the temporal attention in the REG
6.
module, where o, is set to 1 if the i-th segment is located within the ground truth time interval and 0 otherwise.
3) Distinet query attention loss Ldga = ([AAA)) 2/ to enforce query attention weights to be distinct along different steps in the LGI module, where A N is the concatenated query attention weights across S steps, || .lle denotes Frobenius norm of a matrix, and X E [0, 1] controls the extent of overlap between query attention distributions. The supervised loss the sum of the three loss terms L grn Lreg + Ling + LAqo and we use the default setting in LGI [22], in which we refer readers to their paper for more details.
Implementation Details. We generate optical flow and depth maps using the RAFT [27] and MiDaS [25] method respectively. For the visual encoder Ed, E, and E,, we employ the 13D [1] and C3D [28] networks for Charades-STA and ActivityNet Captions datasets respectively. As for the textual encoder E,, we adopt a bi-directional LSTM, where the feature is obtained by concatenating the last hidden states in forward and backward directions. The LGI module in our framework contains the sequential query attention and local-global video-text interactions as in the LGI model [22]. The REG module generates temporal attention weights to aggregate the features and performs regression via an MI.P layer. The operations are defined similar to I.I [22]. The feature dimension e is set to 512. In the contrastive loss (I), the temperature parameter 7 set to 0. The projection head h(-) is a 2-layer MLP that project the feature to a 5 212-dimeniional latent space. We implement the proposed model in PyTorch with the Adam optimizer and fixed learning rate of 4 x 10-4 The source code and models are available at https: / / thub. com/ hom//enz1 weez116/DRBB 16/DRFT.
4 Experimental Results
4.1 Datasets and Evaluation Metric
We evaluate the proposed DRFT method against the state-of-the-art approaches on two benchmark datasets, i.e., Charades- [10] and ActivityNet Captions [17].
Charades-STA. It is built upon the Charades dataset for evaluating the video temporal grounding task. It contains 6,672 videos involving 16, 128 video-query pairs, where 12,408 pairs are used for training and 3,720 pairs for testing. The average length of the videos is 29.76 seconds. There 2.4 annotated moments with duration 8.2 seconds in each video.
ActivityNet Captions. It is originally constructed for dense video captioning from the ActivityNet dataset. The captions are used as queries in the video temporal grounding task. It consists of 20k YouTube videos with an average duration of 120 seconds. The videos are annotated with 200 activity categories, which is more diverse compared to the Charades-STA dataset. Each video contains 3.65 queries, where each query has an average length of 13.48 words. The dataset split into training, validation and testing set with a ratio of 2:1:1, resulting in 37, 1,,421, 17,505 and 17,031 video-query

---

Table 2: Comparisons with stafe-of-the-art methods. For the results on ActivityNet Captions, DRN [33] is evaluated on the two validation sets valı and val2 separately, while others are evaluated on the entire validation set. We also note that the singlemooda model is RGB only, while RGB and flow are used in the two-modality case. More results are presented the supplementary material.
Charades-STi -TA
ActivityNet Captions
Method
R @@0.3
R@0.5
R@0.7 mloU R@0.3
R@0.5
R@0.7
mloU
CTRL [10]
21.42
7.15
28.70
14.00
20.54
RWM [13]
36.70
36.90
MAN [34]
46.53
22.72
TripNet [12]
51.33
36.61
14.50
45.42
32.19
13.93
PITML-GA [26]
67.53
52.02
33.74
51.28
33.04
19.26
37.78
DRN [33]
53.09
31.75
42.49/45.45
22.25/24.36
LGI [22]
72.96
59.46
35.48
51.38
58.52
41.51
23.07
41.13
Single-stream DRFT
73.85
60.79
36.72
52.64
60.25
42.37
25.23
43.18
Two-stream DRFT
74.26
61.93
38.69
53.92
61.80
43.71
26.43
44.82
Three-stream streas DRFT
76.68
63.03
40.15
54.89
62.91
45.72
27.79
45.86
pairs respectively. Since the testing set not publicly available, we follow previous methods to evaluate the performance on the combination of the two validation sets, which are denoted as val and val2.
Following the typical evaluation setups [10, 22], we employ two metrics to assess the performance of video temporal grounding: 1) Recall at various thresholds of temporal Intersection over Union (R @IU)... . It measures the percentage of predictions that have IoU with the ground truth larger than the threshold. We adopt values (0.3, 0.5, 0.7) for the IoU threshold. 2) mean tloU (mloU). It is the average loU over all results.
4.2 Overall Performance
In Table 2, we evaluate our framework against state-of-art approaches, including two-stage methods that rely on propose-and-rand schemes [10, 13, 34] and one-stage methods that only consider RGB videos the input [12, 26, 33, 22]. First, compared to baseline LGI [22], results with single/two/three modalities are consistently better than theirs in all the evaluation metrics, which demonstrates the benefit of our intra-modal feature learning scheme and the inter-modal feature fusion mechanism. We also note that for our single-mode model, we use RGB as the input and only apply the intra-modal contrastive learning across videos, where already performs favorably against existing algorithms. More results of the amnll-strean models using other modalities are provided in the supplementary material.
Second, we show that with the increased modality used in our model (bottom group in Table 2), the performance on two benchmarks are consistently improved, which demonstrates the complementary property RGB, depth, and flow for video temporal grounding. Morcover, compared to the worse baseline results when adding more modalities without proposed modules in Table 1, we validate the importance of designing a proper scheme of exchanging and fusing the information across modalities. It is also worth mentioning that with more modalities involved in the model, our method achieves larger performance gains compared to the baseline, e.g., more than 5% improvement in all the metrics on two benchmarks.
4.3
Ablation Study
In Table 3, we present the ablation of individual components proposed in our framework.
Inter-modal Feature Fusion. To enhance the communication across modalities, we propose to first use co-attentional transformers to learn attentive features across RGB and another modality, and then use dynamic feature fusing scheme with learned weights combine different features. In the first four rows of the middle group in Table 3, we show the following properties in this work:
7

---

Table 3: Ablation study of the proposed DRFT algorithm. We present the importance of the proposed modules in our framework, namely 1) the inter-modal module that includes cross-modal feature learning via transformers and dynamic feature fusion scheme via learnable weights: 2) the modal feature learning method via cross-video contrastive feature learning for individual modalities.
Charades-STA
ActivityNet Captions
Method
@@0.3
R@0.
R @@0.7 mloU R@0.3
@@0.5 R@0.7 mloU
Three-stream baseline
71.13
57.39
34.69
48.21
56.45
38.63
22.05
39.86
DRFT w/o transformer
74.72
61.05
38.26
52.74
61.04
43.83
25.74
43,90
DRFT w/ flow-RGB, flow-depth
74.96
61.20
38.57
53.01
61.39
43.94
25.85
44.17
DRFT w/o Co-TRM weight sharing
76.24
62.61
39.60
54.47
62.59
45.28
27.36
45.31
DRFT w/o learnable weights
75.03
61.65
38.78
53.11
61.47
44.42
26.31
44.39
DRFT w/o contrastive loss
75.41
61.87
39.02
53.65
61.59
44.50
26.48
44.61
Three-stream DRFT
76.68
63.03
40.15
54.89
62.91
45.72
27.79
45.86
1) Using transformers effective for multi-modes feature learning. As shown the first row of the middle group, the performance drops without the co-attentional transformers for feature fusion.
2) RGB information is essential for the temporal grounding task, and thus we conduct co-altention between a) RGB-flow and b) RGB-depth. In the second row of the middle group, if using the flow modality as the common modality, i.e., flow-RGB and flow-depth, the performance is worse than our final model.
3) Since RGB features are used for both flow and depth attention, we adopt shared co-attention block for RGB as shown in Figure 2, where it can take RGB together with either the flow or depth cue as the input, and further enriches the attention mechanism. This design has not been considered in the prior work. In the third row of the middle group, without sharing the co-attentional module, the performance is worse than our final model.
4) The proposed dynamic fusion scheme via learnable weights is important to fuse features from different modalities. As shown the fourth row of the middle group, the performance drops significantly without learnable weights. Interestingly, learning dynamic weights to combine features almost equally important compared to feature learning via transformers. This indicates that even with the state-of-the-as feature attention module, it is still challenging to combine multi-modal features.
Intra-modal Feature Learning. In the last row of the middle group in Table 3, we show the benefit of having the intra-modal cross video feature learning. While multi-modu feature fusion already provides a strong baseline in our framework, improving feature representations in individual modalities is still critical for enhaneing the entire multi-modal learning paradigm, in which such observations are not widely studied yet.
Qualitative Results. In Figure 3, we show sample results on the Charades-STA and Activity Captions datasets, where the arrows indicate the starting and ending points of the corresponding grounded segment based on the query. Compared to the baseline method that only consider RGB features, the proposed DRFT approach is able to predict more accurate results by leveraging the multi- modal features from RGB, optical flow and depth. More results are presented in the supplementary material.
4.4 Analysis of Multi-modal Learning
To understand the complementary property of each modality, we analyze the video temporal grounding results of some example action categories. Figure 4 shows the performance of the single-strean baseline with RGB as input, single-stream DRFT models with RGB, flow or depth as input, and three-stream DRFT model respectively. The three plots contain categories where RGB, flow or depth performs better than the other two modalities. We first show that the single-stream DRFT model with contrastive learning improves from the single-stream baseline (red bars vs. orange bars). We then investigate the complementary property between the three modalities. For actions with smaller movement (e.g., "smiling") in the left group of Figure 4, models using RGB as input generally
8

---

Query: "People put the plate down on the table"
Ground Truth
Time
Single-Stream Baseline (LGI)
DRFT
Query: "Person drinks from a cup coffee"
Ground Truth
Time
Single-Stream Baseline (LGI)
DRFT
Query: "The little girl is walking a very large white dog"
Ground Truth
Time
Single-Stream m Baseline (LGI)
DRFT
Query: "The guy has his hand handles"
Frenary Bik
Ground Truth
Time
Single-Stream Baseline (LGI)
DRFT
Figure 3: Sample results on the Charades-STA (first and second rows) and ActivityNet Captions (third and fourth rows) datasets. We show that the prediction of the proposed DRFT model is accurate than the single-strean baseline (LGI [22]) with RGB input. The arrows indicate the starting and ending points of the grounded video segment based on the query.

---

HW Direwer caloge
Falm Thars Carigress
prddias O1F dayy
CMP Foet
Adial cleyary
140%
2200 categii
Figure 4: Category-wise results on the Charades-STA dataset. The three plots show categories where RGB, flow or depth perform better than the other modalities.
Table 4: Weights of dynamic fusion. The + means attention from one to another modality.
Action category
Flow -> RGB
RGB Flow Depth + RGB RGB -> Depth
Turning off a light
0.384
0.208
0.261
0.147
Someone is smiling
0.327
0.215
0.293
0.165
Holding phone/camera
0.353
0.220
0.240
0.187
Closing door
0.305
0.339
0.217
0.139
Throwing pillow
0.312
0.379
0.194
0.115
Throwing a bag somewhere
0.280
0.362
0.203
0.155
Sitting in a bed
0.205
0.185
0.236
0.374
Sitting in chair Working at a table
0.221
0.173
0.284
0.322
0.216
0.158
0.262
0.364
perform better. For actions with larger motion (e.g., "closing door" "throwing a pillow") in the middle group, flow provides more useful information (denoted as green bars). As for the actions with small motion but can be easily recognized by their structure (e.g., "sitting a bed" or "working at table") in the right group, depth is superior to the other two modalities (denoted as blue bars). With the complementary property between RGB, flow and depth, we can take advantage of each modality and further improve the performance in the three-stream DRFT model (denoted as purple bars).
To further analyze the impact of each modality, we provide the learned weights for dynamic fusion in the three-stream DRFT for these categories in Table 4, where the top, middle and bottom groups contain categories that RGB, flow and depth help the most respectively. Flow - RGB means flow- conditioned RGB features, etc. We observe that for actions with smaller movement (top group), the weights for RGB features are larger. For actions with larger motion (middle group). the weights for optical flow are larger. Regarding actions with small motion but can be casily recognized by their structure (bottom group), the weights for depth are larger. This shows that the model can exploit each modality based on the complementary property between RGB, flow and depth.
5 Conclusions
In this paper, we focus on the task of text-guided video temporal grounding. contrast to existing methods that consider only RGB images as visual features, we propose the DRFT model to learn complementary visual information from RGB, optical flow and depth modality. While RGB features provide abundant appearance information, alone are not effective for temporal grounding in videos with cluttered backgrounds. We therefore adopt optical flow to capture motion cues, and depth maps to estimate image structure. To combine the three modalities more effectively, we propose an inter-modal feature learning module, which performs co-attention between modalities using transformers, and dynamically fuses the multi-modal features based on the input data. To further enhance the multi-modal training, we incorporate an nntra-modal feature learning module that performs self-supervised contrastive learning within each modality. The contrastive loss enforces cross-video features to be close to each other when they contain the same action, and be far apart otherwise. We conduct extensive experiments on two benchmark datasets, demonstrating the effectiveness of the proposed modti framework with inter- and intr-mode feature learning.
show that representation models based these cues
10

---

Acknowledgements
This work is supported in part by NSF CAREER grant 1149783 and gifts from Snap well as eBay.
References
[1] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics datasct. In CVPR, 2017. 6
[2] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jic, and Tat-Seng Chua. Temporally grounding natural sentence in video. In EMNLP, 2018. 2
[3] Shaoxiang Chen and Yu-Gang Jiang. Semantic proposal for activity localization videos via sentence
query. In AAAI, 2019. 2
[4] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, 2020.
[5] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing
Liu. Uniter: Universal image-text representation lcarning. In ECCV, 2020. 3
[6] Yi-Wen Chen, Yi-Hsuan Tsai, Tiantian Wang, Yen- Yu Lin, and Ming-Hsuan Yang. Referring expression object segmentation with caption- consistency. In BMVC, 2019. 3
[7] Yi-Wen Chen, Yi-Hsuan Tsai, and Ming- Yang. Understanding synonymous referring expressions via contrastive features. arXiv preprint arXiv:2 /0156, 2021. 2,3
[8] Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. 2018. 3
[9] The Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial
training for vision-ande language representation learning, In NewrlPS, 2020. 3
[10] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language
query. In /CCV, 2017. 2, 6.
[11] Soham Ghosh, Anuva Agarwal. Zarana Parekh, and Alexander G. Hauptmann. ExCL: Extractive clip localization using natural language descriptions. In NAACL, 2019.
[12] Meera Hahn, Asim Kadav, James M. Rehg, and Hans Peter Graf. Tripping through time: localization of activities in videos. In BMVC, 2020.
Efficient
[13] Dongliang He, Xiang Zhao, Jizhou Huang. Fu Li, Xiao Liu, and Shilei Wen. Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos. In AAAI, 2019.
7.7
[14] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, 2017. 2
[15] Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang. Xiang Yu, Stan Sclaroff, Kate Saenko. Manmohan Chandraker. Learning cross-modal contrastive features for video domain adaptation. In ICCV, 2021, 2
[16] Junyeong Kim, Minuk Ma. Trung Pham, Kyungsu Kim, and Chang D. Yoo. Modality shifting attention network for multi-modal video question answering. In CVPR, 2020. 3,
[17] Ranjay Krishna, Kenji Hata, Frederie Ren, Li Fei-Fei, and Juan Carlos Niehles. Dense-captioning events
in videos. In ICCV, 2017. 1, 2, 6
[18] Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. TVQA: localized, compositional video question
answering. In FMNLP, 2018. 1
[19] Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong
Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In FCCV, 2020. 3
[20] Jiasen La, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolinguistic
representations for vision- anta--nd--aan lnd-uane tasks. NeurlPS, 2019. 2,
[21] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision
and language representation learning. In CVPR, 2020.
[22] Jonghwan Mun, Minsu Cho, and Bohyung Han. Local-global video-text interactions for temporal grounding.
In CVPR, 2020. 1, 6. 7.9
[23] Boxiao Pan, Haoye Cai, De-An Huang. Kuan-Hur Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In CVPR, 2020. 1,
[24] Matthew E. Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
[25] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. PAMI, 2020. 6 [26] Cristian Rodriguez, Edison Marrese- Marrese-Taylor, Fatemeh Sadat Salch, Hongdong LI, and Stephen Gould. Proposal- free temporal moment localization of a natural- language query in video using guided attention. In WACV, 2020. 3.
11

---

[27] Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow. In ECCV, 2020. 6 Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotem- poral features with 3d convolutional networks. In ICCV, 2015.
[28]
[29]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakoh Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention you need. NeurIPS, 2017. 3
[30] Weining Wang, Yan Huang, and Liang Wang. Language-driven temporal activity localization: A semantic matching reinforcement learning model. In CVPR, 2019.
[31] Huijuan Xu, Kun Hc, Bryan A. Plummer, Lconid Sigal, Stan Sclaroff, and Kate Saenko. Multilevel
language and vision integration for text-to-clip retrieval. In AAAI, 2019. 2
[32] Yitian Yuan, Tao Mei, and Wenwu Zhu. find where you talk: Temporal sentence localization in video
with attention hased location regression. In AAAI, 2019. 3
[33] Runhao Zeng, Haoming Xu, Wenbing Huang. Peihao Chen, Mingkui Tan, and Chuang Gan. Dense
regression network for video grounding. CVPR, 2020. 1, 3,7
[34] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang. and Larry S. Davis. Man: Moment alignment network for natural language moment retricval via iterative graph adjustment. In CVPR, 2019. 2,
[35] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In AAAI, 2020. 3
12


Reviews:
[{"review_id": "WdutDJokR5", "reviewer": "Reviewer_wvV3", "summary": "A method for the video temporal grounding in the multi-modal domain (RGB, flow, depth). To this end, the transformer-based co-attention (+ adaptive fusion) and contrastive learning techniques are developed.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed modules are straightforward use of existing techniques (transformers, contrastive learning). Although the co-attention scheme is developed by modifying the self-attention scheme of the transformer, it has already been tried in various multi-modal methods. Comprehensively, the novelty is weak. Also, enough analysis and studies are not contained, and several definitions/descriptions (loss term and compared method in analysis) are unclear. \n\nNot enough analysis and studies and several unclear definitions/descriptions, although there is room for the amount of pages.\n1) In table 1, does the single-stream DRFT means replacing the inter-modal feature learning module with self-attention? If not, the single-stream DRFT is just the method w/o any attention mechanism, and then it is not proper to verify the effect of the proposed co-attention. \n\n2) Unclear description for REG module. For example, how the start and end timing is obtained if there is multiple start-end in a video?\n\n3) why the definition/formulation of L_grn is not described?\n\n4) in line 205-206, the authors mentioned that their LGI module is similar to LGI model [19]. Then, is the proposed method built on top of [19] by adding the proposed modules?\n\n5) There is no analysis or studies for if the weight of dynamic fusion differs depending on videos or frames. Without showing this point, the reviewer thinks that the gain of dynamic fusion results from merely using more computational cost.\n\n6) Is there no previous multi-modal methods for this task? As the multi-modal baseline, the authors use the simple combination of independently learned uni-modal models. But, the lower performance of the baseline is not surprising at all. To show the effectiveness of the proposed multi-modal fusion, it is required to compare the proposed fusion with other (even basic) intermediate fusion schemes.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors don't address limitations", "review_text": "", "extra_scores": {"ethical_concerns": "no ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 day"}, "extra_text": {}}, {"review_id": "WP0QFHQBZdX", "reviewer": "Reviewer_9jTB", "summary": "This paper studies the problem of multi-modal video grounding. The authors exploit multiple modalities including RGB, depth, optical flow to extract complementary information for improving video grouding. A dynamic fusion scheme with transformer is proposed to better learn the interaction and integration of multiple modalities. Moreover, a self-supervised intrao-modal module is proposed to obtain better feature representation. The extensive experiments support that the proposed method achieves new state-of-the-art performance on multiple bechmark datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Satisfactory paper writing. Clear technical presentation.\n2. Good performances and thorough empirical studies.\n\nWeaknesses:\n1. My main concern is the novelty. All the proposed modules look natural but kind of straighforward to me. They indeed contribute to the performance, but there is no inspiring techniques or insightful technical conclusion. For example, co-attentional transformer layers are adopted from [17] to model the interaction among different modalities; contrastive learning is adopted to model the intra-modal feature learning. \n\nQuestions:\n1. Why actions like \"sitting in a bed\" require depth information? Can they be recognized by RGB information?\n2. In the intra-modal feature learning, given only the sentence query, how to judge whether two videos are from the same action category?\n\nRebuttal summary:\n1. I agree with other reviewers that depth information is interesting and properly leveraged for the multi-modality task. From non-technical aspect, it is a novel idea.\n2. The technical novelty might be limited but acceptable. I increase my rating to \"Marginally above acceptence threshold\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations provided. No more suggestions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "BwncHCg608x", "reviewer": "Reviewer_bBpz", "summary": "This paper addresses text-guided video grounding, which identifies the time interval of an event according to the query text. In contrast to prior work that only leverages RGB information, the authors propose DRFT to combine RGB, depth, and flow maps to boost the performance. To facilitate representation learning, the authors adopt a co-attentional transformer to fuse multi-modal information and contrastive learning to enhance the features across different videos. Experimental results shows that the proposed DRFT outperforms state-of-the-art on Charades-STA and ActivityNet.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written and easy to follow. Experimental results also justify the effectiveness of the proposed method. However, the authors are expected to justify their design choices in the proposed method rather than just give a combination of existing techniques. It would help further extension if the authors could elaborate more on the current design. More details can be found below.\n\n1.\tIn DRFT, the authors only consider two kinds of fusion, namely RGB-depth and RGB-flow. Is there any specific reason not to use other combinations, e.g., depth-flow?\n2.\tThe baseline models in Sec. 3.1 do not make sense to me. It seems to me that the most straightforward would be to concatenate different modalities and learn a 1x1 conv to fuse those features. Is there any reason that the authors adopt an average strategy? What’s the gap between the concatenation and other models mentioned in the paper?\n3.\tI am curious about the necessity of using a co-attentional transformer. There have been a few lightweight attention modules, such as CBAM [1]. Can the authors briefly elaborate on this design choice?\n4.\tAlthough the experiments suggest multi-modality does help, it is unclear how they help. Maybe the authors could provide visualization showing what kind of information helps which activities more, by showing the attention weights and the corresponding activities.\n5.\tThe reason why intra-model feature learning works remains unclear to me. It might be a novel point of view, but the authors are encouraged to provide more insights in the paper.\n\n[1] Woo, Sanghyun, et al. \"Cbam: Convolutional block attention module.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors are encouraged to discuss potential abuse and misuse.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "0H5vI2Q9hXV", "reviewer": "Reviewer_wDga", "summary": "Paper addresses the problem of text-guided video temporal grounding, which aims to localize the starting and ending time of a segment corresponding to a text query. The key contribution is integration of three modalities of data in this context: video, motion (flow), and depth; along with the textural query. To integrate the three modalities, paper proposes a dynamic fusion scheme with transformers, which takes the form of co-attention (similar to ViLBERT). Further, to improve the performance within each modality contrastive learning is applied to enhance the feature discriminability. In this contrastive learning formulation positive pairs come from instance of the same action class, while negative pairings are formed from videos coming from two different action classes. Competitive (state-of-the-art) performance is illustrated on Charades-STA and ActivityNet Captions benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the approach is intuitive and easy to understand and follow. Performance is also competitive and improves on state-of-the-art.  The technical novelty is somewhat increment, with components effectively borrowed from other recent works. That being said the choices are well motivated and work well together for the task at hand. Overall, I feel the novelty is sufficient for a poster publication in NeurIPS. \n\nA few minor comments\n\n- The supervised loss (L_{and}) is not defined. It should be defined in the paper, if for no other reason than completeness.\n- The extraction of video segment features (Lines 188-190) is unclear and should be clarified. \n- In addition, one of the more surprising aspects of the paper, for me, is that depth was helpful for the task. I am not aware of any other works that use estimated depth for video tasks. As such, I would be interested in seeing more analysis on the depth modality. For example, how well would depth features work on their own (as one stream DRFT)? How well would they perform in combination with RGB (as a two stream DRFT)? etc. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. I do not believe there are deep societal impacts to discuss for this work. ", "review_text": "", "extra_scores": {"ethical_concerns": "None. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach for multi-modal video temporal grounding. The paper received diverging review scores (4, 6, 6, 7),
with the positive reviewer championing the paper. All reviewers agree that the approach is somewhat incremental, but it introduces as one of the first the use of depth information. The rebuttal addressed a significant part of the concerns of the reviewers.

The recommendation is accept as a poster, with the expectation that the authors include the information provided in the rebuttal. 
Decision: Accept (Poster)