End-to-end Multi-modal Video Temporal Grounding

1 Introduction . With the rapid growth of video data in our daily lives , video understanding has become an ever increasingly important task in computer vision . Research involving other modalities such as text and speech has also drawn much attention in recent years , e.g. , video captioning [ 17 , 23 ] , and video question answering [ 18 , 16 ] . In this paper , we focus on text-guided video temporal grounding , which aims to localize the starting and ending time of a segment corresponding to a text query . It is one of the most effective approaches to understand video contents , and applicable to numerous tasks , such as video retrieval , video editing and human-computer interaction . This problem is considerably challenging as it requires accurate recognition of objects , scenes and actions , as well as joint comprehension of video and language . Existing methods [ 34 , 26 , 33 , 22 ] usually consider only RGB images as visual cues , which are less effective for recognizing objects and actions in videos with complex backgrounds . To understand the video contents more holistically , we propose a multi-modal framework to learn complementary visual features from RGB images , optical flow and depth maps . RGB images provide abundant visual information , which is essential for visual recognition . However , existing methods based on appearance alone are likely to be less effective for complex scenes with cluttered backgrounds . For example , since the query text descriptions usually involve moving objects such as “ Closing a door ” or “ Throwing a pillow ” , using optical flow as input is able to identify such actions with large motion . On the other hand , depth is another cue that is invariant to color and lighting , and is often used to complement the RGB input in object detection and semantic segmentation . In our task , depth information helps the proposed model recognize actions involving objects with distinct shapes as the context . For example , actions such as “ Sitting in a bed ” or “ Working at a table ” are not easily recognized by optical flow due to small motion , but depth can provide structural information to assist the learning process . We also note that , our goal is to design an end-to-end multi-modal framework 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for video grounding by directly utilizing low-level cues such as optical flow and depth , while other alternatives based on object detector or semantic segmentation is out of the scope of this work . To leverage multi-modal cues , one straightforward way is to construct a multi-stream model that takes individual modality as the input in each stream , and then averages the multi-stream output predictions to obtain final results . However , we find that this scheme is less effective due to the lack of communication across different modalities , e.g. , using depth cues alone without considering RGB features is not sufficient to learn the semantic information as the appearance cue does . To tackle this issue , we propose a multi-modal framework with 1 ) an inter-modal module that learns cross-modal features , and 2 ) an intra-modal module to self-learn feature representations across videos . For inter-modal learning , we design a fusion scheme with co-attentional transformers [ 20 ] to dynamically fuse features from different modalities . One motivation is that , different videos may require to adopt a different combination of modalities , e.g. , “ Working at a table ” would require more appearance and depth information , while optical flow is more important for “ Throwing a pillow ” . To enhance feature representations for each modality and thereby improve multi-modal learning , we introduce an intra-modal module via self-supervised contrastive learning [ 7 , 15 ] . The goal is to ensure the feature consistency across video clips when they contain the same action . For example , with the same action “ Eating ” , it may happen at different locations with completely different backgrounds and contexts , or with different text descriptions that “ eats ” different food . With our intra-modal learning , it enforces features close to each other when they describe the same action and learn features that are invariant to other distracted factors across videos , and thus it can improve our multi-modal learning paradigm . We conduct extensive experiments on the Charades-STA [ 10 ] and ActivityNet Captions [ 17 ] datasets to demonstrate the effectiveness of our multi-modal learning framework for video temporal grounding using ( D ) epth , ( R ) GB , and optical ( F ) low with the ( T ) ext as the query , and name our method as DRFT . First , we present the complementary property of multi-modality and the improved performance over the single-modality models . Second , we validate the individual contributions of our proposed components , i.e. , inter- and intra-modal modules , that facilitate multi-modal learning . Finally , we show state-of-the-art performance for video temporal grounding against existing methods . The main contributions of this work are summarized as follows : 1 ) We propose a multi-modal framework for text-guided video temporal grounding by extracting complementary information from RGB , optical flow and depth features . 2 ) We design a dynamic fusion mechanism across modalities via co-attentional transformer to effectively learn inter-modal features . 3 ) We apply self-supervised contrastive learning across videos for each modality to enhance intra-modal feature representations that are invariant to distracted factors with respect to actions . 2 Related Work . Text-Guided Video Temporal Grounding . Given a video and a natural language query , textguided video temporal grounding aims to predict the starting and ending time of the video clip that best matches the query sentence . Existing methods for this task can be categorized into two groups , i.e. , two-stage and one-stage schemes ( see Figure 1 ( a ) ( b ) ) . Most two-stage approaches adopt a propose-and-rank pipeline , where they first generate clip proposals and then rank the proposals based on their similarities with the query sentence . Early two-stage methods [ 10 , 14 ] obtain proposals by scanning the whole video with sliding windows . Since the sliding window mechanism is computationally expensive and usually produces many redundant proposals , numerous methods are subsequently proposed to improve the efficiency and effectiveness of proposal generation . The TGN model [ 2 ] performs frame-by-word interactions and localize the proposals in one single pass . Other approaches focus on reducing redundant proposals by generating query-guided proposals [ 31 ] or semantic activity proposals [ 3 ] . The MAN method [ 34 ] models the temporal relationships between proposals using a graph architecture to improve the quality of proposals . To alleviate the computation of observing the whole video , reinforcement learning [ 13 , 30 ] is utilized to guide the intelligent agent to glance over the video in a discontinuous way . While the two-stage methods achieve promising results , the computational cost is high for comparing all proposal-query pairs , and the performance is largely limited by the quality of proposal generation . To overcome the issues of two-stage methods , some recent approaches adopt a one-stage pipeline to directly predict the temporal segment from the fusion of video and text features . Most of the one-stage approaches focus on the attention mechanisms or interaction between modalities . For example , the ABLR method [ 32 ] predicts the temporal coordinates using a co-attention based location regression algorithm . The ExCL mechanism [ 11 ] exploits the cross-modal interactions between video and text , and the PfTML-GA model [ 26 ] improves the performance by introducing the queryguided dynamic filter . Moreover , the DRN scheme [ 33 ] leverages dense supervision from the sparse annotations to facilitate the training process . Recently , the LGI model [ 22 ] decomposes the query sentence into multiple semantic phrases and conducts local and global interactions between the video and text features . In our framework , we adopt LGI as the baseline that uses the hierarchical video-text interaction . However , different from LGI that only considers RGB frames as input , we take RGB , optical flow and depth as input , and design the inter-modality learning technique to learn complementary information from the video . Furthermore , we apply contrastive learning across videos to enhance the feature representations in each modality , which helps the learning of the whole model ( see Figure 1 ( c ) ) . Multi-Modal Learning . As typical event or actions can be described by signals from multiple modalities , understanding the correlation between different modalities is crucial to solve problems more comprehensively . Research on joint vision and language learning [ 6 , 23 , 16 , 7 ] has gained much attention in recent years since natural language is an intuitive way for human communication . Recent studies [ 24 , 8 ] based on the transformer [ 29 ] have shown great success in self-supervised learning and transfer learning for natural language tasks . The transformer-based BERT model [ 8 ] has also been widely used to learn joint representations for vision and language . These methods [ 20 , 21 , 35 , 19 , 5 , 9 ] aim to learn generic representations from a large amount of image-text pairs in a self-supervised manner , and then fine-tune the model for downstream vision and language tasks . The ViLBERT scheme [ 20 ] extracts features from image and text using two parallel BERT-style models , and then connects the two streams with the co-attentional transformer layers . In this work , we focus on the video temporal grounding task guided by texts , while introducing multi-modality to improve model learning , which is not studied before . For fusing the multi-modal information , we leverage the co-attentional transformer layers [ 20 ] in our framework and design an approach by fusing the RGB features with optical flow and depth features respectively . 3 Proposed Framework . In this work , we address the problem of text-guided video temporal grounding using a multi-modal framework . The pipeline of the proposed framework is illustrated in Figure 2 . Given an input video V = { Vt } Tt=1 with T frames and a query sentence Q = { Qi } Ni=1 with N words , we aim to localize the starting and ending time [ ts , te ] of the event corresponding to the query . To this end , we design a multi-modal framework to learn complementary visual information from RGB images , optical flow and depth maps . From the input video , we first compute the depth map of each frame and the optical flow of each pair of consecutive frames . We then apply the visual encoders Ed , Er , Ef to extract features from the depth , RGB and flow inputs . A textual encoder Et is utilized to extract the feature of the query sentence Q . The local-global interaction modules ( LGI ) then incorporate the textual feature into each visual modality , and generate the multi-modal features Md , Mr and Mf for depth , RGB and flow respectively . To effectively integrate the features from different modalities and enable inter-modal feature learning , we propose a dynamic fusion scheme with transformers to model the interaction between modalities . The feature after integration is then fed into a regression module ( REG ) to predict the starting and ending time [ ts , te ] of the target video segment . To enhance the feature representations in each modality , we introduce an intra-modal learning module that conducts self-supervised contrastive learning across videos . The intra-modal learning is applied on the multi-modal features Md , Mr and Mf separately to enforce features of video segments containing the same action to be close to each other , and those from different action categories to be far apart .

Summary: Paper addresses the problem of text-guided video temporal grounding, which aims to localize the starting and ending time of a segment corresponding to a text query. The key contribution is integration of three modalities of data in this context: video, motion (flow), and depth; along with the textural query. To integrate the three modalities, paper proposes a dynamic fusion scheme with transformers, which takes the form of co-attention (similar to ViLBERT). Further, to improve the performance within each modality contrastive learning is applied to enhance the feature discriminability. In this contrastive learning formulation positive pairs come from instance of the same action class, while negative pairings are formed from videos coming from two different action classes. Competitive (state-of-the-art) performance is illustrated on Charades-STA and ActivityNet Captions benchmark datasets.