End-to-end Multi-modal Video Temporal Grounding

1 Introduction . With the rapid growth of video data in our daily lives , video understanding has become an ever increasingly important task in computer vision . Research involving other modalities such as text and speech has also drawn much attention in recent years , e.g. , video captioning [ 17 , 23 ] , and video question answering [ 18 , 16 ] . In this paper , we focus on text-guided video temporal grounding , which aims to localize the starting and ending time of a segment corresponding to a text query . It is one of the most effective approaches to understand video contents , and applicable to numerous tasks , such as video retrieval , video editing and human-computer interaction . This problem is considerably challenging as it requires accurate recognition of objects , scenes and actions , as well as joint comprehension of video and language . Existing methods [ 34 , 26 , 33 , 22 ] usually consider only RGB images as visual cues , which are less effective for recognizing objects and actions in videos with complex backgrounds . To understand the video contents more holistically , we propose a multi-modal framework to learn complementary visual features from RGB images , optical flow and depth maps . RGB images provide abundant visual information , which is essential for visual recognition . However , existing methods based on appearance alone are likely to be less effective for complex scenes with cluttered backgrounds . For example , since the query text descriptions usually involve moving objects such as “ Closing a door ” or “ Throwing a pillow ” , using optical flow as input is able to identify such actions with large motion . On the other hand , depth is another cue that is invariant to color and lighting , and is often used to complement the RGB input in object detection and semantic segmentation . In our task , depth information helps the proposed model recognize actions involving objects with distinct shapes as the context . For example , actions such as “ Sitting in a bed ” or “ Working at a table ” are not easily recognized by optical flow due to small motion , but depth can provide structural information to assist the learning process . We also note that , our goal is to design an end-to-end multi-modal framework 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for video grounding by directly utilizing low-level cues such as optical flow and depth , while other alternatives based on object detector or semantic segmentation is out of the scope of this work . To leverage multi-modal cues , one straightforward way is to construct a multi-stream model that takes individual modality as the input in each stream , and then averages the multi-stream output predictions to obtain final results . However , we find that this scheme is less effective due to the lack of communication across different modalities , e.g. , using depth cues alone without considering RGB features is not sufficient to learn the semantic information as the appearance cue does . To tackle this issue , we propose a multi-modal framework with 1 ) an inter-modal module that learns cross-modal features , and 2 ) an intra-modal module to self-learn feature representations across videos . For inter-modal learning , we design a fusion scheme with co-attentional transformers [ 20 ] to dynamically fuse features from different modalities . One motivation is that , different videos may require to adopt a different combination of modalities , e.g. , “ Working at a table ” would require more appearance and depth information , while optical flow is more important for “ Throwing a pillow ” . To enhance feature representations for each modality and thereby improve multi-modal learning , we introduce an intra-modal module via self-supervised contrastive learning [ 7 , 15 ] . The goal is to ensure the feature consistency across video clips when they contain the same action . For example , with the same action “ Eating ” , it may happen at different locations with completely different backgrounds and contexts , or with different text descriptions that “ eats ” different food . With our intra-modal learning , it enforces features close to each other when they describe the same action and learn features that are invariant to other distracted factors across videos , and thus it can improve our multi-modal learning paradigm . We conduct extensive experiments on the Charades-STA [ 10 ] and ActivityNet Captions [ 17 ] datasets to demonstrate the effectiveness of our multi-modal learning framework for video temporal grounding using ( D ) epth , ( R ) GB , and optical ( F ) low with the ( T ) ext as the query , and name our method as DRFT . First , we present the complementary property of multi-modality and the improved performance over the single-modality models . Second , we validate the individual contributions of our proposed components , i.e. , inter- and intra-modal modules , that facilitate multi-modal learning . Finally , we show state-of-the-art performance for video temporal grounding against existing methods . The main contributions of this work are summarized as follows : 1 ) We propose a multi-modal framework for text-guided video temporal grounding by extracting complementary information from RGB , optical flow and depth features . 2 ) We design a dynamic fusion mechanism across modalities via co-attentional transformer to effectively learn inter-modal features . 3 ) We apply self-supervised contrastive learning across videos for each modality to enhance intra-modal feature representations that are invariant to distracted factors with respect to actions . 2 Related Work . Text-Guided Video Temporal Grounding . Given a video and a natural language query , textguided video temporal grounding aims to predict the starting and ending time of the video clip that best matches the query sentence . Existing methods for this task can be categorized into two groups , i.e. , two-stage and one-stage schemes ( see Figure 1 ( a ) ( b ) ) . Most two-stage approaches adopt a propose-and-rank pipeline , where they first generate clip proposals and then rank the proposals based on their similarities with the query sentence . Early two-stage methods [ 10 , 14 ] obtain proposals by scanning the whole video with sliding windows . Since the sliding window mechanism is computationally expensive and usually produces many redundant proposals , numerous methods are subsequently proposed to improve the efficiency and effectiveness of proposal generation . The TGN model [ 2 ] performs frame-by-word interactions and localize the proposals in one single pass . Other approaches focus on reducing redundant proposals by generating query-guided proposals [ 31 ] or semantic activity proposals [ 3 ] . The MAN method [ 34 ] models the temporal relationships between proposals using a graph architecture to improve the quality of proposals . To alleviate the computation of observing the whole video , reinforcement learning [ 13 , 30 ] is utilized to guide the intelligent agent to glance over the video in a discontinuous way . While the two-stage methods achieve promising results , the computational cost is high for comparing all proposal-query pairs , and the performance is largely limited by the quality of proposal generation . To overcome the issues of two-stage methods , some recent approaches adopt a one-stage pipeline to directly predict the temporal segment from the fusion of video and text features . Most of the one-stage approaches focus on the attention mechanisms or interaction between modalities . For example , the ABLR method [ 32 ] predicts the temporal coordinates using a co-attention based location regression algorithm . The ExCL mechanism [ 11 ] exploits the cross-modal interactions between video and text , and the PfTML-GA model [ 26 ] improves the performance by introducing the queryguided dynamic filter . Moreover , the DRN scheme [ 33 ] leverages dense supervision from the sparse annotations to facilitate the training process . Recently , the LGI model [ 22 ] decomposes the query sentence into multiple semantic phrases and conducts local and global interactions between the video and text features . In our framework , we adopt LGI as the baseline that uses the hierarchical video-text interaction . However , different from LGI that only considers RGB frames as input , we take RGB , optical flow and depth as input , and design the inter-modality learning technique to learn complementary information from the video . Furthermore , we apply contrastive learning across videos to enhance the feature representations in each modality , which helps the learning of the whole model ( see Figure 1 ( c ) ) . Multi-Modal Learning . As typical event or actions can be described by signals from multiple modalities , understanding the correlation between different modalities is crucial to solve problems more comprehensively . Research on joint vision and language learning [ 6 , 23 , 16 , 7 ] has gained much attention in recent years since natural language is an intuitive way for human communication . Recent studies [ 24 , 8 ] based on the transformer [ 29 ] have shown great success in self-supervised learning and transfer learning for natural language tasks . The transformer-based BERT model [ 8 ] has also been widely used to learn joint representations for vision and language . These methods [ 20 , 21 , 35 , 19 , 5 , 9 ] aim to learn generic representations from a large amount of image-text pairs in a self-supervised manner , and then fine-tune the model for downstream vision and language tasks . The ViLBERT scheme [ 20 ] extracts features from image and text using two parallel BERT-style models , and then connects the two streams with the co-attentional transformer layers . In this work , we focus on the video temporal grounding task guided by texts , while introducing multi-modality to improve model learning , which is not studied before . For fusing the multi-modal information , we leverage the co-attentional transformer layers [ 20 ] in our framework and design an approach by fusing the RGB features with optical flow and depth features respectively . 3 Proposed Framework . In this work , we address the problem of text-guided video temporal grounding using a multi-modal framework . The pipeline of the proposed framework is illustrated in Figure 2 . Given an input video V = { Vt } Tt=1 with T frames and a query sentence Q = { Qi } Ni=1 with N words , we aim to localize the starting and ending time [ ts , te ] of the event corresponding to the query . To this end , we design a multi-modal framework to learn complementary visual information from RGB images , optical flow and depth maps . From the input video , we first compute the depth map of each frame and the optical flow of each pair of consecutive frames . We then apply the visual encoders Ed , Er , Ef to extract features from the depth , RGB and flow inputs . A textual encoder Et is utilized to extract the feature of the query sentence Q . The local-global interaction modules ( LGI ) then incorporate the textual feature into each visual modality , and generate the multi-modal features Md , Mr and Mf for depth , RGB and flow respectively . To effectively integrate the features from different modalities and enable inter-modal feature learning , we propose a dynamic fusion scheme with transformers to model the interaction between modalities . The feature after integration is then fed into a regression module ( REG ) to predict the starting and ending time [ ts , te ] of the target video segment . To enhance the feature representations in each modality , we introduce an intra-modal learning module that conducts self-supervised contrastive learning across videos . The intra-modal learning is applied on the multi-modal features Md , Mr and Mf separately to enforce features of video segments containing the same action to be close to each other , and those from different action categories to be far apart .

Summary: This paper studies the problem of multi-modal video grounding. The authors exploit multiple modalities including RGB, depth, optical flow to extract complementary information for improving video grouding. A dynamic fusion scheme with transformer is proposed to better learn the interaction and integration of multiple modalities. Moreover, a self-supervised intrao-modal module is proposed to obtain better feature representation. The extensive experiments support that the proposed method achieves new state-of-the-art performance on multiple bechmark datasets.