End-to-end Multi-modal Video Temporal Grounding
Year: 2021

Abstract:
We address the problem of text-guided video temporal grounding, which aims to identify the time interval of a certain event based on a natural language description. Different from most existing methods that only consider RGB images as visual features, we propose a multi-modal framework to extract complementary information from videos. Specifically, we adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. While RGB images provide abundant visual cues of certain events, the performance may be affected by background clutters. Therefore, we use optical flow to focus on large motion and depth maps to infer the scene configuration when the action is related to objects recognizable with their shapes. To integrate the three modalities more effectively and enable inter-modal learning, we design a dynamic fusion scheme with transformers to model the interactions between modalities. Furthermore, we apply intra-modal self-supervised learning to enhance feature representations across videos for each modality, which also facilitates multi-modal learning. We conduct extensive experiments on the Charades-STA and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.

heading: 1 Introduction
text: With the rapid growth of video data in our daily lives, video understanding has become an ever increasingly important task in computer vision. Research involving other modalities such as text and speech has also drawn much attention in recent years, e.g., video captioning [17, 23], and video question answering [18, 16]. In this paper, we focus on text-guided video temporal grounding, which aims to localize the starting and ending time of a segment corresponding to a text query. It is one of the most effective approaches to understand video contents, and applicable to numerous tasks, such as video retrieval, video editing and human-computer interaction. This problem is considerably challenging as it requires accurate recognition of objects, scenes and actions, as well as joint comprehension of video and language.
Existing methods [34, 26, 33, 22] usually consider only RGB images as visual cues, which are less effective for recognizing objects and actions in videos with complex backgrounds. To understand the video contents more holistically, we propose a multi-modal framework to learn complementary visual features from RGB images, optical flow and depth maps. RGB images provide abundant visual information, which is essential for visual recognition. However, existing methods based on appearance alone are likely to be less effective for complex scenes with cluttered backgrounds. For example, since the query text descriptions usually involve moving objects such as “Closing a door” or “Throwing a pillow”, using optical flow as input is able to identify such actions with large motion. On the other hand, depth is another cue that is invariant to color and lighting, and is often used to complement the RGB input in object detection and semantic segmentation. In our task, depth information helps the proposed model recognize actions involving objects with distinct shapes as the context. For example, actions such as “Sitting in a bed” or “Working at a table” are not easily recognized by optical flow due to small motion, but depth can provide structural information to assist the learning process. We also note that, our goal is to design an end-to-end multi-modal framework
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
for video grounding by directly utilizing low-level cues such as optical flow and depth, while other alternatives based on object detector or semantic segmentation is out of the scope of this work.
To leverage multi-modal cues, one straightforward way is to construct a multi-stream model that takes individual modality as the input in each stream, and then averages the multi-stream output predictions to obtain final results. However, we find that this scheme is less effective due to the lack of communication across different modalities, e.g., using depth cues alone without considering RGB features is not sufficient to learn the semantic information as the appearance cue does. To tackle this issue, we propose a multi-modal framework with 1) an inter-modal module that learns cross-modal features, and 2) an intra-modal module to self-learn feature representations across videos.
For inter-modal learning, we design a fusion scheme with co-attentional transformers [20] to dynamically fuse features from different modalities. One motivation is that, different videos may require to adopt a different combination of modalities, e.g., “Working at a table” would require more appearance and depth information, while optical flow is more important for “Throwing a pillow”. To enhance feature representations for each modality and thereby improve multi-modal learning, we introduce an intra-modal module via self-supervised contrastive learning [7, 15]. The goal is to ensure the feature consistency across video clips when they contain the same action. For example, with the same action “Eating”, it may happen at different locations with completely different backgrounds and contexts, or with different text descriptions that “eats” different food. With our intra-modal learning, it enforces features close to each other when they describe the same action and learn features that are invariant to other distracted factors across videos, and thus it can improve our multi-modal learning paradigm.
We conduct extensive experiments on the Charades-STA [10] and ActivityNet Captions [17] datasets to demonstrate the effectiveness of our multi-modal learning framework for video temporal grounding using (D)epth, (R)GB, and optical (F)low with the (T)ext as the query, and name our method as DRFT. First, we present the complementary property of multi-modality and the improved performance over the single-modality models. Second, we validate the individual contributions of our proposed components, i.e., inter- and intra-modal modules, that facilitate multi-modal learning. Finally, we show state-of-the-art performance for video temporal grounding against existing methods.
The main contributions of this work are summarized as follows: 1) We propose a multi-modal framework for text-guided video temporal grounding by extracting complementary information from RGB, optical flow and depth features. 2) We design a dynamic fusion mechanism across modalities via co-attentional transformer to effectively learn inter-modal features. 3) We apply self-supervised contrastive learning across videos for each modality to enhance intra-modal feature representations that are invariant to distracted factors with respect to actions.

heading: 2 Related Work
text: Text-Guided Video Temporal Grounding. Given a video and a natural language query, textguided video temporal grounding aims to predict the starting and ending time of the video clip that best matches the query sentence. Existing methods for this task can be categorized into two groups, i.e., two-stage and one-stage schemes (see Figure 1(a)(b)). Most two-stage approaches adopt a propose-and-rank pipeline, where they first generate clip proposals and then rank the proposals based on their similarities with the query sentence. Early two-stage methods [10, 14] obtain proposals by scanning the whole video with sliding windows. Since the sliding window mechanism is computationally expensive and usually produces many redundant proposals, numerous methods are subsequently proposed to improve the efficiency and effectiveness of proposal generation. The TGN model [2] performs frame-by-word interactions and localize the proposals in one single pass. Other approaches focus on reducing redundant proposals by generating query-guided proposals [31] or semantic activity proposals [3]. The MAN method [34] models the temporal relationships between proposals using a graph architecture to improve the quality of proposals. To alleviate the computation of observing the whole video, reinforcement learning [13, 30] is utilized to guide the intelligent agent to glance over the video in a discontinuous way. While the two-stage methods achieve promising results, the computational cost is high for comparing all proposal-query pairs, and the performance is largely limited by the quality of proposal generation.
To overcome the issues of two-stage methods, some recent approaches adopt a one-stage pipeline to directly predict the temporal segment from the fusion of video and text features. Most of the one-stage approaches focus on the attention mechanisms or interaction between modalities. For
example, the ABLR method [32] predicts the temporal coordinates using a co-attention based location regression algorithm. The ExCL mechanism [11] exploits the cross-modal interactions between video and text, and the PfTML-GA model [26] improves the performance by introducing the queryguided dynamic filter. Moreover, the DRN scheme [33] leverages dense supervision from the sparse annotations to facilitate the training process. Recently, the LGI model [22] decomposes the query sentence into multiple semantic phrases and conducts local and global interactions between the video and text features. In our framework, we adopt LGI as the baseline that uses the hierarchical video-text interaction. However, different from LGI that only considers RGB frames as input, we take RGB, optical flow and depth as input, and design the inter-modality learning technique to learn complementary information from the video. Furthermore, we apply contrastive learning across videos to enhance the feature representations in each modality, which helps the learning of the whole model (see Figure 1(c)).
Multi-Modal Learning. As typical event or actions can be described by signals from multiple modalities, understanding the correlation between different modalities is crucial to solve problems more comprehensively. Research on joint vision and language learning [6, 23, 16, 7] has gained much attention in recent years since natural language is an intuitive way for human communication. Recent studies [24, 8] based on the transformer [29] have shown great success in self-supervised learning and transfer learning for natural language tasks. The transformer-based BERT model [8] has also been widely used to learn joint representations for vision and language. These methods [20, 21, 35, 19, 5, 9] aim to learn generic representations from a large amount of image-text pairs in a self-supervised manner, and then fine-tune the model for downstream vision and language tasks. The ViLBERT scheme [20] extracts features from image and text using two parallel BERT-style models, and then connects the two streams with the co-attentional transformer layers. In this work, we focus on the video temporal grounding task guided by texts, while introducing multi-modality to improve model learning, which is not studied before. For fusing the multi-modal information, we leverage the co-attentional transformer layers [20] in our framework and design an approach by fusing the RGB features with optical flow and depth features respectively.

heading: 3 Proposed Framework
text: In this work, we address the problem of text-guided video temporal grounding using a multi-modal framework. The pipeline of the proposed framework is illustrated in Figure 2. Given an input video V = {Vt}Tt=1 with T frames and a query sentence Q = {Qi}Ni=1 with N words, we aim to localize
the starting and ending time [ts, te] of the event corresponding to the query. To this end, we design a multi-modal framework to learn complementary visual information from RGB images, optical flow and depth maps. From the input video, we first compute the depth map of each frame and the optical flow of each pair of consecutive frames. We then apply the visual encoders Ed, Er, Ef to extract features from the depth, RGB and flow inputs. A textual encoder Et is utilized to extract the feature of the query sentence Q. The local-global interaction modules (LGI) then incorporate the textual feature into each visual modality, and generate the multi-modal features Md, Mr and Mf for depth, RGB and flow respectively.
To effectively integrate the features from different modalities and enable inter-modal feature learning, we propose a dynamic fusion scheme with transformers to model the interaction between modalities. The feature after integration is then fed into a regression module (REG) to predict the starting and ending time [ts, te] of the target video segment. To enhance the feature representations in each modality, we introduce an intra-modal learning module that conducts self-supervised contrastive learning across videos. The intra-modal learning is applied on the multi-modal features Md, Mr and Mf separately to enforce features of video segments containing the same action to be close to each other, and those from different action categories to be far apart.

heading: 3.1 Inter-Modal Feature Learning
text: Videos contain rich information in both spatial and temporal dimensions. To learn information more comprehensively, in addition to the RGB modality, we also consider optical flow that captures motion, and depth feature that represents image structure. An intuitive way to combine the three modalities is to utilize a multi-stream model and directly average the outputs of individual streams. However, since the importance of each modality is not the same in different situations, directly averaging them may downweigh the importance of a specific modality and degrade the performance. In Table 1, we present the results of two-stream (RGB and flow) and three-stream (RGB, flow and depth) baseline models, where the outputs from different modalities are averaged before the final output layer. Compared to the single-stream (RGB) baseline model, the multi-stream models do not improve the performance, which shows that it is not intuitive to learn complementary information from multi-modal features.
Such cases may happen frequently in certain actions. For example, flow features would not help much for “Sitting in a bed” but would help more for “Closing a door”. Therefore, having a dynamic mechanism is critical for multi-modal fusion.
Co-attentional Feature Fusion. The ensuing question becomes how to learn effective features across modalities and also fuse them dynamically. First, we observe that, although depth and flow modalities are effective in some situations, they alone are not able to capture the semantic information, which is crucial for video-text understanding. Thereby, we design a co-attentional scheme to allow joint feature learning between RGB and another modality (either depth or flow).
Inspired by the co-attentional transformer layer [20] that consist of multi-headed attention blocks, where it takes a paired feature as the input (e.g., Md and Mr) and forms three matrices, Q, K, and V that represent queries, keys, and values (also see Figure 2). In this way, the multi-headed attention block for each modality takes the keys and values from the other modality as the input, and thus outputs the attention-pooled features conditioned on the other modality. For instance, we consider the pair of Md = {m1d, ...,mTd } and Mr = {m1r, ...,mTr } features for depth and RGB, where T is the number of frames in a video clip. The co-attentional transformer layer performs depth-conditioned RGB feature attention, as well as RGB-conditioned depth feature attention. Similarly, for flow and RGB features, Mf = {m1f , ...,mTf } and Mr = {m1r, ...,mTr }, we obtain another set of flowconditioned RGB feature attention and RGB-conditioned flow feature attention. Note that, since RGB feature generally contains the most abundant information in the video and is used for both depth and flow attention, we adopt a shared multi-headed attention block for RGB as shown in Figure 2.
Dynamic Feature Fusion. To effectively combine these output features from co-attentional transformers and perform the final prediction, we dynamically learn the weights for each multi-modal feature and linearly combine the four features using the weights (see Figure 2). Since the importance of each modality depends on the input data, we generate the weights by feeding each feature into a fully-connected (FC) layer, and normalize the weights to make the sum equal to 1. By dynamically generating weights from the features, we are able to adapt the multi-modal fusion process according to the input video and the query text. The fused feature is then served as input of the regression module (REG) to predict the starting and ending time.

heading: 3.2 Intra-Modal Feature Learning
text: To facilitate the multi-modal training, we introduce an intra-modal feature learning module, which enhances feature representations within each modality by applying self-supervised contrastive learning. Our motivation is that features in the same action category should be similar even if they are from different videos. To this end, for each input video V , we randomly sample positive videos V+ that contain the same action category, and negative videos V− with different action categories. We perform contrastive learning on the multi-modal features Md, Mr and Mf ∈ Rc×T separately for each modality, where c is the feature dimension and T is the number of frames. Since the multi-modal feature contains information from the whole video, we only consider features that contain the action by extracting the corresponding video segment. We then conduct average pooling in the temporal dimension and obtain a feature vector M ∈ Rc. The contrastive loss Lcl is formulated as:
Lcl = − log
∑ M+∈Q+
eh(M) ⊤h(M+)/τ∑
M+∈Q+ eh(M)⊤h(M+)/τ + ∑ M−∈Q− eh(M)⊤h(M−)/τ , (1)
where Q+ and Q− are the sets of positive and negative samples, and τ is the temperature parameter. Following the SimCLR approach [4], we use a linear layer h(·) to project the feature M to another embedding space where we apply contrastive loss. We accumulate the loss from each modality to be the final contrastive loss, namely Lcl = Lrcl(hr(AvgPool(Mr))) + L d cl(hd(AvgPool(Md))) + Lfcl(hf (AvgPool(Mf ))) for RGB, depth, and flow, respectively.

heading: 3.3 Model Training and Implementation Details
text: Overall Objective. The overall objective of the proposed method is composed of the supervised loss Lgrn for predicting temporal grounding that localizes the video segment and the self-supervised contrastive loss Lcl for intra-modal learning in (1): L = Lgrn + Lcl. The supervised loss Lgrn is the same as the loss defined in the LGI method [22], which includes:
1) Location regression loss Lreg = smoothL1(t̂s − ts) + smoothL1(t̂e − te) that calculates the L1 distance between the normalized ground truth time interval (t̂s, t̂e) ∈ [0, 1] and the predicted time interval (ts, te), where smoothL1 is defined as 0.5x2 if |x| < 1 and |x| − 0.5 otherwise. 2) Temporal attention guidance loss Ltag = − T∑ i=1 ôi log(oi) T∑
i=1 ôi
for the temporal attention in the REG
module, where ôi is set to 1 if the i-th segment is located within the ground truth time interval and 0 otherwise.
3) Distinct query attention loss Ldqa = ||(A⊤A) − λI||2F to enforce query attention weights to be distinct along different steps in the LGI module, where A ∈ RN×S is the concatenated query attention weights across S steps, || · ||F denotes Frobenius norm of a matrix, and λ ∈ [0, 1] controls the extent of overlap between query attention distributions. The supervised loss is the sum of the three loss terms Lgrn = Lreg +Ltag +Ldqa and we use the default setting in LGI [22], in which we refer readers to their paper for more details.
Implementation Details. We generate optical flow and depth maps using the RAFT [27] and MiDaS [25] method respectively. For the visual encoder Ed, Er and Ef , we employ the I3D [1] and C3D [28] networks for Charades-STA and ActivityNet Captions datasets respectively. As for the textual encoder Et, we adopt a bi-directional LSTM, where the feature is obtained by concatenating the last hidden states in forward and backward directions. The LGI module in our framework contains the sequential query attention and local-global video-text interactions as in the LGI model [22]. The REG module generates temporal attention weights to aggregate the features and performs regression via an MLP layer. The operations are defined similar to LGI [22]. The feature dimension c is set to 512. In the contrastive loss (1), the temperature parameter τ is set to 0.1. The projection head h(·) is a 2-layer MLP that project the feature to a 512-dimensional latent space. We implement the proposed model in PyTorch with the Adam optimizer and a fixed learning rate of 4× 10−4. The source code and models are available at https://github.com/wenz116/DRFT.

heading: 4 Experimental Results

heading: 4.1 Datasets and Evaluation Metric
text: We evaluate the proposed DRFT method against the state-of-the-art approaches on two benchmark datasets, i.e., Charades-STA [10] and ActivityNet Captions [17].
Charades-STA. It is built upon the Charades dataset for evaluating the video temporal grounding task. It contains 6,672 videos involving 16,128 video-query pairs, where 12,408 pairs are used for training and 3,720 pairs are for testing. The average length of the videos is 29.76 seconds. There are 2.4 annotated moments with duration 8.2 seconds in each video.
ActivityNet Captions. It is originally constructed for dense video captioning from the ActivityNet dataset. The captions are used as queries in the video temporal grounding task. It consists of 20k YouTube videos with an average duration of 120 seconds. The videos are annotated with 200 activity categories, which is more diverse compared to the Charades-STA dataset. Each video contains 3.65 queries, where each query has an average length of 13.48 words. The dataset is split into training, validation and testing set with a ratio of 2:1:1, resulting in 37,421, 17,505 and 17,031 video-query
pairs respectively. Since the testing set is not publicly available, we follow previous methods to evaluate the performance on the combination of the two validation sets, which are denoted as val1 and val2.
Following the typical evaluation setups [10, 22], we employ two metrics to assess the performance of video temporal grounding: 1) Recall at various thresholds of temporal Intersection over Union (R@IoU). It measures the percentage of predictions that have IoU with the ground truth larger than the threshold. We adopt 3 values {0.3, 0.5, 0.7} for the IoU threshold. 2) mean tIoU (mIoU). It is the average IoU over all results.

heading: 4.2 Overall Performance
text: In Table 2, we evaluate our framework against state-of-art approaches, including two-stage methods that rely on propose-and-rank schemes [10, 13, 34] and one-stage methods that only consider RGB videos as the input [12, 26, 33, 22]. First, compared to our baseline LGI [22], our results with single/two/three modalities are consistently better than theirs in all the evaluation metrics, which demonstrates the benefit of our intra-modal feature learning scheme and the inter-modal feature fusion mechanism. We also note that for our single-modal model, we use RGB as the input and only apply the intra-modal contrastive learning across videos, where it already performs favorably against existing algorithms. More results of the single-stream models using other modalities are provided in the supplementary material.
Second, we show that with the increased modality used in our model (bottom group in Table 2), the performance on two benchmarks are consistently improved, which demonstrates the complementary property of RGB, depth, and flow for video temporal grounding. Moreover, compared to the worse baseline results when adding more modalities without our proposed modules in Table 1, we validate the importance of designing a proper scheme of exchanging and fusing the information across modalities. It is also worth mentioning that with more modalities involved in the model, our method achieves larger performance gains compared to the baseline, e.g., more than 5% improvement in all the metrics on two benchmarks.

heading: 4.3 Ablation Study
text: In Table 3, we present the ablation of individual components proposed in our framework.
Inter-modal Feature Fusion. To enhance the communication across modalities, we propose to first use co-attentional transformers to learn attentive features across RGB and another modality, and then use a dynamic feature fusing scheme with learned weights to combine different features. In the first four rows of the middle group in Table 3, we show the following properties in this work:
1) Using transformers is effective for multi-modal feature learning. As shown in the first row of the middle group, the performance drops without the co-attentional transformers for feature fusion.
2) RGB information is essential for the temporal grounding task, and thus we conduct co-attention between a) RGB-flow and b) RGB-depth. In the second row of the middle group, if using the flow modality as the common modality, i.e., flow-RGB and flow-depth, the performance is worse than our final model.
3) Since RGB features are used for both flow and depth attention, we adopt a shared co-attention block for RGB as shown in Figure 2, where it can take RGB together with either the flow or depth cue as the input, and further enriches the attention mechanism. This design has not been considered in the prior work. In the third row of the middle group, without sharing the co-attentional module, the performance is worse than our final model.
4) The proposed dynamic fusion scheme via learnable weights is important to fuse features from different modalities. As shown the fourth row of the middle group, the performance drops significantly without learnable weights. Interestingly, learning dynamic weights to combine features is almost equally important compared to feature learning via transformers. This indicates that even with the state-of-the-art feature attention module, it is still challenging to combine multi-modal features.
Intra-modal Feature Learning. In the last row of the middle group in Table 3, we show the benefit of having the intra-modal cross-video feature learning. While multi-modal feature fusion already provides a strong baseline in our framework, improving feature representations in individual modalities is still critical for enhancing the entire multi-modal learning paradigm, in which such observations are not widely studied yet.
Qualitative Results. In Figure 3, we show sample results on the Charades-STA and ActivityNet Captions datasets, where the arrows indicate the starting and ending points of the corresponding grounded segment based on the query. Compared to the baseline method that only consider RGB features, the proposed DRFT approach is able to predict more accurate results by leveraging the multimodal features from RGB, optical flow and depth. More results are presented in the supplementary material.

heading: 4.4 Analysis of Multi-modal Learning
text: To understand the complementary property of each modality, we analyze the video temporal grounding results of some example action categories. Figure 4 shows the performance of the single-stream baseline with RGB as input, single-stream DRFT models with RGB, flow or depth as input, and three-stream DRFT model respectively. The three plots contain categories where RGB, flow or depth performs better than the other two modalities. We first show that the single-stream DRFT model with contrastive learning improves from the single-stream baseline (red bars vs. orange bars). We then investigate the complementary property between the three modalities. For actions with smaller movement (e.g., “smiling”) in the left group of Figure 4, models using RGB as input generally
perform better. For actions with larger motion (e.g., “closing a door” or “throwing a pillow”) in the middle group, flow provides more useful information (denoted as green bars). As for the actions with small motion but can be easily recognized by their structure (e.g., “sitting in a bed” or “working at a table”) in the right group, depth is superior to the other two modalities (denoted as blue bars). With the complementary property between RGB, flow and depth, we can take advantage of each modality and further improve the performance in the three-stream DRFT model (denoted as purple bars).
To further analyze the impact of each modality, we provide the learned weights for dynamic fusion in the three-stream DRFT for these categories in Table 4, where the top, middle and bottom groups contain categories that RGB, flow and depth help the most respectively. Flow → RGB means flowconditioned RGB features, etc. We observe that for actions with smaller movement (top group), the weights for RGB features are larger. For actions with larger motion (middle group), the weights for optical flow are larger. Regarding actions with small motion but can be easily recognized by their structure (bottom group), the weights for depth are larger. This shows that the model can exploit each modality based on the complementary property between RGB, flow and depth.

heading: 5 Conclusions
text: In this paper, we focus on the task of text-guided video temporal grounding. In contrast to existing methods that consider only RGB images as visual features, we propose the DRFT model to learn complementary visual information from RGB, optical flow and depth modality. While RGB features provide abundant appearance information, we show that representation models based on these cues alone are not effective for temporal grounding in videos with cluttered backgrounds. We therefore adopt optical flow to capture motion cues, and depth maps to estimate image structure. To combine the three modalities more effectively, we propose an inter-modal feature learning module, which performs co-attention between modalities using transformers, and dynamically fuses the multi-modal features based on the input data. To further enhance the multi-modal training, we incorporate an intra-modal feature learning module that performs self-supervised contrastive learning within each modality. The contrastive loss enforces cross-video features to be close to each other when they contain the same action, and be far apart otherwise. We conduct extensive experiments on two benchmark datasets, demonstrating the effectiveness of the proposed multi-modal framework with inter- and intra-modal feature learning.

heading: Acknowledgements
text: This work is supported in part by NSF CAREER grant 1149783 and gifts from Snap as well as eBay.

Review summaries:
This paper studies the problem of multi-modal video grounding. The authors exploit multiple modalities including RGB, depth, optical flow to extract complementary information for improving video grouding. A dynamic fusion scheme with transformer is proposed to better learn the interaction and integration of multiple modalities. Moreover, a self-supervised intrao-modal module is proposed to obtain better feature representation. The extensive experiments support that the proposed method achieves new state-of-the-art performance on multiple bechmark datasets.

This paper addresses text-guided video grounding, which identifies the time interval of an event according to the query text. In contrast to prior work that only leverages RGB information, the authors propose DRFT to combine RGB, depth, and flow maps to boost the performance. To facilitate representation learning, the authors adopt a co-attentional transformer to fuse multi-modal information and contrastive learning to enhance the features across different videos. Experimental results shows that the proposed DRFT outperforms state-of-the-art on Charades-STA and ActivityNet.

Paper addresses the problem of text-guided video temporal grounding, which aims to localize the starting and ending time of a segment corresponding to a text query. The key contribution is integration of three modalities of data in this context: video, motion (flow), and depth; along with the textural query. To integrate the three modalities, paper proposes a dynamic fusion scheme with transformers, which takes the form of co-attention (similar to ViLBERT). Further, to improve the performance within each modality contrastive learning is applied to enhance the feature discriminability. In this contrastive learning formulation positive pairs come from instance of the same action class, while negative pairings are formed from videos coming from two different action classes. Competitive (state-of-the-art) performance is illustrated on Charades-STA and ActivityNet Captions benchmark datasets.