Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos

1 Introduction . Content creators often add a voice-over narration to their videos to point out important moments and guide the viewer on where to look . While the timing of the narration in the audio track in the video gives the viewer a rough idea of when the described moment occurs , there is not explicit information on where to look . We , as viewers , naturally infer this information from the narration and often direct our attention to the spatial location of the described moment . Inspired by this capability , we seek to have a recognition system learn to spatially localize narrated moments without strong supervision . We see a large opportunity for self-supervision as there is an abundance of online narrated video content . In this work , we primarily focus on spatially localizing transcribed narrated interactions in a video , illustrated in Figure 1 ( right ) . Unlike prior phrase grounding work ( Figure 1 left ) , which primarily focuses on matching a noun phrase to an object , our task involves matching entire sentences to regions containing multiple objects and actions . Not only is this task integral to advancing machine perception of our world where information often comes in different modalities , it also has important implications in fundamental vision-and-language research such as robotics , visual question answering , video captioning , and retrieval . Our task is challenging as we do not know the correspondence between spatial regions in a video and words in the transcribed narration . Moreover , there is a loose temporal alignment of the narration with the described moment and not all words refer to spatial regions ( and vice versa ) . Finally , narrated interactions may be complex with long-range dependencies and multiple actions . In Figure 1 ( right ) , notice how “ them ” refers to the onions and that there are two actions – “ cut ” and “ add ” . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contrastive learning [ 38 ] and computing contextual features via an attention model [ 52 ] are natural tools for addressing the aforementioned alignment and long-range cross-modal dependency challenges . While contrastive learning has been successfully applied for temporal alignment of a video clip with a narration [ 35 ] , the alignment of spatial regions with narrations has not been addressed . Naively applying a joint attention model over the set of features for the two modalities , followed by optimizing a contrastive loss over the two modalities ’ aggregated contextual features , results in a fundamental difficulty . Recall that an attention computation comprises two main steps : ( i ) selecting a set of features given a query feature , and ( ii ) aggregating the selected features via weighted averaging . By selecting and aggregating features over the two modalities during the attention computation and subsequently optimizing a contrastive loss , the model in theory may select and aggregate features from only one modality to trivially align the contextual representations . While there are approaches for grounding natural language phrases in images without spatial supervision , the best-performing approaches involve matching individual words to regions [ 22 ] or not using any paired text-image supervision at all [ 54 ] . This strategy is effective for simpler short phrases , as illustrated in Figure 1 ( left ) . However , consider the interaction “ cut the onions and add them to the tray ” , shown in Figure 1 ( right ) . This interaction is described with more complex , compositional language . In the first two frames , there are multiple “ onion '' objects visible and the correct one depends on the action being applied . Unlike the task of object grounding , it is not sufficient to simply co-localize all of the mentioned objects . We will show that it is difficult to effectively employ a strategy that matches individual words to regions to localize such complex interactions . To address these challenges , we make the following contributions . First , we learn to localize the spatial location of a narrated interaction from abundantly available narrated instructional videos [ 36 ] . Our approach does not require manually collecting sentence descriptions or the locations of the described interactions for training . Here , the automatically transcribed narrations allow for self-supervised learning via alignment with the video clip . Second , we propose a new approach for directly contrasting aggregated representations from the two modalities while computing joint attention over spatial regions in a video and words in a narration . We show that optimizing a loss over the aggregated , sentence-level representation allows for a global alignment of the described interaction with the video clip , and offers an improvement over optimizing a matching loss over words and regions . To overcome the network learning a trivial solution while directly contrasting jointly attended features from the two modalities during training , we design network attention layers that do not allow feature aggregation across the two modalities . Our strategy involves alternating network layers that compute inter- and intra-modal attention . Our inter-modal attention layer allows features from one modality to select and aggregate features from only the other modality , and not within its own modality . Our intra-modal attention layer selects and aggregates features from within the same modality . This strategy ensures that the output representations do not aggregate features across the two modalities . In combination with stacking the inter- and intra-modal attention layers , our approach attends jointly and deeply and directly contrasts the resulting contextual representations from the two modalities . Finally , we introduce an evaluation dataset that provides bounding box annotations for interactions described by natural language sentences . Our dataset is built on the validation split of the YouCook2 dataset [ 65 ] and contains approximately 1000 segments of varying durations . We demonstrate our approach on our collected evaluation dataset of localized interactions and on localizing objects via the YouCook2-BB benchmark where we show that we outperform shallow and full cross-modal attention . We also apply our approach to grounding phrases in images with weak supervision on Flickr30K and show that stacking multiple attention layers is effective with our loss . Furthermore , we show that our loss is complementary with the word-to-region loss of Gupta et al . [ 22 ] , and when combined with it , achieves state of the art on recall-at-one and pointing-hand accuracies . 2 Related Work . Self-supervised learning . There has been significant progress in natural language processing , resulting in effective and robust learned word representations [ 37 , 17 , 60 , 39 ] that achieve state-of-theart performance on downstream tasks . Recently , it has also garnered a lot of interest in the computer vision community , achieving state-of-the-art performance in unsupervised pre-training of deep visual models . [ 10 , 24 , 11 ] have demonstrated that effective image representations can be learnt simply by contrasting between augmented views of the same image without labels . [ 62 , 45 ] found that the resulting representations are biased by background pixels and tried to improve foreground object localization via data augmentation or saliency . In video , self-supervised tasks like pace prediction ad future prediction [ 53 , 23 , 19 , 26 , 6 , 55 ] were used for pre-training , while [ 2 ] used self-supervision on audio-visual data for tasks like unsupervised speaker localization . In this work , we propose a self-supervised method to spatially localize activity descriptions in video . Object and action localization . Approaches generally leverage a region proposal network ( RPN ) [ 51 , 21 , 20 ] as well as region-based convolutional neural network ( CNNs ) to detect objects or localize actions temporally [ 59 ] or spatio-temporally [ 27 ] . However , they are often trained on curated datasets and are , consequently , limited by a fixed predefined number of object categories in these datasets . [ 65 ] collected a dataset of cooking videos labeled with descriptions and temporal segments but did not address spatial grounding of activities . Motivated by this limitation , our proposed approach aims to learn to recognize human interactions with objects that belong to the long-tailed distribution from uncurated and unlabeled online videos . There have been video datasets collected with bounding box annotations with associated natural language descriptions [ 63 ] . These datasets provide bounding boxes for noun phrases [ 15 , 63 ] or for egocentric videos [ 15 ] . We collect the first dataset of captioned interactions annotated with spatial bounding boxes for evaluating narrated interactions . Vision-and-language . Semantic information from natural language has often been exploited to provide an additional source of supervision for learning visual representations . Such approaches have generally leveraged image [ 29 , 31 , 40 ] and video [ 5 ] datasets that are annotated with natural language descriptions to learn a joint embedding space , where visual and language representations of semantically-similar pairs are close . Prior approaches have mainly focused on retrieving short clips from a large video corpus [ 35 ] or localizing relevant segments within untrimmed videos [ 5 , 9 , 12 ] . However , they do not identify the relevant spatial locations of the described interactions . Some existing works aim to spatially localize a natural language query [ 44 , 61 , 30 ] but focus on objects and train on ground truth bounding boxes , whereas our approach is unsupervised . Object grounding with weak or no supervision is addressed in [ 12 , 4 , 7 ] . To alleviate the costly annotations of curated datasets , [ 35 ] have proposed to learn robust video representations as well as an effective joint videotext embedding space from a large corpus of unlabeled and uncurated instructional videos [ 36 ] . In concurrent work , CLIP [ 41 ] trained image-text representations on a dataset of 400 million ( image , text ) pairs collected from the internet . These efforts to learn from “ free ” data are similar in spirit to our work , however , we focus on the localization task . 3 Self-Supervised Grounding of Narrated Video Interactions . Given a video clip and a narration , our objective is to learn to localize the relevant spatial regions in the clip that correspond to the interaction described in the narration without localization supervision , e.g. , bounding boxes or segmentation masks . To this end , we propose a novel Contrastive Multilayer Multimodal Attention ( CoMMA ) module that allows for attention interactions between the features for spatiotemporal regions and words , as well as an effective contrasting of the two modalities ’ signals . We define two modalities to have attention interactions when their features attend to each other to compute their final contextualized representations . This setup is in contrast to when the two modalities ’ features only interact when the training loss is computed [ 35 ] . Our CoMMA module comprises alternating bidirectional cross-modal attention and self-attention layers ( Figure 2 ( a ) ) . The intuition underlying our proposed approach is that a hierarchical multimodal attention model with multiple stacked layers will encourage a fine-grained alignment between video spatiotemporal regions and narration words and ignore irrelevant regions/words . More specifically , each bidirectional cross-modal attention layer computes new representations for a target modality with the latent representations from the source modality to learn the relevance of each spatiotemporal region to each word or phrase , and vice versa . Additionally , the self-attention layer serves to identify relevant words and regions by aggregating contextual information between the augmented unimodal features . Through repeated attention interactions of the two modalities and aggregation of unimodal contextual information , our CoMMA module becomes more discriminative of irrelevant regions and words . Moreover , our module allows for applying a contrastive loss to the final representations for the two modalities since there is no cross-modal feature aggregation , i.e. , a modality may self-select features or select features from another modality by attending to them , but it may never select features from both modalities . Our approach goes beyond shallow attention architectures [ 3 ] and applying a contrastive loss to independent fixed-length features for each modality ( i.e. , no attention interaction between the two modalities ) [ 35 ] . Our multimodal attention module can be easily stacked on top of base feature encoders for our task ( Figure 2 ( b ) shows our full network ) . Next , we describe our CoMMA module in detail , how we employ a contrastive loss for training , and how we employ our network for the localization task .

Summary: This paper concerns a new problem of grounding narrated interactions in video. It is similar to existing work on (object) phrase localization/grounding, but with a twist on grounding more complex narrations that could involve both entities and predicates (e.g., put tomatoes into the baking tray). The model training is performed in a weakly supervised fashion on a subset of HowTo100M. Training loss is the standard InfoNCE loss at the sentence level (vs. word-level as in existing grounding work). Model architecture is designed to capture inter-modal correspondence through cross-modal attention. Evaluation is conducted on YouCook2-BB, Flickr30k-Entities, and a newly collected dataset for narration grounding called YouCook2-Interaction.