Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos

1 Introduction . Content creators often add a voice-over narration to their videos to point out important moments and guide the viewer on where to look . While the timing of the narration in the audio track in the video gives the viewer a rough idea of when the described moment occurs , there is not explicit information on where to look . We , as viewers , naturally infer this information from the narration and often direct our attention to the spatial location of the described moment . Inspired by this capability , we seek to have a recognition system learn to spatially localize narrated moments without strong supervision . We see a large opportunity for self-supervision as there is an abundance of online narrated video content . In this work , we primarily focus on spatially localizing transcribed narrated interactions in a video , illustrated in Figure 1 ( right ) . Unlike prior phrase grounding work ( Figure 1 left ) , which primarily focuses on matching a noun phrase to an object , our task involves matching entire sentences to regions containing multiple objects and actions . Not only is this task integral to advancing machine perception of our world where information often comes in different modalities , it also has important implications in fundamental vision-and-language research such as robotics , visual question answering , video captioning , and retrieval . Our task is challenging as we do not know the correspondence between spatial regions in a video and words in the transcribed narration . Moreover , there is a loose temporal alignment of the narration with the described moment and not all words refer to spatial regions ( and vice versa ) . Finally , narrated interactions may be complex with long-range dependencies and multiple actions . In Figure 1 ( right ) , notice how “ them ” refers to the onions and that there are two actions – “ cut ” and “ add ” . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contrastive learning [ 38 ] and computing contextual features via an attention model [ 52 ] are natural tools for addressing the aforementioned alignment and long-range cross-modal dependency challenges . While contrastive learning has been successfully applied for temporal alignment of a video clip with a narration [ 35 ] , the alignment of spatial regions with narrations has not been addressed . Naively applying a joint attention model over the set of features for the two modalities , followed by optimizing a contrastive loss over the two modalities ’ aggregated contextual features , results in a fundamental difficulty . Recall that an attention computation comprises two main steps : ( i ) selecting a set of features given a query feature , and ( ii ) aggregating the selected features via weighted averaging . By selecting and aggregating features over the two modalities during the attention computation and subsequently optimizing a contrastive loss , the model in theory may select and aggregate features from only one modality to trivially align the contextual representations . While there are approaches for grounding natural language phrases in images without spatial supervision , the best-performing approaches involve matching individual words to regions [ 22 ] or not using any paired text-image supervision at all [ 54 ] . This strategy is effective for simpler short phrases , as illustrated in Figure 1 ( left ) . However , consider the interaction “ cut the onions and add them to the tray ” , shown in Figure 1 ( right ) . This interaction is described with more complex , compositional language . In the first two frames , there are multiple “ onion '' objects visible and the correct one depends on the action being applied . Unlike the task of object grounding , it is not sufficient to simply co-localize all of the mentioned objects . We will show that it is difficult to effectively employ a strategy that matches individual words to regions to localize such complex interactions . To address these challenges , we make the following contributions . First , we learn to localize the spatial location of a narrated interaction from abundantly available narrated instructional videos [ 36 ] . Our approach does not require manually collecting sentence descriptions or the locations of the described interactions for training . Here , the automatically transcribed narrations allow for self-supervised learning via alignment with the video clip . Second , we propose a new approach for directly contrasting aggregated representations from the two modalities while computing joint attention over spatial regions in a video and words in a narration . We show that optimizing a loss over the aggregated , sentence-level representation allows for a global alignment of the described interaction with the video clip , and offers an improvement over optimizing a matching loss over words and regions . To overcome the network learning a trivial solution while directly contrasting jointly attended features from the two modalities during training , we design network attention layers that do not allow feature aggregation across the two modalities . Our strategy involves alternating network layers that compute inter- and intra-modal attention . Our inter-modal attention layer allows features from one modality to select and aggregate features from only the other modality , and not within its own modality . Our intra-modal attention layer selects and aggregates features from within the same modality . This strategy ensures that the output representations do not aggregate features across the two modalities . In combination with stacking the inter- and intra-modal attention layers , our approach attends jointly and deeply and directly contrasts the resulting contextual representations from the two modalities . Finally , we introduce an evaluation dataset that provides bounding box annotations for interactions described by natural language sentences . Our dataset is built on the validation split of the YouCook2 dataset [ 65 ] and contains approximately 1000 segments of varying durations . We demonstrate our approach on our collected evaluation dataset of localized interactions and on localizing objects via the YouCook2-BB benchmark where we show that we outperform shallow and full cross-modal attention . We also apply our approach to grounding phrases in images with weak supervision on Flickr30K and show that stacking multiple attention layers is effective with our loss . Furthermore , we show that our loss is complementary with the word-to-region loss of Gupta et al . [ 22 ] , and when combined with it , achieves state of the art on recall-at-one and pointing-hand accuracies . 2 Related Work . Self-supervised learning . There has been significant progress in natural language processing , resulting in effective and robust learned word representations [ 37 , 17 , 60 , 39 ] that achieve state-of-theart performance on downstream tasks . Recently , it has also garnered a lot of interest in the computer vision community , achieving state-of-the-art performance in unsupervised pre-training of deep visual models . [ 10 , 24 , 11 ] have demonstrated that effective image representations can be learnt simply by contrasting between augmented views of the same image without labels . [ 62 , 45 ] found that the resulting representations are biased by background pixels and tried to improve foreground object localization via data augmentation or saliency . In video , self-supervised tasks like pace prediction ad future prediction [ 53 , 23 , 19 , 26 , 6 , 55 ] were used for pre-training , while [ 2 ] used self-supervision on audio-visual data for tasks like unsupervised speaker localization . In this work , we propose a self-supervised method to spatially localize activity descriptions in video . Object and action localization . Approaches generally leverage a region proposal network ( RPN ) [ 51 , 21 , 20 ] as well as region-based convolutional neural network ( CNNs ) to detect objects or localize actions temporally [ 59 ] or spatio-temporally [ 27 ] . However , they are often trained on curated datasets and are , consequently , limited by a fixed predefined number of object categories in these datasets . [ 65 ] collected a dataset of cooking videos labeled with descriptions and temporal segments but did not address spatial grounding of activities . Motivated by this limitation , our proposed approach aims to learn to recognize human interactions with objects that belong to the long-tailed distribution from uncurated and unlabeled online videos . There have been video datasets collected with bounding box annotations with associated natural language descriptions [ 63 ] . These datasets provide bounding boxes for noun phrases [ 15 , 63 ] or for egocentric videos [ 15 ] . We collect the first dataset of captioned interactions annotated with spatial bounding boxes for evaluating narrated interactions . Vision-and-language . Semantic information from natural language has often been exploited to provide an additional source of supervision for learning visual representations . Such approaches have generally leveraged image [ 29 , 31 , 40 ] and video [ 5 ] datasets that are annotated with natural language descriptions to learn a joint embedding space , where visual and language representations of semantically-similar pairs are close . Prior approaches have mainly focused on retrieving short clips from a large video corpus [ 35 ] or localizing relevant segments within untrimmed videos [ 5 , 9 , 12 ] . However , they do not identify the relevant spatial locations of the described interactions . Some existing works aim to spatially localize a natural language query [ 44 , 61 , 30 ] but focus on objects and train on ground truth bounding boxes , whereas our approach is unsupervised . Object grounding with weak or no supervision is addressed in [ 12 , 4 , 7 ] . To alleviate the costly annotations of curated datasets , [ 35 ] have proposed to learn robust video representations as well as an effective joint videotext embedding space from a large corpus of unlabeled and uncurated instructional videos [ 36 ] . In concurrent work , CLIP [ 41 ] trained image-text representations on a dataset of 400 million ( image , text ) pairs collected from the internet . These efforts to learn from “ free ” data are similar in spirit to our work , however , we focus on the localization task . 3 Self-Supervised Grounding of Narrated Video Interactions . Given a video clip and a narration , our objective is to learn to localize the relevant spatial regions in the clip that correspond to the interaction described in the narration without localization supervision , e.g. , bounding boxes or segmentation masks . To this end , we propose a novel Contrastive Multilayer Multimodal Attention ( CoMMA ) module that allows for attention interactions between the features for spatiotemporal regions and words , as well as an effective contrasting of the two modalities ’ signals . We define two modalities to have attention interactions when their features attend to each other to compute their final contextualized representations . This setup is in contrast to when the two modalities ’ features only interact when the training loss is computed [ 35 ] . Our CoMMA module comprises alternating bidirectional cross-modal attention and self-attention layers ( Figure 2 ( a ) ) . The intuition underlying our proposed approach is that a hierarchical multimodal attention model with multiple stacked layers will encourage a fine-grained alignment between video spatiotemporal regions and narration words and ignore irrelevant regions/words . More specifically , each bidirectional cross-modal attention layer computes new representations for a target modality with the latent representations from the source modality to learn the relevance of each spatiotemporal region to each word or phrase , and vice versa . Additionally , the self-attention layer serves to identify relevant words and regions by aggregating contextual information between the augmented unimodal features . Through repeated attention interactions of the two modalities and aggregation of unimodal contextual information , our CoMMA module becomes more discriminative of irrelevant regions and words . Moreover , our module allows for applying a contrastive loss to the final representations for the two modalities since there is no cross-modal feature aggregation , i.e. , a modality may self-select features or select features from another modality by attending to them , but it may never select features from both modalities . Our approach goes beyond shallow attention architectures [ 3 ] and applying a contrastive loss to independent fixed-length features for each modality ( i.e. , no attention interaction between the two modalities ) [ 35 ] . Our multimodal attention module can be easily stacked on top of base feature encoders for our task ( Figure 2 ( b ) shows our full network ) . Next , we describe our CoMMA module in detail , how we employ a contrastive loss for training , and how we employ our network for the localization task .

Summary: The paper proposes a method for grounding sentences in videos and images without explicit localization supervision.  The approach rely on a cross-attention architecture that is used in a way to make contrastive learning possible (since it prevents the final feature from containing information from the other modalities while still enabling information to be shared across modalities earlier in the network). Localization can be obtained from the cross-attention patterns between the language and the visual signal.  The authors apply their method on a subset of the HowTo100M dataset for videos and evaluate on the YouCookBB object for objects and a newly introduced YouCookInteraction where they show improvements over baselines. The method is also applied on the Flickr30K dataset on images.