Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Authors: Reuben Tan
Bryan A. Plummer
Kate Saenko
Hailin Jin
Bryan Russell
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Self-supervision
video understanding
natural language

Abstract: We introduce the task of spatially localizing narrated interactions in videos. Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations. 
To achieve this goal, we propose a multilayer cross-modal attention network that enables effective optimization of a contrastive loss during training. We introduce a divided strategy that alternates between computing inter- and intra-modal attention across the visual and natural language modalities, which allows effective training via directly contrasting the two modalities' representations. We demonstrate the effectiveness of our approach by self-training on the HowTo100M instructional video dataset and evaluating on a newly collected dataset of localized described interactions in the YouCook2 dataset. We show that our approach outperforms alternative baselines, including shallow co-attention and full cross-modal attention. We also apply our approach to grounding phrases in images with weak supervision on Flickr30K and show that stacking multiple attention layers is effective and, when combined with a word-to-region loss, achieves state of the art on recall-at-one and pointing hand accuracies.

Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Reuben Tan Bryan A. Plummer' Kate Saenko 1.2 Hailin Jin
Bryan Russell
Boston University, "MIT-IBM Watson AI Lab, IBM Research, 3 Adobe Research
{ |rxtan, bplum, saenko) ) 8bu edu, (hijin, brussell 1 @adobe. com
narrations
Abstract
We introduce the task of spatially localizing narrated interactions in videos. Key to our approach is the ability to learn to spatially localize interactions with self- supervision on large corpus of videos with accompanying transcribed narrations. To achieve this goal, we propose a multilayer cross cross- modal attention network that enables effective optimization of contrastive loss during training. We introduce divided strategy that alternates between computing inter- and intra-modal attention across the visual and natural language modalities, which allows effective training via directly contrasting the two modalities' representations. We demonstrate the effectiveness of our approach by self-training the HowTTo 100M instructional video dataset and evaluating on a newly collected dataset of localized described interactions the YouCook2 dataset. We show that our approach outperforms alternative baselines, including shallow co-attention and full cross-modal attention. We also apply our approach grounding phrases images with weak supervision on Flickr30K and show that stacking multiple attention layers is effective and, when combined with word-to-region loss, achieves state of the art on recall-at-on -t-one and pointing hand accuracies.
Introduction
Content creators often add voice-over narration to their videos to point out important moments and guide the viewer on where to look. While the timing of the narration in the audio track in the video gives the viewer rough idea of when the described moment occurs, there is not explicit information on where to look. We, as viewers, naturally infer this information from the narration and often direct attention to the spatial location of the described moment.
Inspired by this capability, we seek to have recognition system learn to spatially localize narrated moments without strong supervision. We see large opportunity for self-supervision as there is an abundance of online narrated video content, In this work, we primarily focus on spatially localizing transcribed narrated interactions in a video, illustrated in Figure [ (right). Unlike prior phrase grounding work (Figure left), which primarily focuses on matching a noun phrase to an object, our task involves matching entire sentences to regions containing multiple objects and actions.
[
Not only this task integral advancing machine perception of our world where information often comes in different modalities, it also has important implications in fundamental vision-and- lnn-lang research such as roboties, visual question answering, video captioning, and retrieval. Our task challenging as we do not know the correspondence between spatial regions in a video and words in the transcribed narration. Moreover, there is a loose temporal alignment the narration with the described moment and not all words refer to spatial regions (and vice versa). Finally, narrated interactions may be complex with long-range dependencies and multiple actions. In Figure (right), notice how "them" refers to the onions and that there two actions "cut" and "add".
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

"A dog playing with blue ball."
"Cut the onions and add them to the tray."
Phrase Grounding (prior work)
Narrated Interaction Localization (this paper)
Figure Prior work on phrase grounding focuses on localizing single objects in static images (left, example from Flickr30K [40]). In this paper, we propose new task of spatially localizing narrated interactions in video (right). In the task of phrase grounding, the goal is to localize nouns or adjective-noun phrases in images. In contrast, interaction grounding may involve multiple objects and actions and use entire sentences, Our approach self-trains on a large corpus of narrated videos and does not require supervision. (Video credit: Woolworths [57]
Contrastive learning 38 and computing contextual features via an attention model 2 are natural tools for addressing the aforementioned aligmment and long-range cross-modal dependency challenges. While contrastive learning has been successfully applied for temporal alignment of a video clip with narration [35] the alignment of spatial regions with narrations has not been addressed. Naively applying a joint attention model over the set of features for the two modalities, followed by optimizing a contrastive loss over the two modalities' aggregated contextual features, results in fundamental difficulty. Recall that an attention computation comprises two main steps: (i) selecting a set of features given a query feature, and (ii) aggregating the selected features via weighted averaging. By selecting and aggregating features over the two modalities during the attention computation and subsequently optimizing a contrastive loss, the model in theory may select and aggregate features from only one modality to trivially align the contextual representations.
While there are approaches for grounding natural language phrases in images without spatial supervi- sion, the best-performing approaches involve matching individual words to regions [22] or not using any paired text-image supervision all [54]. This strategy is effective for simpler short phrases, as illustrated in Figure I (left). However, consider the interaction "cut the onions and add them to the tray", shown in Figure (right). This interaction is described with more complex, compositional language. In the first two frames, there are multiple "onion" objects visible and the correct one depends on the action being applied. Unlike the task of object grounding, it not sufficient to simply co-localize all of the mentioned objects. We will show that it is difficult to effectively employ strategy that matches individual words to regions to localize such complex interactions.
To address these challenges, we make the following contributions. First, we learn to localize the spatial location of a narrated interaction from abundantly available narrated instructional videos [36). Our approach does not require manually collecting sentence descriptions or the locations of the described interactions for training. Here, the automatically transcribed narrations allow for self-supervised learning via alignment with the video clip.
Second, we propose new approach for directly contrasting aggregated representations from the two modalities while computing joint attention over spatial regions in a video and words in a narration. We show that optimizing loss over the aggregated, sentence-level representation allows for a global alignment of the described interaction with the video clip, and offers an improvement over optimizing matching loss over words and regions. To overcome the network learning a trivial solution while directly contrasting jointly attended features from the two modalities during training, we design network attention layers that do not allow feature aggregation across the two modalities. Our strategy involves alternating network layers that compute inter- and intra-modal attention. Our inter-modal attention layer allows features from one modality to select and aggregate features from only the other modality, and not within its own modality. Our antra-modal attention layer selects and aggregates features from within the same modality. This strategy ensures that the output representations do not aggregate features across the two modalities. In combination with stacking the inter- and intra-modal attention layers, our approach attends jointly and deeply and directly contrasts the resulting contextual representations from the two modalities.
2

---

Finally, we introduce an evaluation dataset that provides bounding box annotations for interactions described by natural language sentences. Our dataset is built on the validation split of the YouCook2 dataset [65] and contains approximately 1000 segments of varying durations. We demonstrate our approach on our collected evaluation dataset of localized interactions and on localizing objects via the YouCook2- BB benchmark where we show that we outperform shallow and full cross-modal attention. We also apply our approach to grounding phrases in images with weak supervision on F0ickr30K and show that stacking multiple attention layers is effective with our loss. Furthermore, we show that our loss is complementary with the word-to-region loss of Gupta et al. [22], and when combined with it, achieves state of the art on recall-at-on and pointing- -hand accuracies.
2 Related Work
Self-supervised learning. There has been significant progress in natural language processing, resulting in effective and robust learned word representations 37 17 60 39] that achieve state-of-the- art performance downstream tasks. Recently, it has also garnered a lot of interest in the computer vision community, achieving state-of-the-ar performance in unsupervised pre-trate pre-training of deep visual models. [TO 24 [I] have demonstrated that effective image representations can be learnt simply by contrasting between augmented views of the same image without labels. [62 45] found that the resulting representations are biased by background pixels and tried to improve foreground object localization via data augmentation or saliency. In video, self-supervised tasks like pace prediction ad future prediction [53] 23 19 26, 6 [55] were used for pre-training, while [2] used self-supervision
on audio-visual data for tasks like unsupervised speaker localization. In this work, we propose a self-supervised method spatially localize activity descriptions in video.
Object and action localization. Approaches generally leverage a region proposal network (RPN) well as region- based convolutional neural network (CNNs) to detect objects localize actions temporally [59] or spatio-temporally [27]. However, they are often trained on curated datasets and are, consequently, limited by fixed predefined number of object categories in these datasets. [65] collected a dataset of cooking videos labeled with descriptions and temporal segments but did not address spatial grounding of activities. Motivated by this limitation, our proposed approach aims to learn to recognize human interactions with objects that belong to the long-tailed distribution from uncurated and unlabeled online videos. There have been video datasets collected with bounding box annotations with associated natural language descriptions 663]. These datasets provide bounding boxes for noun phrases [15] 633 or for egocentric videos [IS]. We collect the first dataset of captioned interactions annotated with spatial bounding boxes for evaluating narrated interactions.
[5 21 20
Vision-and lad-aauugge. Semantic information from natural language has often been exploited to provide an additional source of supervision for learning visual representations. Such approaches have generally leveraged image [2]]BI] 40) and video [5] datasets that are annotated with natural language descriptions to learn a joint embedding space, where visual and language representations of semanticaall-siiiill pairs are close. Prior approaches have mainly focused on retrieving short clips from a large video corpus (35) or localizing relevant segments within untrimmed videos (5) 9 12).
However, they do not identify the relevant spatial locations of the described interactions. Some existing works aim to spatially localize natural language query [44] 61 (30 but focus on objects and train on ground truth bounding boxes, whereas our approach is unsupervised. Object grounding with weak or no supervision is addressed in [12 4 77. To alleviate the costly annotations of curated datasets, (55 have proposed to learn robust video representations as well as an effective joint video- text embedding space from large corpus of unlabeled and uncurated instructional videos [36]. In concurrent work, CLIP [1] trained image-text representations on a dataset of 400 million (image, text) pairs collected from the internet. These efforts to learn from "free" data are similar in spirit to our work, however, we focus on the localization task.
3 Self-Supervised Grounding of Narrated Video Interactions
Given a video clip and a narration, our objective is to learn to localize the relevant spatial regions in the clip that correspond to the interaction described in the narration without localization supervision, e.g., bounding boxes or segmentation masks. To this end, we propose a novel Contrastive Multilayer Multimodal Attention (CoMMA) module that allows for attention interactions between the features
3

---

Contextual Word Features
Spatiotemporal
Contextual
Contrastive Loss
Cross-Modal Attention
Self-Attention
Features
Su..du dy CJ 6y
Sq...ly G G.
Similarity
Cross-modal Attention Layer
Contrastive Multilayer Multimodal Attention
Self-Attention Layer
MLP Projection
Linear Projection
Full-Attention
S: fu Qu.er
Cross-modal Attention Layer
Mixed 5c
Word2Vec
Encoder
Word
Spatiotemporal
Features
Features
Narration
Video Clip
(a)
(b)
(c)
Figure 2: Approach overview. (a) Our proposed Contrastive Multilayer Multimodal Attention (CoMMA) module. (b) Our proposed architecture for computing joint attention over narrations and videos. (c) Binary masks for computing the different types of cross-modal attention. The shaded regions represent "true" Boolean values where query feature may select a key. the mask value is "false", then that corresponding (query, key) pair is not considered in the computation of attention and weighted feature aggregation.
for spatiotemporal regions and words, as well effective contrasting of the two modalities' signals. We define two modalities to have attention interactions when their features attend to each other to compute their final contextualized representations. This setup is in contrast to when the two modalities' features only interact when the training loss is computed (55
Our CoMMA module comprises alternating bidirectional cross-modal attention and self-attention layers (Figure The intuition underlying our proposed approach is that a hierarchical multimodal attention model with multiple stacked layers will encourage fine-grained alignment between video spatiotemporal regions and narration words and ignore irrelevant regions/words. More specifically, each bidirectional cross-modal attention layer computes new representations for a target modality with the latent representations from the source modality to learn the relevance of each spatiotemporal region to each word or phrase, and vice versa. Additionally, the self-attention layer serves identify relevant words and regions by aggregating contextual information between the augmented unimodal features. Through repeated attention interactions of the two modalities and aggregation of unimodal contextual information, our CoMMA module becomes more discriminative of irrelevant regions and words. Moreover, our module allows for applying a contrastive loss the final representations for the two modalities since there is no cross-modal feature aggregation, i.e., a modality may self-select features select features from another modality by attending to them, but it may never select features from both modalities. Our approach goes beyond shallow attention architectures (3] and applying a contrastive loss independent fixed-length features for each modality (i.e., no attention interaction between the two modalities)
[35]. Our multimodal attention module can be easily stacked on top of base feature encoders for our task (Figure 2b) shows our full network). Next, we describe our CoMMA module in detail, how we employ contrastive loss for training, and how we employ our network for the localization task.
3.1 Contrastive Multilayer Multimodal Attention
We formulate each attention layer in our CoMMA module as a key/query/valuee attention mechanism that is commonly used in attention models (52). Our module accepts a concatenated sequence of narration word and video spatiotemporal region features and a set of binary attention masks. Let c, be a spatiotemporal feature for region and sy be word feature for the j-th word in a narration with N words. We obtain features c, and 8j via base feature encoders that are projected to a common embedding space using linear and MLP layers. We form matrix inputs Co and So by stacking the spatiotemporal and word features c, and 8j. respectively, as column vectors.
In Figure 2 c). the shaded regions of the binary masks indicate which features interact during the attention computation for a layer. Specifically, the cross-modal attention (CA) mask MCA computes the attention over the words with respect each region and vice versa. Note that this setup differs

---

from the full-attention (FA) mask M (illustrated in Figure e) bottom) that computes attention over the entire multimodal sequence of words and regions and the self-attention (SA) mask M (illustrated in Figure top-right) that computes attention within each modality. We extend the mechanism of Vaswani et al. [52] by incorporating a mask that modulates the attention computation. Formally, let K, Q. V be the matrices of features for the keys, queries and values, respectively (the individual features are stacked as column vectors). Let the masked attention mechanism be represented as:
(QTK) @ VD
(K, Q. M) = softmax
(1)
where M is any one of the binary masks shown in Figure 2e), D denotes the scalar dimensionality of the query features, and @ is the Hadamard product.
For simplicity, let matrix A = [Co, Su] denote the stacked input features from both modalities. and W denote the projection matrices for keys, queries, and values, respectively, for attention type i and layer I. Here, i can be used to represent cross-attention (CA) or self-attention (SA) layer and is the layer number. We compute the output of the cross-modal attention layer by passing in the initial input features or contextualized feature outputs of the self-attention layer:
In addition, let W K. W
V = Cross = Attntt (W Y,S,, W.A. WCA WVAYY MCA)
(2)
where MCA is the cross-modal attention mask. Note that if the input to Equation (2) is Y,SA [G,, Si], then the output will be YCC |, C4.1), i.e., (contextualized) video features become (contextualized) language features and vice versa. We emphasize that in the cross antss-attntionn layer, there is no mixing of features from different modalities where an elementwise sum is applied to features across the different modalities. Consequently, it allows both modalities to attend to each other without leaking any information between them. We include mathematical formalization of this operation in the supplementary. The similarity scores computed between the queries and the keys as the scaled dot product in Equation ( act as a soft attention mechanism to measure the relevance of each word with respect to region. The the value (words) vectors to compute a region-specifis narration representation. To illustrate this concept, consider a YouTube video clip about cooking fried chicken. The words 'fried' and "cook' will have a higher similarity score to regions in the video clip than 'subscribe' since some regions will be highly relevant to the former. Conversely, the modalities of the keys and queries can be swapped to compute word-specific clip representations.
scores are multiplied with
To aggregate contextual information over the augmented representations of the two modalities, the outputs of the cross-mon modal attention layer are passed into a self-attention layer. The self-attention mask SSA ensures that attention is only computed between elements of the same modality. Formally, this computation can be represented as
(3) where SA is the self-attention mask. Its output is passed into the next cross-attention layer. We denote the final output of our module after L layers as CL and SL where C2 E RXT is the set of contextualized spatiotemporal region features and S, E R DXN is the set of contextualized word representations.
Y Self = Att SAYCA W W BaR/CA,MSM, AMSA + Y
Comparison to multimodal attention models. To evaluate the importance of multimodal attention modules for localizing narrations, we briefly compare CoMMA against other state-of-the-art variants. Akbari et al. (3) is an image-level approach that utilizes features from different levels of an encoder. However, unlike CoMMA. only utilizes single round of cross-modal interaction. Another highly relevant work is the Contrastive Bidirectional Transformer (CBT) [50]. Despite the similarities in attention layers, the CBT model computes self-attention over the entire multimodal sequence (similar to the full-attention mask in Figure 2 ()) and aggregates a summary knowledge into sentinel vector. In contrast, CoMMA allows for repeated attention interactions without mixing of features from different modalities. We show in Section 4 that this mechanism is critical for localization.
3.2 Contrastive loss and inference
We train our proposed multimodal attention module by contrasting between positive and negative pairs of video clips and narrations. Formally, we aim to learn language-aligned visual representations
5

---

for video clips such that features for corresponding videos and narrations are similar and non- corresponding features are dissimilar. Ct.,- denotes the column vector for the final contextualized feature for region r and St.j denotes the column vector for the final contextualized representation for word j. The final video clip representation Ć. and narration sentence representation 5 are computed by mean-pooling over the spatiotemporal regions Ć = R Σ Ct.- and words $ Σ St.j
respectively, Let ( ,S(i)) be the i-th training example pair. We adapt the InfoNCE loss 38 by
defining the sentence loss Lisent as the sum of log ratios over self-training pairs:
exp (C(0) ((()))))))
Lisent
log
X exp
(C0) (C.))) + +
Σ exp ((6) S(m) m
Σ moN.C.
exp (C(m), S(())
(4)
where the negative sets Ni.c and Ni. comprise indices for correeppsnon video clip and narration pairs for the i-th training sample, and n denotes the total number training samples. Note that CoMMA module is applied negative pairs well. In contrast the word-level objective (defined in Section 4 used [2 which contrasts word features separately, we empirically find that contrasting aggregated features in the sentence loss Leent better handles when the temporal alignment between narrations and videos is noisy.
Inference. During inference, given natural language sentence and a video clip, we aim to localize the salient spatial regions across the frames the clip. Beginning from the attention weights of the last cross-modal attention layer, apply attention rollout [1 to obtain the final attention heatmap over all spatiotemporal regions. In attention rollout, the attention weight matrices from all cross- attention and self-attention layers are multiplied recursively to yield the output localization scores A for each spatiotemporal region, where layer. The resulting localization scores aggregate the total amount of attention by the entire set of spatiotemporal and word features assigned a query feature.
I Wi for attention weights W, from the 1-th
4 Experiments
Word-level loss. We experiment with a word-level loss that is used in state-of-the-art phrase ground- ing models [22] 3 which aims to learn an alignment between each word in the narration and all spatial regions. Gupta el al. [22 has demonstrated that the word-level loss is effective at localizing noun phrases in images, which seeks maximize the mutual information between each word and region features for corresponding video clip and narration pairs. We also consider an extension of the word-level loss where we incorporate the output contextualized representations from our CoMMA module. Let N denote the number of words in the narration, and the total number of training samples. s(.), denotes the representation output of our CoMMA for the j-th word of the i-th training sample. Finally, for given input word So." we compute its value representation with multilayer perceptron (MLP): 5/ (L) Then, the word-level loss is formulated as the sum of log ratios:
exp (S()) 5
N log
Lword
(5)
Σ exp (S/m)) 5(00 m~N,
1=1 1=1
exp
where the negative set N₁ of the i-th training sample comprises indices for the word features that attended to by non-corresponding video clips.
4.1 Grounding interactions and objects in video
Self-training and evaluation datasets. We self-train our proposed model on instructional videos from the HowTo100M dataset (36). To reduce the computational burden of training over the entire 100M clips, we identify a set of clips from the dataset that roughly aligns with the YouCook2 dataset. To achieve this goal, we extract list of verb and nouns from the vocabulary of YouCook2 and

---

filter HowTo100M video clips with narrations that contain these words. Our final pre- pre-training set comprises approximately 250,000 video clips. We use the publicly available base video and language feature encoders [35] that are trained on the entire How TowTo100M dataset, which provides a good initialization for our learning task.
For evaluation, we introduce a new evaluation dataset, CouCCook-Interration that provides spatial bounding box annotations for interactions that are described by natural language sentence. Our dataset is built on the validation split of the YouCook2 dataset [65]. We describe our dataset in full in the supplemental. Please see more details of the self- celf-training and our collected dataset in the supplemental. We evaluate using our collected SanCCooo-Innerrctinn dataset for localizing narrated interactions.
In addition to localizing interactions given a sentence narration, we also consider localizing objects in video given single object keyword. For localizing single objects, we evaluate the BouCook2-BB dataset [64]. The YouCook2-BB dataset augments the original YouCook2 dataset with bounding box annotations for object locations. In our experiments on YouCook2-BB, we only evaluate our approach on the validation split since the test split annotations are not readily available.
Evaluation criteria. For both video tasks, we evaluate the quality of the output detections using the pointing hand accuracy criterion. Specifically, given the ground-truth bounding box, we consider our model to have "hit" if the pixel with the highest co-attention similarity score lies within the box. Otherwise, it is "miss". The final localization accuracy is computed as the ratio of hits to the total hits
number of hits and misses # Whits # misses"
Implementation details, We use publicly available implementations for the separable 3D CNN- gated (S3D-G) encoder [58] and shallow language encoder [35] built top of Werd2 [3 embeddings for our video and language models, respectively. To train our proposed model, we set learning rate of le-4 and optimize the model using the AdamW optimizer [32) with one-epoch linear warmup. In our experiments, we also explore using two CoMMA modules but, due to the large memory requirements, the batch size has to be reduced significantly. Results from our initial experiments and prior work on contrastive learning show that a large batch size is crucial to achieving strong performance. More details are in the supplemental.
During inference, since the temporal and spatial dimensions of the attention heatmap have been downsampled from the original input resolution, the localization scores are temporally and spatially interpolated back to the input resolution. We return the final spatiotemporal location as the mode over all output scores.
Baselines. In our evaluations, we compare our approach against center prior, optical flow, and state- of-the-art imwge-grooneooo saagge-round baselines. In the center prior baseline, we return the center pixel for each frame. This is a simple yet competitive bascline that performs well due to the nature of these vidcos, where the main subject is generally in the center of the camera view. As most of the interactions involve motion in cooking videos, we compute an optical flow baseline. For this baseline, we generate an optical flow map for each frame using the OpenCV implementation of the Kanad- optical flow estimation method prediction.
8 33
Finally, we select the pixel with the highest magnitude as the location
Additionally, we compare against four cross-modal attention baselines. First, we compare against the model of Miech el al. [35] adapted our task ("MIL-NCE"). We adapt their publicly available pretrained model by removing the penultimate global average pooling layer in the S3D-G video encoder. Consequently, the output of the video encoder is set of spatiotemporal features that are projected into the joint embedding space. Then, we compute the dot product between the sentence feature and the spatiotemporal features before performing linear interpolation to determine the mode pixel. Second, we compare against baseline that computes full attention and leverages a sentinel vector ("Full attention + sentinel") to predict the similarity of the two modalities' features; an MLP used to convert the output contextualized sentinel vector into a scalar MI-score which is then passed to an InfoNCE-like loss (inspired by CBT (49 ). Third, we compare our multilayer model against a shallow cross-modal attention layer ("Shallow attention") inspired by Akbari el al. (3. To maintain fair comparisons, we implement similar version of their model except we use for the base features the convolutional feature map outputs of the S3D video encoder. Finally, we include comparisons
7

---

Table 1: Interaction localization evaluation on our YouCook2-Interactions dataset. Our ap-
proach outperforms baselines, including shallow and full attention.
Approach
Training Loss
Localization Accuracy (%)
Center Prior
31.81
Optical flow
18.75
MIL-NCE [35]
Sentence level
27.18
Contrastive phrase grounding [22]
Word level
24.04
Shallow Attention
Sentence level
48.30
Full attention sentinel
Sentence level
39.03
CoMMA (Ours)
Word level
34.91
CoMMA (Ours)
Sentence level
55.80
Table 2: Spatiotemporal self-attention ablation and attention layer ablation. (Left) We report an ablation of the different attention module components in our model. We find that multiple stacked layers outperform shallower architectures. (Right) We ablate different types of self-attention spatiotemporal features and find computing full self-attention over spatiotemporal features works best.
Cross- Modal
Cross- Modal Attention 2
Self- Attention
Localization Accuracy (%)
Self-Attention
Localization Accuracy (%)
Type
Attention
Spatial
53.88
X
48.30
Temporal
52.72
51.09
Spatial + Temp.
52.15
55,80
Spatiotemporal 55.80
against a state-of-the-art phrase grounding model [22]. We apply the model out of the box our task with the exception of using our features for fair comparison.
Interaction localization. We report in Table results of our approach and the baselines the task of interaction localization. Our proposed approach outperforms the center prior baseline by significant margin of 20% on the task of localizing interactions given narration sentences. Surprisingly, computing the region with the maximum degree of motion through optical flow does not provide a good proxy for localization. The large performance gain achieved by proposed approach over the optical flow baseline suggests that tracking regions with maximum movement alone is insufficient for observe that our proposed approach outperforms state-of-thee phrase grounding model [22,, which trained via optimizing the word-level loss (Equation[3) (5). by a large margin. One possible reason is that enforcing an alignment between words and regions not suitable for grounding interactions in videos, especially when there is a weak temporal alignment between clips and narrations. This is corroborated by huge drop of 20% in localization accuracy when our CoMMA module is trained with the word-level loss instead.
task. In addition,
The difference in performance obtained by our model and the MIL-NCE baseline suggests that model trained for retrieval may not be focusing on the relevant regions that are described by the sentence narration. Our proposed approach outperforms the "full attention sentinel" baseline by 15%. This result suggests that aggregating contextual information into a sentinel vector causes the model to lose fine-grainess information required for localization. We note that, similar Sun al. [49], this baseline outputs a scalar value that plugged in directly to the InfoNCE loss; it does not compute cosine similarity between the video clip and natural language query contextualized features. We hypothesize that computing this MI-like score (instead of directly contrasting the two modalities' features) may somewhat mitigate this baseline from catastrophic failure where features from both modalities are aggregated during the attention interaction process.
Multilayer attention and spatiotemporal self-attention ablations. We report in Table (left) an ablation over stacking the different attention layers in our network. Notice that the localization accuracy increases as we stack more attention layers in our network, suggesting that deeper cross- modal attention models help improve the alignment between the different modalities. We report in Table 2 (right) results from an ablation of the different forms of self-attention over spatiotemporal regions. It interesting to observe that by applying self-attention to spatial regions alone, the model performs competitively by obtaining localization accuracy of 53.88%. We obtain our best model

---

sscce and toast the bread in the pan"
put grape seed oil into pan add chopped onions and till translucent"
Figure 3: Examples of correct localization predictions by model. Our approach localizes narrated interactions involving a person's hands and interacted objects. The boxes in these visualizations denote the ground-truth bounding boxes for the transcribed narrations. The white circle and cross indicate the locations of the center prior and the mode pixel of the computed attention score map, respectively. Finally, the blue regions indicate the regions that our CoMMA module determines be most relevant to the transcribed narrations. (Video credit: Woolworths 57 and Alícia Kirby (28)
when self-attention is computed over all spatiotemporal regions. This finding suggests that reasoning about temporal context is important but has be done in conjunction with spatial context reasoning.
Object localization. Besides localizing full narrated sentences, we also evaluate our model's capa- bility localize single objects. Specifically, we compare our approach against the state-of-the-an "not all frames are equal" (NAFAE) model [47]. We report results in Table 3 Despite being trained with sentence-level loss, we observe in Table 3 that our approach successfully localizes objects corresponding to single noun words. Notably, our proposed approach outperforms the NAFAE baseline. We give more details of the NAFAE baseline in the supplemental.
Qualitative results. We show qualitative results in Figure 3 Each frame overlaid with the computed attention score map to depict the visual cues that model uses to make its prediction. The white circle indicates the location the center prior and the cross denotes the mode pixel of the computed attention score map. Notice how our approach correctly localizes the described interaction.
Table 3: Object localization evaluation the YouCook2-BB dataset. By self-training on nar- rations, our approach learns to lo- calize objects and outperforms the baseline.
Approach NAFAE 47
Full Loc. (%) 46.95
4.2 Weakly supervised phrase grounding in images
Ours
59.25
We evaluate our approach on the task of weakly supervised phrase grounding to determine its capability to generalize to localizing noun phrases in images. For this task, adopt the same setup in prior work [22 16 [8 by training MSCOCO 31 and evaluating on Flickr30k [40]
We report results on the standard recall-at-K K (R (R@K) and pointing hand recall accuracy criteria 22 For training, we optimize the sum of the word-to-r ro-dton- matching loss and our sentence loss: [()) = Lword + ALsent- We set À to be 0,005 in our experiments. We compare our approach against state-of-the-art phrase grounding models. Details how the model [2 extended with our CoMMA architecture well as a description of the baselines are included in the supplemental. We note that the value projection layer for region features in (22 is omitted to incorporate CoMMA. Finally, we build the same set of visual and language features from 22 for fair comparison.

---

Table 4: Weakly supervised phrase grounding experiments on Flickr30K. Adding additional
attention layers improves performance on the R@j I and pointing recall criteria. We include a
description of the visual features and baseline approaches in the supplemental. IN and VG denote
ImageNet and Visual Genome respectively.
Approach
Visual Features
R@1
R@5
R@ 10
Pt Recall
Fang et al [18]
VGG-cls (IN)
29.00
Akbari et al [3]
VGG-cls (IN)
61.66
Akbari et al [3]
PNAS Net (IN)
69.19
Align2Grround [16]
Vaster-RCNI (VG)
71.00
Gupta et al [22
Fasicr-RCNN (VG)
51.67
77.69
83.25
76.74
ours (word + sent): CA layer
Faster-RCNy (VG)
49.99
76.72
83,51
74.97
ours (word sent): 2 CA layers
Faster-RCNN (VG)
53,80
76.69
82.28
76.78
Results and analysis. We report results in Table 4 As mentioned above, omitting the region value projection layer results a performance drop, as evidenced by the results obtained by using a single attention layer. Adding second cross-attention layer generally helps improve the localization accuracy and leads to notable increases in R@! I and pointing hand recall accuracies, surpassing state-of-ther to-tr--rttt [22] on those criteria. We note that our full CoMMA model with unimodal self-attention layers hurts performance (R @1 = 48.81 (8)..)). %). One possible reason is that optimizing CoMMA on this task is challenging and it may be overfitting. However, it is very promising that adding more layers works out of the box in general on a task that is typically addressed by shallow attention models. In contrast, we show in the supplemental that stacking multiple layers in our default CoMMA architecture when trained with the sentence-level loss Leent only results in improved R @ K localization accuracies.
Finally, we also observe that, unlike the observations in the video experiments, the word-level loss is critical for phrase grounding in images. We hypothesize that this finding is result of the captions consisting primarily of noun phrases and being strongly correlated with the images. In contrast, the video task requires spatially localizing different interactions that are mentioned in a narration, which are more semantically complex as compared to noun phrases, making the word loss less effective.
Discussion. This work introduces the problem of localizing narrated interactions in uncurated videos without localization supervision. To address this task, we propose a novel Contrastive Multilayer Multimodal Attention module that facilitates repeated attention interactions between spatiotemporal regions and words to learn their latent alignment. Additionally, we introduce new evaluation dataset that provides spatial annotations for narrated instructions in videos. Evaluations on the three separate tasks of interaction and object localization in videos as well as phrase grounding in images demonstrate the ability of our model to ground both words and phrases more effectively than the shallow and full-attention baselines. Our approach opens up the possibility of further exploration in other modalities. Learning to associate relevant spatial regions with natural language sentences may also be beneficial to learning richer video representations, particularly with the recent gravitation towards self- supervuee representation learning.
References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00 00928, 2020.
[2] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. ECCV, 2020.
[3] Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fi Fu Chang Multi-level multimodal common semantic space for image-phrase grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 02476-12486 2019.
[4] Elad Amrani, Rami Ben-Ari, Inhar Shapira, Tal Hakim, and Alex Bronstein. Self-superviser object detection and retrieval using unlabeled videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 954-955. 2020.
[5] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812 2017.
[6] Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9922-9931, 2020.
10

---

[7] Gedas Bertasius and Lorenzo Torresani. Cobe: Contextualized object embeddings from narrated instrue- tional video. arXiv preprint (xXiv:0007007300 2020.
[8] G. Bradski. The OpenCV Library. Dr: Dobb's ss Journal of Software Tools, 2000.
[9] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. Temporally grounding natural
sentence in video. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 162-171, 2018
[10] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages
1597-1607. PMLR, 2020.
[11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003, 044,4477, 2020.
[12] Zhenfang Chen, Lin Ma, Wenhan I Luo, and Kwan- Yee K Wong. Weakly-superviiee spatio-temporally
grounding natural sentence in video. arXiv preprini 1906 2019.
[13] Collard Valley Cooks. ww wwwpuueee.com//oo/ comm/atch?v==jeee
http:///ww..oouubbb com http///thhhhhhaaattttte
[14] Curiosity Culture. https://www. youtube,
[15] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic-kitchent dataset: Collection, challenges and baselines. IEFE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2020.
[16] Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. Align2ground: Weakly supervised phrase grounding guided by image-caption alignment. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 121000226,,,200 2019.
[17] Jacob Devlin, Ming- Benn-Wei&& Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1etp:0/0000000 2018.
[18] Hao Fang. Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong
He. Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1473-1482. 2015.
[19] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pages 3636-3645, 2017.
[20] Ross Girshick. Fast r-cnn. In Proceedings of the IEFF international conference on computer vision, pages
1440-1448, 2015.
[21] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedingy of the IEEE conference on computer vision and pattern recognition. pages 580-587, 2014.
[22] Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Contrastive learning
for weakly supervised phrase grounding. arXiv preprint arXiv:2006.0900 2020.
[23] Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmenter dense predictive coding for video representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August
23-28, 2020, Proceedings, Part III 16, pages 312-329. Springer, 2020.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised
visual representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patern Recognition, pages 1729-9738.77 2020.
[25] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMIRR 2015.
[26]
Allan Jabri. Andrew Owens, and Alexei A Efros. Space-time correspondence as a contrastive random walk.
arXiv preprint arXiv:2006.1 1 14613, 2020.
[27] Vicky Kalogciton, Philippe Weinzaepfel, Vittorio Ferrari, and Cordelia Schmid, Action tubelct detector for spatio-temporal action localization. In Proceedings of the IEEE International Conference Computer Vision,
pages 4405-4413, 2017.
[28]
Alicia Kirby. https://www.wy www.yoouuue.com..e wwtpe//oo/wwwcch//eeeeem
oh8a.
[29]
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yan- nis Kalantidis. Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017. WM Smeulders. Tracking by natural
[30] Zhenyang Li. Ran Tao, Efstratios Gavves, Cees GM Snoek, and language specification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6495-6503, 2017.
[31]
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan. Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference computer vision,
pages 740-755. Springer. 2014.
[32] flya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 171.::1111.050011
2017.
[33] Bruce D Lucas. Takeo Kanade, al. An iterative image registration technique with an application to stereo vision. Vancouver, British Columbia, 1981
[34] Kharma Medic. http:///wwyooutbbb www.youuube.oowwwaatet http///athh/bbbb//rrrrrr
[35] Antoine Miech, -can-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivie, and Andrew Zisserman. End-to- end leaming of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF
11

---

Conference on Computer Vision and Putter Recognition, pages 9879-9889, 2020.
[36] Antoine Miech, Dimitri Zhukov, Jean- can-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howto 100:: Leaming text-video embedding by watching hundred million narrated vidco clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-26440, 2019.
[37] Tomas Mikolov, Kai Chen, Greg Corrado, and Jettrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1 10.::1311.78.. 2013,
[38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.
arxiv
p preprint arXiv: htt.:/887707444 2019.
[39] Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.0533 2018.
[40] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik, Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision. pages 2641-2649, 2015.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Hyya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[42]
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv, 2018.
[43]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection
with region proposal networks. In Advances Neural Information Processing Systems, 2015.
[44]
Arka Sadhu, Kan Chen, and Ram Nevatia. Video object grounding using semantic roles in language description. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2020. Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Learning to localize improves self-supervised representations. arXiv preprint arXiv:2012.00 2012.046300 2020.
[45]
[46]
Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. 2020.
[47] Jing Shi, Jia Xu, Boqing Gong, and Chenliang Xu. Not all frames are equal: Weakly-superviiee video grounding with contextual similarity and visual clustering losses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10444-10452, 2019.
[48] Six Sisters' Stuff.
[49]
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive hidirectional transformer for
temporal representation leamning In arXiv, June 2019.
[50] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint ar /906 05743, 2019.
[51] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision, 104(2):154-177 2013.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, arXiv preprint 0ttb::/00.03766 2017.
[S3] Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervieed video representation learning by pace prediction. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Man-Michael Frahm, editors, Computer Vision ECCV 2020, 2020.
[54] Josiah Wang and Lucia Specia. Phrase localization without paired training examples. In Proceedings of the
IEEE international conference on computer vision, 2019.
[55]
Donglai Wei, Joseph Lim. Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. Computer Vision and Puttern Recognition, pages 8052-8060, 2018. Smokin' Grillin' wit AB. https://www.j wttps///ww.youuubb...mm
In Proceedings of the IEEE Conference
[56]
- m.
[57]
Waolworths. https://ww..yuuthbe ceattuub.com/wwee. wwt.w////////ooooo hoom/wtch?vvv www.wttthhwwwmyiii...
[58]
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature
learning: Speed-accuracy trade-offs in video classification. In European Conference on Computer Vision, 2018. Huijuan Xu, Abir Das, and Kate Saenko. R-c3d: Region convolutional 3d network for temporal activity detection.
[59]
In Proceedings of the IEEE international conference computer vision, pages 5783-5792, 2017.
[60]
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: 1906.08237 2019.
[61] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio- temporal video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10668-10677, 2020.
[62] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. Distilling localization for sufe-super representation learning. arXiv preprint Stt1:2004.066333 2020.
[63] Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. Grounded video description.
In CVPR, 2019.
[64]
Luowei Zhou, Nathan Louis, and Jason Corso. Weakly-supervisee vidco object grounding from text by loss weighting and object interaction. In British Machine Vision Conference, 2018.
[65] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional vidcos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
12


Reviews:
[{"review_id": "Qvd7wyHpVk4", "reviewer": "Reviewer_rFC4", "summary": "The paper proposes a method for grounding sentences in videos and images without explicit localization supervision.  The approach rely on a cross-attention architecture that is used in a way to make contrastive learning possible (since it prevents the final feature from containing information from the other modalities while still enabling information to be shared across modalities earlier in the network). Localization can be obtained from the cross-attention patterns between the language and the visual signal.\n\nThe authors apply their method on a subset of the HowTo100M dataset for videos and evaluate on the YouCookBB object for objects and a newly introduced YouCookInteraction where they show improvements over baselines. The method is also applied on the Flickr30K dataset on images.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- The paper tackles an important problem (localization without strong supervision)\n- The proposed architecture is interesting as it enables to have early fusion of text and visual features while still allowing contrastive learning to be used.\n- The quantitative numbers are good\n\n## Weaknesses\n\n- **Important technical clarifications**: there are a few key questions that needs clarifications as its not completely clear from the paper.\n  - *Number of layers?* When talking about layers, if I understand correctly you refer to whether or not there is one or two CA block in the CoMMA layer? Have you also explored using more than a single CoMMA blocks? \n  -  *Resolution of input videos and feature maps*: What is the size of the input videos in practice and the final feature map?\n  - *Word level loss*: The word level loss description and equation are confusing. The text says `between each word in the narration and all spatial regions`. However the word level loss only contains features related to text `S^T_{fj}`. Is that because these features are extracted after the first cross attention layer and hence effectively contains visual features? This is not clear at all from the description. Figure (4) (b) was helping to understand this in suppmat but I am unsure this is the same architecture used for videos. \n  - CoMMa architecture figure: why is there a + sign at the top of the figure? If I understand correctly it is important to not have residual connection to avoid information being leaked from one modality to another which would make contrastive learning trivial.\n  - CBT baseline: When using full self attention, isn't there the problem mentioned in L42-48 as it allows final features to have both image and video information hence making the contrastive task trivial to solve? How come this does not affect that much the performance? This require a more precise description as it is currently not clear how this baseline can really work. \n  - Equations (5): there is no dependence in $i$ in the loss. Some index are probably missing? Similar comment applies to equation (6).\n\n- **Conclusion between video and image results do not seem to align**: It seems that the word loss is very important for the flick30K experiment but is not working for the video case. Similarly the number of cross attention layers seem to be optimal at 2 for videos while for images it depends on the metrics. Why does this discrepancy exist? \n\n- **Additional baseline request without cross attention mechanism**: for the video model it would be interesting to compare to the localization ability of the S3D-G pretrained model you use from [24]. For that a simple modification can be done by simply removing the average pooling [here](https://github.com/antoine77340/MIL-NCE_HowTo100M/blob/master/s3dg.py#L342) hence obtaining a spatio-temporal grid of features similar to the one used in the work. Then the fully connected layer can simply be applied on the grid of features to obtain a grid of feature living in the joint text-video space (see [here](https://github.com/antoine77340/MIL-NCE_HowTo100M/blob/master/s3dg.py#L345)). This will result in a matrix of visual features of shape `V= [X, d]` where `X` is the number of spatio temporal features and `d` is the dimension of the joint text-video space. Finally to obtain the scores for each feature in the grid given a text query, one can simply do a matrix multiplication with the text vector of the query `y` of shape `[1,d]` by doing `np.matmul(V, y.T)`. Then the same evaluation method can be applied. This would give another interesting point of comparison.\n\n- **Minor comments**: \n  - L296-L297: Table (left) should be Table (right), \n  - nit the Tables could be made more pretty, e.g. following guidance from [here](https://people.inf.ethz.ch/markusp/teaching/guides/guide-tables.pdf)\n  - Typo: inclided L328\n\n## Overall assessment \n\nThe paper presents an interesting approach for an important problem. However some important things require additional clarifications. For this reason I am putting a borderline score for now (slightly leaning towards reject but I can update my score based on the rebuttal).\n\n## Post reading reviews and responses\n\nI read the other reviews and responses. \n\n## Post Rebuttal comment\n\nI would like to thank the authors for their effort in the rebuttal. I very much agree with what Reviewer gTL6 said, i.e. in the current version there are lots of things to incorporate for the camera ready to be really good. \n\nThat being said, given the engagement of the authors in the discussion, especially the clarifying discussion about the notations, I am ready to raise my score to accept (7) and be optimistic that the authors can include all the recommendations reviewers made to the final version of the paper. In particular I found the idea of cross attention quite clever and it also seems to work well (but it has to be clearly presented for it to be understood and impactful in the community), hence my leap of faith in the authors for this paper.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "KiiqjHbTwQq", "reviewer": "Reviewer_Broi", "summary": "This paper concerns a new problem of grounding narrated interactions in video. It is similar to existing work on (object) phrase localization/grounding, but with a twist on grounding more complex narrations that could involve both entities and predicates (e.g., put tomatoes into the baking tray). The model training is performed in a weakly supervised fashion on a subset of HowTo100M. Training loss is the standard InfoNCE loss at the sentence level (vs. word-level as in existing grounding work). Model architecture is designed to capture inter-modal correspondence through cross-modal attention. Evaluation is conducted on YouCook2-BB, Flickr30k-Entities, and a newly collected dataset for narration grounding called YouCook2-Interaction.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the work comes from the proposed weakly-supervised grounding framework. It demonstrates that i) with a carefully-designed decoupled multimodal interaction layer (the bottom line of Tab. 2, right), grounding performance can be greatly enhanced, ii) and sentence-level contrastive loss has an edge over the traditional word-level contrastive loss. Experimental results and ablation studies are overall comprehensive.\n\nIn terms of the weaknesses of the work, the paper can be significantly benefited from better clarity, with main technical contributions highlighted. In the current shape, the paper attempts to motivate with the new problem setting (and dataset), which is somewhat incremental considering existing work and complicates the storyline. More detailed comments are as follows.\n\ni) The motivation is somewhat disconnected from the method itself. For example, at L54, the paper mentioned that \"[...] there are multiple 'onion' objects visible and the correct one depends on the action being applied.\" However, the paper has not addressed this problem particularly but instead proposes a rather generic approach for grounding. Also, the YouCook2-BB benchmark adopted in the experiment is agnostic to if the action is applied (i.e., 'onion' will be annotated throughout the video as long as it's visible and will be cut at some point).\n\nii) The pointing hand evaluation metric is somewhat weak. Consider proposal-based metrics. Also, in L257, what if there are multiple boxes match the same narration?\n\niii) The proposed method is based on a well-trained model (appear to be MIL-NCE? More details should be included in the main paper) and further refined on a well-curated dataset for the downstream task. It's not surprising that it performs well (which is somewhat unfair to methods without access to these resources). More analyses need to be conducted on the impact of each factor. \n\niv) Minor comments:\n\nL227, should \"for the j-th word\" be \"region\" instead?\n\nL230, \"self-training\" should be \"pre-training\". \"Self-training\" usually refers to a training scheme that leverages generated pseudo labels. \n\nL277, state-of-the-art.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Appear to be.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "FCqpcvniIck", "reviewer": "Reviewer_q3gz", "summary": "- This paper considers the task of spatially grounding the language description in videos, specifically, instructional videos are used to train the model, where the language description and visual signals are likely to be synchronised.\n\n- The proposed architecture consists of stacks of intra- and inter attention modules, which allows the video and language to attend each other at early stage, and effectively highlighting or suppressing the corresponding features.\n\n- An evaluation dataset is proposed with bounding box annotations for interactions described by natural language sentences, and proposed approach shows good performance on it. In addition, authors have also evaluated on Flickr30K under the setting of weakly supervised language grounding, showing state-of-the-art performance.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\n- The proposed problem in this paper is very challenging, as it aims to train models with the raw instructional videos without manual box annotations.\n\n- The proposed architecture and training loss are adopted from other papers in the literature, thus, I would say there is no novelty on the computational module and loss function, however, I do think these elements are used in a good way here, and aiming to tackle and challenging problem. \n\nQuality:\n\n- The paper is of good quality, above the acceptance threshold of NeurIPS.\n\nClarity:\n\n- The paper is well-written, easy to follow. One suggestion would be, try to clarify the Eq. 6 a bit more.\n\n- In terms of the related work, the section on self-supervised learning is not satisfactory, since this paper is on grounding actions in video, I would suggest the authors to include more recent papers on self-supervised video representation learning, there is a rich line of work on this topic, while the authors only mention the pace prediction work.\n\nSignificance:\n\n- The proposed idea can be potentially useful for visual grounding, visual question answering, and even generic object detector, thus I think it is a significant work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "12eP8XtCCw7", "reviewer": "Reviewer_gTL6", "summary": "The paper presents an approach for self-supervised spatial localization of interactions in videos by leveraging a large corpus of video clips and narrations. An alternating attention mechanism is presented that combines cross-modal and self (within modality) attention schemes. Pre-trained models on the HowTo100M dataset are evaluated by using pointing hand accuracy on a subset of the YouCook2 dataset, interaction annotations are also created in this work. The approach is also applied to grounding in Flickr30k images.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths:*\n1. Interesting work on leveraging the power of large datasets in a consistent domain (here, cooking related videos) to help learn spatial localization.\n2. Fig. 2 explains the proposed approach in a very clear way. In fact, I would recommend reducing the size of Section 3.1 since the figure is very clear and Transformers have made this type of attention common and instead focus on Attention Rollout used during inference + additional experiments / clarifications. The current explanations in 3.1 are also causing some confusion (see below).\n\n*Weaknesses:*\n1. Claims:  \na) Lines 118-119 indicate that the proposed task is different from object grounding as it requires localizing interactions. However, I'm not sure why the proposed model would be good at localizing specific interactions such as \"add onions to tray\" and not just co-localize the objects \"onions\" and \"tray\".  \nb) Fig. 1 (right), seems to indicate that there are multiple instructions that are localized in various spatio-temporal parts of the video. However, none of the qualitative examples highlight the textual part. In fact, the rest of the paper seems primarily interested in spatial localization, not temporal. This is quite confusing and misleading.  \nc) I'm not sure why the paper claims \"early interactions\" (L46, L132), or \"deep cross-modal attention\" (L137) when as far as I understand, the module from Fig. 2c is applied to the final layer of features extracted from a CNN. See weakness 2c below related to this.\n\n2. Models and ablations:  \na) Word-level loss, Eq. 6. Firstly why is this used? Second, the paragraph talks about \"aims to learn an alignment between each word in the narration and all spatial regions\". However, the attention is computed between $S_j$ and $S_{fj}$ instead of $C_{fix}$. Do the authors wish to indicate that $S_{fj}$ through multiple layers of cross-modal attention already captures information in the spatial regions, or is this a typo?  \nb) Based on Fig. 2c, the extracted features go through a cross-modal, self-attention, and another cross-modal block. In my understanding, all this put together is one CoMMA layer. Eq. 2 and 3 seem like intermediate stages of one CoMMA layer should not use the same notation $Y_{i+1}$.  \nc) L198 states that \"CoMMA allows for repeated early interactions without mixing of features from different modalities\", but the first attention block in Fig. 2c is in fact a cross-modal attention. How is this explained?  \nd) Would stacking CoMMA layers on top of each other help (alternating cross and self attention)?\n\n\n3. Experiments:  \na) While the center prior is a good simple baseline, based on the qualitative examples, I wonder how well would a method that simply predicts the center of the \"active\" hand perform. For example, [A] can provide localization of the interacting object and hand quite well, and it seemed like most GT boxes were inside this region.  \nb) Alternatively, to truly justify the model's capability of learning spatiotemporal attention, it might be interesting to try it on videos which don't have human actors as the prominent agent.  \nc) This work seems to be quite related to the study of referring expressions in videos (VID-Sentence [B], Lingual OTB [21]). These datasets could act as other means to evaluate the proposed approach.  \nd) Results on Flickr30k are not coherent with YouCook2-Interactions, adding to the confusion of the contribution. The whole message changes here, as using self-attention hurts the performance quite drastically (48.8% R@1 with CoMMA block, 53.8% without self-attention in CoMMA block).  \ne) Minor: Table 2, left/right descriptions need to be interchanged (L296-305).\n\n[A] Shan, et al. Understanding Human Hands in Contact at Internet Scale. CVPR 2020.  \n[B] Chen, et al. Weakly-supervised spatio-temporally grounding natural sentence in video. ACL 2019.\n\n\n### Post rebuttal\n**TL:DR; I've increased my rating to 6 (from 4) with some strong conditions**\n\nI thank the authors for their detailed response. Especially appreciate the clarifications on several aspects of the submission, in particular, resolving the superscripts with reviewer rFC4. That would make it much easier to read the paper.\n\nI also really appreciate the additional effort taken in evaluating hand detection models from [A] on the proposed dataset, and including a thorough evaluation of three settings. I agree with the authors that this can be thought of as a supervised method albeit, important to note that it is trained on a proxy task of detecting hands and objects and not actual interactions indicated in the text.\n\nWhile the rebuttal claims that the proposed approach would learn interactions despite the presence of human actors, I have my doubts regarding this, especially when so much of the data points to looking at the hands. It would be really nice to include some qualitative examples where the GT is not close to the hands, and perhaps even a quantitative split by finding samples where hands are not detected. In a future work, I would also encourage the authors to look at using data beyond the human interactions typically present in HowTo100M (think documentaries), where interaction localization is much more tricky and non-obvious.\n\nWhile the rebuttal is very promising, there are a lot of small and big issues to be improved in the submission before it can be accepted. The very fact that such a detailed rebuttal was necessary in the first place is a little concerning. I'm happy to increase my rating to 6, conditional on all the changes being incorporated, but cannot champion it in the current version. One additional round of review may not hurt much.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim that they will release code judiciously to prevent \"further development of surveillance and tracking technology\". However, it's not clear how this will be executed.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has a very nice insight, and its essence has been made much clearer by the discussion and new notation during the rebuttal discussions. The two reviewers who originally recommended rejection have upgraded their scores to accept following these discussions.

The authors have promised to make many updates to the paper, and should take note of the post-rebuttal statements of the reviewers.
Decision: Accept (Spotlight)