Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method for grounding sentences in videos and images without explicit localization supervision.  The approach rely on a cross-attention architecture that is used in a way to make contrastive learning possible (since it prevents the final feature from containing information from the other modalities while still enabling information to be shared across modalities earlier in the network). Localization can be obtained from the cross-attention patterns between the language and the visual signal.

The authors apply their method on a subset of the HowTo100M dataset for videos and evaluate on the YouCookBB object for objects and a newly introduced YouCookInteraction where they show improvements over baselines. The method is also applied on the Flickr30K dataset on images.
main_review: ## Strengths

- The paper tackles an important problem (localization without strong supervision)
- The proposed architecture is interesting as it enables to have early fusion of text and visual features while still allowing contrastive learning to be used.
- The quantitative numbers are good

## Weaknesses

- **Important technical clarifications**: there are a few key questions that needs clarifications as its not completely clear from the paper.
  - *Number of layers?* When talking about layers, if I understand correctly you refer to whether or not there is one or two CA block in the CoMMA layer? Have you also explored using more than a single CoMMA blocks? 
  -  *Resolution of input videos and feature maps*: What is the size of the input videos in practice and the final feature map?
  - *Word level loss*: The word level loss description and equation are confusing. The text says `between each word in the narration and all spatial regions`. However the word level loss only contains features related to text `S^T_{fj}`. Is that because these features are extracted after the first cross attention layer and hence effectively contains visual features? This is not clear at all from the description. Figure (4) (b) was helping to understand this in suppmat but I am unsure this is the same architecture used for videos. 
  - CoMMa architecture figure: why is there a + sign at the top of the figure? If I understand correctly it is important to not have residual connection to avoid information being leaked from one modality to another which would make contrastive learning trivial.
  - CBT baseline: When using full self attention, isn't there the problem mentioned in L42-48 as it allows final features to have both image and video information hence making the contrastive task trivial to solve? How come this does not affect that much the performance? This require a more precise description as it is currently not clear how this baseline can really work. 
  - Equations (5): there is no dependence in $i$ in the loss. Some index are probably missing? Similar comment applies to equation (6).

- **Conclusion between video and image results do not seem to align**: It seems that the word loss is very important for the flick30K experiment but is not working for the video case. Similarly the number of cross attention layers seem to be optimal at 2 for videos while for images it depends on the metrics. Why does this discrepancy exist? 

- **Additional baseline request without cross attention mechanism**: for the video model it would be interesting to compare to the localization ability of the S3D-G pretrained model you use from [24]. For that a simple modification can be done by simply removing the average pooling [here](https://github.com/antoine77340/MIL-NCE_HowTo100M/blob/master/s3dg.py#L342) hence obtaining a spatio-temporal grid of features similar to the one used in the work. Then the fully connected layer can simply be applied on the grid of features to obtain a grid of feature living in the joint text-video space (see [here](https://github.com/antoine77340/MIL-NCE_HowTo100M/blob/master/s3dg.py#L345)). This will result in a matrix of visual features of shape `V= [X, d]` where `X` is the number of spatio temporal features and `d` is the dimension of the joint text-video space. Finally to obtain the scores for each feature in the grid given a text query, one can simply do a matrix multiplication with the text vector of the query `y` of shape `[1,d]` by doing `np.matmul(V, y.T)`. Then the same evaluation method can be applied. This would give another interesting point of comparison.

- **Minor comments**: 
  - L296-L297: Table (left) should be Table (right), 
  - nit the Tables could be made more pretty, e.g. following guidance from [here](https://people.inf.ethz.ch/markusp/teaching/guides/guide-tables.pdf)
  - Typo: inclided L328

## Overall assessment 

The paper presents an interesting approach for an important problem. However some important things require additional clarifications. For this reason I am putting a borderline score for now (slightly leaning towards reject but I can update my score based on the rebuttal).

## Post reading reviews and responses

I read the other reviews and responses. 

## Post Rebuttal comment

I would like to thank the authors for their effort in the rebuttal. I very much agree with what Reviewer gTL6 said, i.e. in the current version there are lots of things to incorporate for the camera ready to be really good. 

That being said, given the engagement of the authors in the discussion, especially the clarifying discussion about the notations, I am ready to raise my score to accept (7) and be optimistic that the authors can include all the recommendations reviewers made to the final version of the paper. In particular I found the idea of cross attention quite clever and it also seems to work well (but it has to be clearly presented for it to be understood and impactful in the community), hence my leap of faith in the authors for this paper.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: - This paper considers the task of spatially grounding the language description in videos, specifically, instructional videos are used to train the model, where the language description and visual signals are likely to be synchronised.

- The proposed architecture consists of stacks of intra- and inter attention modules, which allows the video and language to attend each other at early stage, and effectively highlighting or suppressing the corresponding features.

- An evaluation dataset is proposed with bounding box annotations for interactions described by natural language sentences, and proposed approach shows good performance on it. In addition, authors have also evaluated on Flickr30K under the setting of weakly supervised language grounding, showing state-of-the-art performance.
main_review: Originality:

- The proposed problem in this paper is very challenging, as it aims to train models with the raw instructional videos without manual box annotations.

- The proposed architecture and training loss are adopted from other papers in the literature, thus, I would say there is no novelty on the computational module and loss function, however, I do think these elements are used in a good way here, and aiming to tackle and challenging problem. 

Quality:

- The paper is of good quality, above the acceptance threshold of NeurIPS.

Clarity:

- The paper is well-written, easy to follow. One suggestion would be, try to clarify the Eq. 6 a bit more.

- In terms of the related work, the section on self-supervised learning is not satisfactory, since this paper is on grounding actions in video, I would suggest the authors to include more recent papers on self-supervised video representation learning, there is a rich line of work on this topic, while the authors only mention the pace prediction work.

Significance:

- The proposed idea can be potentially useful for visual grounding, visual question answering, and even generic object detector, thus I think it is a significant work.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents an approach for self-supervised spatial localization of interactions in videos by leveraging a large corpus of video clips and narrations. An alternating attention mechanism is presented that combines cross-modal and self (within modality) attention schemes. Pre-trained models on the HowTo100M dataset are evaluated by using pointing hand accuracy on a subset of the YouCook2 dataset, interaction annotations are also created in this work. The approach is also applied to grounding in Flickr30k images.

main_review: *Strengths:*
1. Interesting work on leveraging the power of large datasets in a consistent domain (here, cooking related videos) to help learn spatial localization.
2. Fig. 2 explains the proposed approach in a very clear way. In fact, I would recommend reducing the size of Section 3.1 since the figure is very clear and Transformers have made this type of attention common and instead focus on Attention Rollout used during inference + additional experiments / clarifications. The current explanations in 3.1 are also causing some confusion (see below).

*Weaknesses:*
1. Claims:  
a) Lines 118-119 indicate that the proposed task is different from object grounding as it requires localizing interactions. However, I'm not sure why the proposed model would be good at localizing specific interactions such as "add onions to tray" and not just co-localize the objects "onions" and "tray".  
b) Fig. 1 (right), seems to indicate that there are multiple instructions that are localized in various spatio-temporal parts of the video. However, none of the qualitative examples highlight the textual part. In fact, the rest of the paper seems primarily interested in spatial localization, not temporal. This is quite confusing and misleading.  
c) I'm not sure why the paper claims "early interactions" (L46, L132), or "deep cross-modal attention" (L137) when as far as I understand, the module from Fig. 2c is applied to the final layer of features extracted from a CNN. See weakness 2c below related to this.

2. Models and ablations:  
a) Word-level loss, Eq. 6. Firstly why is this used? Second, the paragraph talks about "aims to learn an alignment between each word in the narration and all spatial regions". However, the attention is computed between $S_j$ and $S_{fj}$ instead of $C_{fix}$. Do the authors wish to indicate that $S_{fj}$ through multiple layers of cross-modal attention already captures information in the spatial regions, or is this a typo?  
b) Based on Fig. 2c, the extracted features go through a cross-modal, self-attention, and another cross-modal block. In my understanding, all this put together is one CoMMA layer. Eq. 2 and 3 seem like intermediate stages of one CoMMA layer should not use the same notation $Y_{i+1}$.  
c) L198 states that "CoMMA allows for repeated early interactions without mixing of features from different modalities", but the first attention block in Fig. 2c is in fact a cross-modal attention. How is this explained?  
d) Would stacking CoMMA layers on top of each other help (alternating cross and self attention)?


3. Experiments:  
a) While the center prior is a good simple baseline, based on the qualitative examples, I wonder how well would a method that simply predicts the center of the "active" hand perform. For example, [A] can provide localization of the interacting object and hand quite well, and it seemed like most GT boxes were inside this region.  
b) Alternatively, to truly justify the model's capability of learning spatiotemporal attention, it might be interesting to try it on videos which don't have human actors as the prominent agent.  
c) This work seems to be quite related to the study of referring expressions in videos (VID-Sentence [B], Lingual OTB [21]). These datasets could act as other means to evaluate the proposed approach.  
d) Results on Flickr30k are not coherent with YouCook2-Interactions, adding to the confusion of the contribution. The whole message changes here, as using self-attention hurts the performance quite drastically (48.8% R@1 with CoMMA block, 53.8% without self-attention in CoMMA block).  
e) Minor: Table 2, left/right descriptions need to be interchanged (L296-305).

[A] Shan, et al. Understanding Human Hands in Contact at Internet Scale. CVPR 2020.  
[B] Chen, et al. Weakly-supervised spatio-temporally grounding natural sentence in video. ACL 2019.


### Post rebuttal
**TL:DR; I've increased my rating to 6 (from 4) with some strong conditions**

I thank the authors for their detailed response. Especially appreciate the clarifications on several aspects of the submission, in particular, resolving the superscripts with reviewer rFC4. That would make it much easier to read the paper.

I also really appreciate the additional effort taken in evaluating hand detection models from [A] on the proposed dataset, and including a thorough evaluation of three settings. I agree with the authors that this can be thought of as a supervised method albeit, important to note that it is trained on a proxy task of detecting hands and objects and not actual interactions indicated in the text.

While the rebuttal claims that the proposed approach would learn interactions despite the presence of human actors, I have my doubts regarding this, especially when so much of the data points to looking at the hands. It would be really nice to include some qualitative examples where the GT is not close to the hands, and perhaps even a quantitative split by finding samples where hands are not detected. In a future work, I would also encourage the authors to look at using data beyond the human interactions typically present in HowTo100M (think documentaries), where interaction localization is much more tricky and non-obvious.

While the rebuttal is very promising, there are a lot of small and big issues to be improved in the submission before it can be accepted. The very fact that such a detailed rebuttal was necessary in the first place is a little concerning. I'm happy to increase my rating to 6, conditional on all the changes being incorporated, but cannot champion it in the current version. One additional round of review may not hurt much.
limitations_and_societal_impact: The authors claim that they will release code judiciously to prevent "further development of surveillance and tracking technology". However, it's not clear how this will be executed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper concerns a new problem of grounding narrated interactions in video. It is similar to existing work on (object) phrase localization/grounding, but with a twist on grounding more complex narrations that could involve both entities and predicates (e.g., put tomatoes into the baking tray). The model training is performed in a weakly supervised fashion on a subset of HowTo100M. Training loss is the standard InfoNCE loss at the sentence level (vs. word-level as in existing grounding work). Model architecture is designed to capture inter-modal correspondence through cross-modal attention. Evaluation is conducted on YouCook2-BB, Flickr30k-Entities, and a newly collected dataset for narration grounding called YouCook2-Interaction.
main_review: The main contribution of the work comes from the proposed weakly-supervised grounding framework. It demonstrates that i) with a carefully-designed decoupled multimodal interaction layer (the bottom line of Tab. 2, right), grounding performance can be greatly enhanced, ii) and sentence-level contrastive loss has an edge over the traditional word-level contrastive loss. Experimental results and ablation studies are overall comprehensive.

In terms of the weaknesses of the work, the paper can be significantly benefited from better clarity, with main technical contributions highlighted. In the current shape, the paper attempts to motivate with the new problem setting (and dataset), which is somewhat incremental considering existing work and complicates the storyline. More detailed comments are as follows.

i) The motivation is somewhat disconnected from the method itself. For example, at L54, the paper mentioned that "[...] there are multiple 'onion' objects visible and the correct one depends on the action being applied." However, the paper has not addressed this problem particularly but instead proposes a rather generic approach for grounding. Also, the YouCook2-BB benchmark adopted in the experiment is agnostic to if the action is applied (i.e., 'onion' will be annotated throughout the video as long as it's visible and will be cut at some point).

ii) The pointing hand evaluation metric is somewhat weak. Consider proposal-based metrics. Also, in L257, what if there are multiple boxes match the same narration?

iii) The proposed method is based on a well-trained model (appear to be MIL-NCE? More details should be included in the main paper) and further refined on a well-curated dataset for the downstream task. It's not surprising that it performs well (which is somewhat unfair to methods without access to these resources). More analyses need to be conducted on the impact of each factor. 

iv) Minor comments:

L227, should "for the j-th word" be "region" instead?

L230, "self-training" should be "pre-training". "Self-training" usually refers to a training scheme that leverages generated pseudo labels. 

L277, state-of-the-art.
limitations_and_societal_impact: Appear to be.
review_title: null

Meta-review: This paper has a very nice insight, and its essence has been made much clearer by the discussion and new notation during the rebuttal discussions. The two reviewers who originally recommended rejection have upgraded their scores to accept following these discussions.

The authors have promised to make many updates to the paper, and should take note of the post-rebuttal statements of the reviewers.
Decision: Accept (Spotlight)