Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper makes several contributions: a large scale video-language dataset obtained from 6M Youtube videos; a end-to-end VL model that learns multimodal representations, MERLOT; and large number of experiments including ablations, zero-shot transfer, and fine-tuning to achieve a new state of the art on a host of popular video language tasks, as well as challenging image-text tasks such as visual commonsense reasoning.

main_review: *Strengths:*
1. A large new dataset to rival HowTo100M, YT-Temporal-180M, that is not limited primarily to instructional videos. I found the performance improvements somewhat surprising, considering the prevalent understanding that a generic Youtube video (non-instruction, non-vlog) need not have narration describing visual aspects shown on screen.
2. Another pleasant surprise is that the model achieves the performance by simply using a mid-frame of a video segment, instead of needing to rely on complicated 3D CNNs.
3. Thorough pre-processing and three important pretraining objectives probably contribute to much of the success. Unfortunately, it's not very clearly explained how much is gained from each of the three pretraining tasks (Weaknesses 1a).
4. Thorough experiments on a variety of tasks on video/vision-language understanding, achieving new SOTA.

*Weaknesses:*
1. While this is a strong submission, it would be useful to include a few additional ablations / experiments:  
a) Impact of overall pre-processing steps vs. directly using all videos and the three pretraining tasks could be demonstrated perhaps on VCR and TVQA+ as done in Table 3. Seems to me that these are critical aspects of the contribution, and while I understand that they are costly experiments in terms of compute, are necessary to shed light onto what made the model actually work. Some combinations are already presented in Table 3a.  
b) If similar dataset pre-processing steps are applied to HowTo100M, would the performance reported in Table 3b improve further?  
c) The large jump in performance on VIST is probably owing to the "temporal reordering" task. Might be interesting to analyze performance without this objective (as indicated in ablation above).

2. Dataset:  
a) The dataset filtering (Appendix A.2) seemed like it primarily focuses on videos of people showing a variety of common objects (filtered based on presence of at least 4 objects in COCO). This may filter out most documentary style videos from Youtube - but it doesn't affect performance since most of the downstream tasks are about people as well.  
b) Some statistics such as number of seconds per segment, the distribution of videos by Youtube categories, etc. might be nice to include in the appendix.

3. VCR experiments. L249-251, while it's very interesting that the model can see the colored bboxes in the image and reason about them, it might be interesting to also try a baseline model (any of the "visual BERT" models) with such a strategy to properly benchmark improvement.

4. Minor:  
a) L106 says "large body of work" on action forecasting, but only cites one.  
b) L124, I think there might be space to mention the MobileNet-V2.  
c) Are segments of the video that do not have a speech transcript ignored?  
d) I'm not convinced by the word "knowledge" in the title. It gives the impression of a potentially symbolic component or at least an evaluation on such related knowledge graphs - Comet, Atomic, etc.


Note: Many experiments asked in this review will probably take time longer than that provided for the rebuttal. It's sufficient to include a brief plan instead.


### Post-rebuttal

Thanks to the authors for their response. I found that it explains several questions about the implementation details satisfactorily. I also briefly read all other reviews and responses and we all seem to agree that this is a good paper to accept.

I probably mis-framed the question about temporal ordering and VIST. My question was considering this scenario: if the pretraining did not involve the temporal ordering task, would that affect VIST as a downstream task significantly? Essentially, what is the importance of temporal ordering during pretraining as compared to only fine-tuning? I agree with the response that a 'zero-shot' setting would be complicated.

limitations_and_societal_impact: Paper discusses considerations of data collection and resulting social biases well in Section 6. Additionally, the Appendix includes a detailed "Datasheet" for the YT-Temporal-180M dataset.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The submission proposes a model and dataset for learning representations of multimodal events by training over YouTube videos with ASR captions. The dataset (YY-Temporal-180M) consists of 180 video segments pairs with words extracted from ASR.  The dataset is taken from 6 million YouTube videos and spans multiple domains and topics.  The model (MERLOT) is a transformer based model that is trained on image and language to 1) predict frame-caption match 2) recover masked words (masked language modeling) and 3) predict the temporal ordering of frames (given two frames f1, f2, does f1 come before or after f2).  For 1) a contrastive loss is used and the features for the caption for a segment is extracted not just from the segment but from the entire transcript (it is a contextualized representation for the segment). For 2), attention weights from a language-only transformer is used to favor masking of highly attended-to-tokens so to reduce masking of fillers such as 'umm' or 'yeah' that are common in spoken speech (SpanBERT masking is also applied for robustly against noise). 
Experiments on 14 downstream tasks (mostly video QA tasks) demonstrates the proposed model/dataset/training objective combination outperforms existing state-of-the-art results on these tasks.   There are also ablations comparing the number of epochs to train, the dataset used for training, and training setup (different amount of context, different loss terms).

Main contributions
- MERLOT: a Transformer-based 
- YY-Temporal-180M - dataset of frames and transcription from a filtered set of YouTube videos
- Experiments and ablations showing the combination of the proposed model and dataset can outperform prior models on 14 downstream tasks

main_review: Originality: Currently, there is many different models using transformer-based models for vision and language.  Despite some novel elements, the overall originality of this work is limited. 

Quality: The works looks to be well executed with reasonable rationales for the design choices, and a good set of experiments.

Clarity: Overall, the paper is well written and well organized.

Significance: The dataset (if it can be publicly released) will be useful to foster additional research in this area.  The proposed model and experiments are also useful.

Strengths:
- The model is well-designed and experiments are well-executed
- The dataset (YY-Temporal-180M) would be useful for the community

Weaknesses:
- Poor description of how the proposed model (MERLOT) relates to prior work on pretraining with video and language.  It's unclear specifically where the novelty of the model lies.
- Lack of comparison with prior models (ClipBERT, ActBERT) pretrainined on the same data.  It is unclear whether pretrained on the same dataset (YY-Temporal-180M), the proposed model performs better at the specified tasks than models from prior work. 
- The ablation studies is not that thorough.  For instance, while there is ablations on the contrastive VL loss and the attention masking, there is no ablation on the temporal ordering loss. 
- The dataset does not seem like it will be easy to download as only links to the associated YouTube videos will be provided.  This also means that the videos may disappear over time.

Other comments:

Some of the claims in the introduction about learning commonsense and script-knowledge representation is not really substantiated - the main evidence for this is that the model outperforms other models on two very specific tasks (the VCR and SortStory tasks).

References should be proofread and updated so that the publication venue is clear and consistent.
- [20],[21] is duplicated
- [80]: "Leon Sigal" => "Leonid Sigal"
limitations_and_societal_impact: Limitations and societal impact is discussed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a new approach to learn from large collections of uncurated video and text obtained from Automatic Speech Recognition Transcript. The authors introduce a dataset of 6M YouTube videos that lead to better results than existing large scale video and language datasets. The proposed architecture consists of separate backbones adapted to images (ResNet50) and text (Transformer) followed by a joint Vision and Language transformer. The model is trained with three different losses: 
* (i) a contrastive text-vision matching loss applied before the joint transformer, 
* (ii) a masked language modeling loss (with a modified masking mechanism to bias towards visual words), 
* (iii) a temporal ordering loss to learn to reorder vidoe frames.
main_review: ## Strengths

- Clearly written

- A new dataset that can be impactful for the community (important for the authors to confirm that the dataset will be released at least in the form of YouTube ids + narrations).

- State-of-the-art results on 12 different benchmarks

- Interesting innovations on the model side: biased masking strategy, contrastive loss coupled with Joint Transformer based approach.

## Weaknesses + clarification points

- More ablation studies: importance of the different losses is not properly ablated such as the temporal ordering loss does not seem to be ablated in the main paper. Similarly the novel masking technique is not compared to the default random masking strategy. Please include these ablations in the rebuttal.

- Confirm that the training dataset will be made available.

- I found the data collection description quite interesting in the Appendix. It might be worth surfacing this more in the main paper. In particular how important was the procedure to clean the transcripts? Have you ablated that? 

- It would be nice to also evaluate the learned visual representations since they are trained from scratch. For example it would be nice to compare to [6, 56] on benchmarks such as UCF101 or HMDB51.

- Related to the previous point, would it be possible to perform zero-shot text-to-video retrieval on benchmarks such as YouCook2 or MSRVTT to compare to previous work? In particular most benchmarks are evaluated via specific finetuning while it would be Interesting to be able to evaluate the model without any adaptation.

- The model only takes as input single frame per segment, how can it then model dynamic scenes? Can the authors comment on that limitation and how to address it in future work?

- Minor comments and questions:

  * Typos: atttention -> attention (Table 3, (a)).
  * Sentence boundaries: how are the text boundaries exacty obtained? In L138 BPE is mentionned with 32 tokens but it seems that a sentence separator is also used. Are the two things combined somehow?


## Overall assessment

Overall, MERLOT is a good paper that introduces a good large scale pretraining datasets, interesting architecture innovations to deal with the scale of the data and obtains strong results on several benchmarks. As of now, I lean towards acceptance but request the authors to address the points mentionned above.

## Post rebuttal

After reading other reviews and rebuttals, all my questions have been addressed. I believe MERLOT is a strong paper that will likely be influential in the vision and language community.

Of particular interest I believe that showing that leveraging YouTube data can improve **image** tasks such as VCR is quite interesting. Also after the additional ablation requests, I believe this will help make the contributions in terms of architecture and losses clearer in the paper. I hope the authors will include this nicely in the final version.
limitations_and_societal_impact: Limitations and societal impact were properly addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a video-text based self-supervised learning objective to learn strong representations of temporal commonsense. They curate a large scale dataset of videos with transcribed captions (from ASR) from diverse everyday situations to learn stronger and diverse knowledge representations. They perform extensive experiments  on 12 downstream tasks of video question-answering and on the visual commonsense reasoning tasks and outperform state-of-the-art models. They also perform extensive ablation studies to show the importance of pre-training on video datasets compared to image based pre-training. 


main_review: The main contributions of the paper are an end to end vision and language model to learn powerful multimodal representations from videos and transcripts using no labeled data (no bounding boxes and or fine grained captions as in previous work) and a diverse large scale dataset of YouTube videos. They use three objectives to train the joint vision-language model - a contrastive frame caption matching, masked language modeling and temporal ordering of frames. The paper proposes a novel idea and is well written and shows extensive experiments. 

1. The contrastive frame-caption matching loss considers the CLS token to compute the loss. How would the model perform by taking the average embeddings of the image matches instead of the CLS level tokens ? How important are the CLS level tokens for the image encoders ?

2. As both the contrastive and temporal re-ordering loss are over the image CLS token embeddings, is this supervision enough to learn good and meaningful image patch representations ([1,1] … [H,W]) ? What if we only have the image level embeddings from the image encoder?

3. Does the video-pre training work better just because the dataset is much larger than  the image ones?  Is it still possible to achieve good performance (reasoning and understanding) with large scale image datasets with strong cross modal attention and architectures ?



##Post Rebuttal

I have carefully read the authors' response and they address my concerns. I maintain my original rating.

limitations_and_societal_impact: The authors have discussed the limitations and impact of their work. One thing to consider is the cost while training on large scale video datasets versus on image datasets and how potentially that can be curbed by not hurting performance and understanding of the visual world.
review_title: null

Meta-review: This work proposes to jointly learn a visual-language model from a large amount of uncurated videos for (YT-Temporal-180M) with transcribed speech. The authors combine a set of pretraining tasks: image-text contrastive learning, MLM (with a special twist of using attention for selecting words), and temporal predictions of video frames. Reviewers unanimously this is good paper with strong SOTA results across many benchmarks and thoughtful analyses. The authors also thoughtfully respond to ethic reviewers' comments. On my side, I particularly appreciate the additional interesting ablation tests on data processing and ASR that the authors provided during rebuttal. The YT-Temporal-180M seems like a valuable resource for researchers and I hope that a careful release strategy of the dataset will work. I hope the authors can add more details to "Section 5.2 Qualitative analysis: MERLOT learns representations spanning distant frames" and make the figure more viewable. Last point, the title refers to "script knowledge" and to be convincing on that, it would be helpful to have a qualitative analysis on the model understanding of one scenario, e.g., restaurant in details. Given all the plus points of this paper, I recommend Accept.
Decision: Accept (Oral)