MERLOT: Multimodal Neural Script Knowledge Models
Authors: Rowan Zellers
Ximing Lu
Jack Hessel
Youngjae Yu
Jae Sung Park
Jize Cao
Ali Farhadi
Yejin Choi
Venue: NeurIPS 2021 Oral (2021)
Keywords: commonsense reasoning
representation learning
vision and language

Abstract: As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech -- in an entirely label-free, self-supervised manner. By pretraining with a mix of both frame-level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what is happening globally over time. As a result, MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets when finetuned. It also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERLOT~answers questions correctly with 80.6\% accuracy, outperforming state-of-the-art models of similar size by over 3\%, even those that make heavy use of auxiliary supervised data (like object bounding boxes).

Ablation analyses demonstrate the complementary importance of: 1) training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.

MERLOT:
Multimodal Neural Script Knowledge Models
Rowan Zellers
Ximing Lu
Jack Hessel
Youngjae Yu
Jae Sung Park
Jize Cao
Ali Farhadi
Yejin Choi
Paul G. Allen School of Computer Science & Engineering, University of Washington
Allen Institute for Artificial Intelligence
https://rovanzellers com/merlot
Abstract
As humans, we understand events in the visual world contextually, performing multimodal reasoning across time make inferences about the past, present, and future. We introduce MERIOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech in entirely label-freee, self-supervised manner. By pretraining with a mix of both frame- level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what happening globally over time. As a result, MERIOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of- the-art performance on different video QA datasets when finetuned. also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERIOT answers questions correctly with 80.6% accuracy, outperforming state- of-the-art models of similar size by over 3%, even those that make heavy use auxiliary supervised data (like object bounding boxes).
Ablation analyses demonstrate the complementary importance of: training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.
MERIOT
Video CA
Whats holding Which
Commonscnce Stigl: Inige
VCR
6M videos
man
Figure 1: Multimodal Event Representation learning Over Time. We learn representations of multimodal script knowledge from 6 million YouTube videos. These representations can then be applied to variety of downstream tasks that require commonsense or temporal visual reasoning.
Introduction
The human capacity for commonsense reasoning shaped by how we experience causes and effects over time. Consider the still image of people dining at a restaurant in the bottom right of Figure [ while literal, concrete description like "people sitting a table eating" might be technically correct for the static scene, it doesn't 'tt capture the richer temporal, commonsense inferences that are nonetheless obvious: before sitting down, the people had to meet up, agree where to go, and enter the
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

restaurant; ar present, the man is pointing because the server just came to the table, and she might want know whose food is whose; and after. is likely the server will return the kitchen to help another table.
Teaching machines this type of script knowledge [95 is significant challenge in small part because enumerating all facts, inferences, and counterfactuals is prohibitive. As a result, the highest performing models on vision-and- language tasks, including Visual Commonsense Reasoning (VCR) (where Figure Is scene originates from), learn about the visual world exclusively through static images paired with literal captions [008]
119 36.. Though some captions might hint at the past and future, it is not obvious that even training on, e.g., 400M literal image/text pairs [89] will result in models capable of temporal reasoning.
In this paper, we introduce MER[OT, short for Multimodal Event Representation learning Over Time. MERIOT is a model that learns commonsense representations of multimodal events by self- supervised pretraining over 6M unlabelled YouTuub videos. With the goal of learning multimodal reasoning capacity beyond static images/literal captions, we train MERLOT to a) match individual video frames with contextualized representations of the associated transcripts, and to b), contextualize those frame-level representations over time by "unmasking" distant word-level corruptions 27 and reordering scrambled video frames.
We validate our model diverse suite of video tasks, requiring both recognition- and cognition-level reasoning across long and short timescales; when finetuned, MERIOT achieves new statc-of-the 12 such tasks. Additionally, we show that our tript-knowwedge representations transfer to the single image domain. On Visual Commonsense Reasoning (VCR; [23)), our model achieves particularly strong performance, outperforming models that require heavy visual supervision (in the form of object detection bounding boxes, or images paired with pristine captions).
Beyond finetuning, show both quantitatively and qualitatively that MERLOT has a strong out- of-the-box understanding of everyday events and situations. Given a scrambled visual story, [50 2]. MERIOT can sort image sequences match captions which tell globally coherent narrative. Despite considerable domain shift from videos to static images, MERLOT outperforms strong baselines like CLIP [89] and UNITER 22), which independently match images to text and thus cannot reason over long-term contexts as effectively. This capacity for temporal coherence emerges during pretraining: analysis of MERLOT's attention patterns (Figure 11] show that regions attend to captions that are distant in time (and vice versa), allowing it perform cross-modal coreference to piece together holistic view of situations.
Finally, ablations of MERIOT show that 1) pretraining works better when we train on videos rather than still images, aided crucially by our strategy of corrupting highly visual words in the masked language modeling task, 2) using a diverse set of videos covering many aspects of everyday situations improves downstream performance compared to curated instructional video corpora [107 80 which both cover a smaller slice of the visual world (confirming hypotheses from past work [47]); and 3) MERLOT's performance does not saturate even after many epochs of training the pretraining corpus we curated, YT-TTeppoal-- 80M, as it continues to improve performance simply with more pretraining. The combination of these results suggests that learning full-stack visual reasoning and multimodal world knowledge from video data a promising path forward for future research.
In summary, our main contributions are:
1. MERIOT performant end-to-end vision and language model, that learns powerful multimodal world representations from videos and their transcripts using no labeled data. YT TT-Temporal- 180M, diverse corpus of frames/ASR derived from a filtered set of 6M diverse YouTube vidcos, which we show greatly aids performance, and set of experiments/ablationss demonstrating the strong performance of MERLOT on a set of 14 tasks, spanning finetuning and zero-shot transfer, and images and videos.
At wrwwnneelerr...... wwt.leerr....momeeeeeeeeeeeee merlot we have released code, data, and models for public research use.
2

---

2 Related Work
2.1 Joint representations of written text and images
There is a long history of work learning joint text-image representations 14
Recently, several pa-
pers have proposed "Visual BERT" models [008] 22 & 69 75 119, 36), trained on image captioning datasets such as MSCOCO 771. In general, features
extracted using Anderson et al. [10] 's frozen object detector, which was originally trained on Visual Genome [60]. Some exceptions are Zhang al. [25], who use even larger object detector trained on more labeled data; Kim et al. [57], who use an ImageNet-pretraiin pretrained on web image-caption pairs.
backbone (26), and Shen et al. [100), who study a CLIP backbone [99
Overall, these approaches all learn visual representations of static images, and rely on significant human annotation in doing (e.g. through literal image descriptions). Instead, our approach learns dynamic visual representations purely from videos their frames, and a transcript of what is said thus using no human annotation.
2.2 Learning from videos, with automatic speech recognition (ASR) transcripts
Prior works have used web videos with ASR to build weakly-supervised object detectors
87 action
detectors/classifiess [120 6 62
instruction aligners 19), video captioners [96
46 86
[0I,
and visual reference resolvers 49
Of late, works have sought to learn multimodal representations
transferable to many tasks from uncurated sets of (usually how-to) videos [80] [06
J07 81 127 p
4] generally these are applied to video understanding tasks like activity recognition. One challenge is designing an appropriate objective for learning video-level representations. Lei al. [67 's ClipBERT model learns dision-language representations from image captions, which more literally describe image content versus the longer ASR transcripts we consider. Tang et al. [109] use a pretrained dense image captioner [59] provide auxiliary labels for web how-to videos. Both approaches use (supervised) ResNets pretrained on ImageNet (43] as their visual backbones. MERLOT is trained
using a combination of objectives requiring manual supervision; it nonetheless outperforms both prior approaches on downstream tasks.
2.3 Temporal ordering and forecasting
There has been a large body work on analyzing "what happens next' in videos [88] Some modeling choices include using pixels [34] [13,, graphs [i1], euclidean distance using sensors K. studying
cycle consistency across time 32). addition to extrapolation, past work has studied deshuffling objectives in videos [2 115), though this has mostly been limited to the visual modality. In contrast to these papers, our goal is learning multimodal script knowledge representations: using both language and vision as complementary views into the world, instead of just tracking what changes on-sereen.
3 MERIOT: Multimodal Event Representation learning Over Time
We now present our unified model for learning script knowledge through web videos; including our pretraining dataset, architecture, and objectives.
3.1 YT-Temporal-180M
We collect YT-Temporal-180M. a dataset for learning multimodal script knowledge, derived from 6 million public YouTube videos. Our YT TTTTTemporal-880X intentionally spans many domains, datasets, and topics. We began with 27 million candidate video IDs (which we then filtered), including instructional videos from HowTo VLOG dataset (35. and YouTube's s auto-suggested videos for popular topics like 'science' or 'home improvement.' Our intent (in making the corpus as diverse as possible)
(80) lifestyle vlogs of everyday events from the
to encourage the model to learn about a broad range of objects, actions, and scenes [47]: we will later show through an ablation that limiting our pretraining only instructional videos indeed hurts performance downstream.
We filtered videos using the YouTube API, which provides access to videos themselves, their ASR track (automatically transcribed speech tokens), and other metadata. We discard videos I) without
3

---

Contrastive
Temporal ordering
Unmask words
We're making green- house.
Frame-caption matching
mask
T
CLS
18
saw
thin plastic, I'll be easy cut.
CLS
CLS
Joint Vision & Language Transformer Encoder
Aua Proodds
cut with circular
CLS
CLS
M.W
CLS 1.1
CLS
CLS
Image Encoder
Image Encoder
Word embed
embed
For morning
It's thin plastic, it'll easy to cut.
So I'll cut it with circular [NASK]
routine,
Figure 2: Left: MERIOT learns match contextualized captions with their corresponding video frames. Right: the same image encoding is provided, along with (masked) word embeddings, into joint vision-language Transformer model; then unmasks ground words (like 'saw' in this example) and puts scrambled video frames into the correct order.
an English ASR track; 2) that are over 20 minutes long: 3) that belong to visually "ungrounded" categories like video game commentaries; and 4) that have thumbnails unlikely to contain objects, according to a lightweight image classifier. We add punctuation to the ASR by applying sequence- to-sequence model trained add punctuation to sentences/paragraphs from news articles. Full details of the scraping and filtering are in Appendix A
Each video V might contain thousands of frames. In this work, we represent video V sequence of consecutive video segments {st). Each segment 8, consists of:
an image frame I, extracted from the middle timestep of the segment, b. the words w, spoken during the segment, with total length of tokens.
To split the videos into segments, we byte-pair-encode (BPE;
each video transcript and align tokens with YouTube's word-level timestamps. This enables us to split the videos into segments of BPE tokens each (Appendix (A.4); our final dataset has 180 million segments of this form.
T.=22
3.2 MERLOT Architecture
A diagram of MERIOT is given in Figure 2 MERIOT takes sequence of video frames {s, as input. We encode each frame I, using an image encoder, embed the words w; using a learned embedding, and jointly encode both using Transformer [112]. After pretraining, the architecture can be applied to a variety of vision-and-languaee for example, we pass several video frames the image encoder, the question to the text encoder, and extract a single vector representation from the CLS token position. For each task, we learn a lightweight classification head mapping from this hidden state to the task's 's label space; specific modeling/optimizatte details are given in Appendix E.2
tasks with minimal modification. For video QA,
Image encoder. We train our image encoder end-to-end, alongside the rest of the model, from random initialization (thus without learning from supervised data). While most performant vision-
and-language models pre-extract features from a (supervised) object detector [108 69 75 22 68).
for the sake of pre- pre-training efficiency we use grid-based hybrid ResNet/Vision Transformer
Specifically: our encoder uses ResNet-50 backbone, followed by 12-layer, 768-dimensional Vision Transformer 4 112 31. We made additional modifications that improve efficiency, including: 1) we trained on smaller, widescreen images of size 192x352 (because most YouTuue videos are
'Standard object detectors have expensive operations for proposing regions, and extracting features from those regions (Rol-pooli) (Rol-pooling): our grid approach avoids these. Recent work has proposed using 'grid features' broadly (53). yet on tasks like VCR these approaches have so far underperformed the more expensive object detector backbones [[3]] results suggest that 'grid features' can perform well broadly.

---

widescreen) using patch size of 16x 16 pixels; 2) we mirror [3]'s alterations of removing the C5 block in ResNet-50; and 3) we save compute further by average-pooling the final-layer region cells using a kernel size of 2 x 2. With these modifications, our image encoder requires 40 gigaFI.OP, for a forward pass, which 2% of the teraFL.OPs required for the Faster-RCNN.
In summary: given an image of size W x H, the image encoder will output a w/32X H/ feature map, along with two CLS hidden states: one for pooling a global representation of the image, and another for pretraining (Task L.
Joint Vision-Language Encoder. The joint encoder is 12-layer. 768- dimensionnl Transformer [12]. mirroring the RoBERTa base architecture [72]; we initialize it with pretrained RoBERTa weights. To compute joint representations, we first embed the tokens {w,} } via lookup. and then add position embeddings to both language and vision components (i.e., {I.}) The position embeddings differ between different segments, so to distinguish between images and captions at different timesteps. Finally, we pass the independent visual and textual feature maps to our joint encoder.
The tokens w; in each segment begin with CLS token; recall that the feature maps for each frame I start with one as well. At those positions, we will later pool final-layer hidden-state representations, for use in pretraining along with downstream tasks.
3.3 Pretraining Tasks and Objectives
We use the following three objectives to pretrain MERLOT, that cover 'full-stack' visual reasoning from recognition subtasks (like object detection) that operate at the frame level, to more 'cognitive' tasks that operate at the video level.
Contrastive frame-transerig matching 126 89 encoder produces helpful image representations. Thus, we use the video transcript to compute a Manguage-only representation of each video segment; and use contrastive loss to maximize its similarity to corresponding representations from the image encoder ?
We want to ensure that the underlying image
Unlike what is the case for many image captions, the words w; in each segment are often not sufficient to describe the gist of I, or even what the key objects might be - for that, video-level contextualization is often required. We thus pass the entire transcript into the language-only encoder, which then extracts hidden states for each segment at the segment-level CLS tokens.
Given matching representations for each frame I and caption w; as positive examples, the negative examples come from all other frame-caption pairs in the batch whether or not they come from the same video. We project both of these representations into size-768 hidden state which is then representations. We divide these logits by temperature of = 0.05, and then apply a pairwise cross entropy loss to encourage matching captions and frames.
and compute an all- all-pairs dot-product between all image and text
(Attention) Masked Language Modeling When providing words into the joint vision-and- language encoder, we randomly replace 20% with MASK token, random word, the same word; MERLOT must then reconstruct the correct word with a cross-entropy loss, following [27].
This approach is commonly used by 'visual BERT" models in the image captioning domain, where captions are concise, and thus the identity of masked concrete words difficult for models to recover given language context alone. However, we observed qualitatively that videos break these assumptions: people tend to ramble, and often mention key objects multiple times. Thus, applying vanilla BERT-style masking often causes ungrounded fillers like 'umm' or "yeah" to get masked, while the (repeated) names of important objects are often partially masked, penalizing the learning of multimodal representations.
We introduce simple solution to this problem. that we call attention masking: weights from a language-only transformer (introduced in the previous objective) as a heuristic for which words grounded. 50% of the time, mask out random token; the other 50% of the time, we mask out one of the top 20% moss-attendde--to-ooeee
attention
We then apply SpanBERT masking (54), randomly corrupting the following or preceding tokens with an average length of 0.5 tokens in each direction; this makes it harder for models to over-rely on BPE artifacts. We show in ablations that this improves performance.
"To save memory, our 'language-only encoder" for this subtask shares parameters with the joint vision-and- language encoder.
5

---

Q-A
QA
R
Q
Q-AR
Spearman
Pairwisc ace Distance
VILBERT
75
73.3
74.6
54.8
(1)
(1)
(4)
Unicoder-
VL
68
73.4
74.4
54.9
CLIP 89
.609
78.7
.638
VLBERT 69
73.8
74,4
55.2
UNITER
.545
75.2
.745
UNITER
[2
75.0
77.2
58.2
VIILA
36
76.4
79.1
60.6
MERIOT
.733
84.5
.498
ERNIE- ViL
119
77.0
80.3
62.1
Table 2: Results unscrambling SIND visual stories 50 2]. Captions are provided in the cor- rect order; models must arrange the images tem- porally. MERIOT performs best on all metrics by reasoning over the entire story, instead of in- dependently matching images with captions.
MERIOT (base-sized)
80.6
80.4
65.1
Table 1: Results on VCR 123 We compare against SOTA models of the same 'base' size ours (12-layer usioonaadd-lnguage Transform- ers). MERLOT performs best on all metrics.
Temporal Reordering. We have the model order the image frames in a video, forcing it to explicitly learn temporal reasoning and giving it an interface to measure such temporal reasoning. Here, 40% of the time, we randomly pick an integer between 2 and (the number of segments provided to the joint encoder). Then randomly scramble video frames chosen at random, by replacing the segment-level position embeddings (e.g. [Amage_t]] ) for that frame with a random and unique position embedding, e.g. [imagg_] Simage_unk_000 unk_o]). These random position embeddings are learned, and separate from the 'unshuffled' position embeddings. This allows the model to order each 'shuffled' frame conditioned frames provided in the correct order (if any).
To compute the reordering loss, we extract hidden states from each frame the CLS token position. For each pair of frames, we concatenate their hidden states he, and fı, and pass the result through two-layer MLP, predicting
1ú t > bj We optimize this using cross-entropy loss.
3.4 Pretraining MERLOT
We pretrain our model for 40 epochs over our video dataset. We preprocess the dataset into examples with sequences of N-16 video segments each, each containing up to L-32 BPE tokens The language-only encoder computes contrastive representations given this entire sequence, its total length thus 512 tokens. To save memory, provide the joint vision- vision-language language encoder 4 groups of = 4 segments each. At an image training resolution of 192 x 352, the joint model's sequence length is 396 tokens. To combine the losses, multiply [te contrastive loss by coefficient of 0.25, which we found scaled its gradient magnitudes to roughly the same magnitude as the Mask LM loss.
We train the model using v3-1 13-1024 TPU pod, at batch size of 1024 sequences (or 16k segments) in total. This pretraining process on this hardware takes 30 hours. We provide additional information about hyperparameters and experimental setup in Appendix E.1
Experiments: Transferring MERIOT to Downstream Tasks
In this section, we explore MERLOT on 14 different tasks, covering vision-languee reasoning on static images as well as videos; present analysis and ablations to dig deeper into our performance.
4.1 Image tasks
VCR. We consider VCR 123 task and dataset where models must answer commonsense visual questions about images. These questions, about e.g. "what might happen next' or "what are people's intentions," force MERIOT transfer video-level understanding the world of single images.
VCR provides additional 'referring expression' information to models in the form of bounding boxes around named entities. For example, if Person1 is referenced in the question, the location of Person1 also given in the image. We provide this information to models by drawing (in pixel space) a
Too train the model on as much data as possible, merged together the segments of short videos, and split up longer videos, such that all preprocessed examples dataset have exactly N N= 16 video segments.

---

Tasks
Split Vid. Length ActBERT 127
ClipBERT 82
SOTA
MERLOT
MSRVTT-QA
Tost
Short
37.4
41.5
118
43.1
MSR-VTT-MC
Test
Short
88.2
88.2
127
90.9
IGll-Action Transirantttinn
Test
Short
82.8
94.0
Test
Short
87.8
96.2
TGIF-Frame QA
Test Short
60.3
60.3 67
69.5
LSMDC-FiB QA
Test Short
48.6
48.6 127
52.9
LSMDC-MC
Test Short
73.5 121
81.7
ActivityNeRQA
Test
Long
38.9 [18
41.4
Drama-Q2 QA
Val
Long
81.0 56
81.4
TVQA
Test
Long
76.2 56
78.7
TVQA+
Test
Long
76.2 56
80.9
VLEP
Test
Long
67.5 66
68.4
Table 3: Comparison with state-of-the-ar methods on video reasoning tasks. MERIOT outperforms state of the art methods in 12 downstream tasks that involve short and long videos.
colored highlight around the referenced entity (Appendix E.3.1 this differs from prior works (that integrate these entities into detection architectures).
Our results on the three VCR settings, in comparison to other models the same ('base') scale, given in Table | Our model outperforms these other models, that all learn from exclusively static images (paired with captions and supervised object detections).
Unsupervised ordering of Visual Stories. To probe our model's ability to do out-of-the-box com- monsense reasoning over events in images, we next consider the Visual Storytelling dataset [50] (74.. Each story in this dataset contains five images and captions in a certain order; the order tells a joint narrative between the captions and the images. Past work has considered unshuffling image- caption pairs (2], but we take a slightly different approach
this work to avoid language-only biases, which rely discursive clues to order text [27] [002.. In formulation, models given the captions in sorted order, and must match frames to the captions. Our formulation disarms language-only basclines, while still allowing to quantify MERIOT's capacity for commonsense temporal reasoning.
We compare MERIOT with two strong out-of-the-box baselines for text-image matching: CLIP 89 which encodes each caption and image separately and computes similarity through dot product, and UNITER [22] which jointly represents each image/caption pair, and is trained in part using 'text- image matching' objective. We use our temporal reordering loss to find the most probable ordering of the video frames (Appendix E.I.I: for CLIP and UNITER we compute a maxinum-weight bipartite matching (63 over the pairwise image-text similarity scores.
Results over 5K stories are given in Table MERIOT's performance in comparison to the algorithms trained from image-litera caption pairs suggests that, with no fine-tuning, our model has strong capability to reason about past and future events expressed in collections of temporal visual stories.
4.2 Video Reasoning
We report results on 12 video reasoning tasks: TVQA 64], TVQA(+) 65 VLEP (6). MSRVTT-QA [17], MSRVTT-Mullichoit 121. LSMDC-Mullichoice, LSMDCC
blank QA IIO. 92],
ActivityNeIQA 122 45). TGIFQA (52). and DramaQA (33) We apply MERIOT to these tasks in the same way. We sample a sequence parameters only to map the model's pooled CLS hidden state into the output labels, and finetune MERLOT with a softmax cross entropy loss; see Appendix E.2 for details.
still frames from each video clip, initialize new
As shown in Table 3 for all these datasets MERLOT sets new state-of-the-art. Given the diversity of tasks and the strengths of the comparison models, these results provide strong evidence that MERIOT learned strong multimodal and temporal representations.
4.3 Ablations
We present ablations over VCR and TVQA+ to study the effect of several modeling decisions.
7

---

Training sctup
|VCR
TVQA+
Training sctup
VCR
TVQA+
|CC
One segment (N=1)
73.8
75.2
No contrastive V.1. loss
57.5
67.6
No buxes
74.8
One segment, attention masking
Four segments
73.5
74.1
73.3
74,5
No temporal ordoring loss
V. All losses
75.5
75.2 75.8
75.6
Drawn-on boxes
79,4
Four segments, atlention musking
17..2
75.8
(b) Contrastive V+L loss is cru- cial. Removing it makes perfor- mance drop significantly; the tem- poral ordering loss is not as impor- tant for downstream finetuning.
Drawing on bounding boxes helps, suggesting that our model uses to decode the "referring expres- sion' information (e.g.
(a) Context helps together with attention masking. Pretraining on more segments at once improves performance, but more context can encourage language-only representation learning. Attention masking counteracts this, giving an additional point boost.
person1).
Dataset
VCR
epochs
VCR
Conceptual U COCO
58.9
5 spochs
75.2
How To100M V YT-Temporal-1885
66.33
10 epochs
75.9
75.2
20 epochs
77.0
How 1ooTTal000-szeed Te--oopr 180M YTT180M, raw ASR
72.8
30 epochs
78.5
72.8
40 epochs
79.4
(d) Diverse (video) data is important. Applying our architecture to cap- tion data leads to poor results. Our model performs better on HowTo100M, yet still below our (more diverse) YT-Temporal- 80M, even when con- trolled for size. Using raw ASR (vs. denoised ASR) reduces performance.
(e) Training for longer helps, with performance increasing mono- tonically over training iterations.
Table 4: Ablation study on the validation set of VCR question answering (Q + A) and TVQA+, in accuraty (%). We put next to the configurations we chose for MERLOT.
Context size. Table A shows the effect of varying the number of segments N given to the joint dision-and-languagg encoder during pretraining. In the first two rows, we provide only a single video segment (N=1) to the model In this limited regime, we find that our 'attention masking' approach (preferential masking of tokens that were highly attended-to by the contrastive language-only encoder) does outperform strong baseline of masking spans randomly (54). Yet, when expand the sequence length to N=4 segments/125 //28 tokens, our masking becomes more effective, improving by 1 point over the baseline. This supports our hypothesis (Section 2.) that text-only shortcuts become increasingly viable with length, and that our attention -eceninnnmmaking approach counteracts them
Losses. In Table 4p, we ablate the losses. We find that the contrastive frame-transcript matching loss is crucial to performance, suggesting that an explicit objective is critical for the (randomly initialized) image backbone to learn visual representations. The temporal ordering loss appears less critical for downstream tasks; it helps for TVQA but performance drops slightly for VCR. Thus, we find that it helps primarily an interface by which we can query the model about temporal events (i.e. for the story ordering experiments); the model might be learning this information from other objectives.
Drawing bounding boxes. Table shows the effects of providing grounding information to VCR models by drawing boxes. Performance drops 5% when they are removed, suggesting that they help.
Dataset source. In Table 41, we investigate pretraining MERLOT on two datasets beyond YT- TeM. First, we train on million static image-caption pairs from Conceptual Captions [99 combined with MSCOCO [7]]; for fair comparison, we train for the same number of steps as epochs on our dataset. The resulting model achieves 58.9% accuracy on VCR. We suspect this might be due to 1) a smaller context window (Table[Th). and 2) overfitting (5 epochs on Ye--Tepprral 180M corresponds to 300 epochs on the caption data). Because our vision pipeline is trained from scratch, the scale of the curated/supervised image pairing corpora is a concern.
We next investigate the impact of video selection, comparing YTT-Tepporal 180M with 1owTo100M [80]. To control for number of videos, we train for an equivalent amount of steps: 5 epochs on our dataset, 30 epochs on HowTol00M, and likewise 30 epochs on a HowTol00M-siize YT- Temporal-180m 180M. Using diverse YT-Temporal-1800 data vs. only instructional videos improves VCR performance by 6.5 points. This suggests that the how-to domain is limited in terms of visual
"We keep the effective batch size the same, that use 4x the number of sequences at 1 th the length. 'Additional qualitative analyses of the attention patterns produced by the language-only encoder are in Appendix C. we find that highly attended-to tokens are typically more 'visual', and, thus, masking them may make the Masked LM objective require more cross-mode reasoning.
8

---

Serie po ko wore ateie shat on
slready
the soralatoon
He was wtrond to the top.
the top
They than get on the bus.
151
leet my M8ds last weekend.
We carft
people them
They a barn.
cnimals
laber.
Figure 3: Zero-shot story ordering (same setup as Table 2 MERLOT performs temporal common- sense reasoning accross frames. In the first row, it uses 'the old man' mentioned to identify the 'kids' as parent-aged; in the second, identifies riding a merry-go-round as an activity that takes a while.
phenomena covered, and that other domains (like web dramas and VLOGs) provide helpful signal for tasks like VCR 47 Using all the data gives an additional 2.4-point performance boost.
Last, we investigate our choice to preprocess the YouTube ASR with a language model (adding punctuation, etc); using 'raw ASR' instead of this preprocessing reduces performance by 2.4 points.
Pretraining longer. Last, in Table we investigate the effect of pretraining MERIOT for longer. The performance increases monotonically and doesn't begin to plateau, which suggests that had we pretrained MERIOT for even longer, its performance could improve even further.
4.4 Qualitative examples
we show two qualitative examples of MERIOT's zero-shot story ordering capability. More examples (and comparison with the best-scoring baseline, CLIP [89]) are in Appendix C.2 The examples here show that MERIOT has strong understanding events, transcending individual frames. In the first row, it orders the story correctly, performing across several frames (e.g. frames and captions and 3 use 'he' to refer to 'the old man' only mentioned in the first caption). Without resolving this coreference (establishing the subject as an elderly family member), it seems unlikely that anyone would describe the adults in frame (3) as 'kids.' Investigating the attention patterns of MERLOT (Appendix C.3) backs up this claim; they show that MERLOT frequently addresses video tasks by merging attention across (distant) video segments.
In Figure 3
lan-uuuge coreference
MERIOT gets the second row 'wrong', but for an interesting reason. It reverses the order of frames (3) and (4), which groups the merry-go-round pictures together - even though caption (3) mentions barn. This seems to capture the temporal commonsense intuition that people might ride merry-go- go-nu for a while, i.e., it not an atomic event 225
5 Conclusion, Limitations, and Broader Impacts
We introduced Multimodal Event Representation lgarning Over Time (MERLOT), We trained the model through combination of self-supervised objectives 6M YouTube videos, in service of learning powerful multimodal representations that go beyond single frames. The model achieves strong performance on tasks requiring event-level reasoning over videos and static images. We hope that MERLOT can inspire future work for learning vision+language representations a more human-like like fashion compared to learning from literal captions and their corresponding images.
There are several potential limitations MERLOT that would make for promising avenues of future work, including: 1) exploring finer-grained temporal reasoning pretraining objectives vs. frame ordering e.g., a temporal frame localization within transcripts; and 2) learning multilingually from non-English videos and communities on YouTube.

---

Like other pretraining work, MERIOT risks some potential negative impacts. We discuss these in more detail below, in addition to the steps we took reduce these harms.
5.1 Data collection and privacy.
As with other corpora gathered from the web used for pretraining data, YT-Temporal--185 contains publicly available content posted by users. We thus shaped our data gathering and release strategy to minimize inherent privacy and consent harms (Appendix [A.5). Perhaps most importantly, we plan to
only share video IDs for download, following release strategy from prior work [I 80 and giving
users the right to opt out of not just YouTube, but our dataset as well.
5.2 Social biases.
The curation choices we made in this work could cause the model to exhibit undesirable social biases for this reason, along with others, we do not advocate for deployed use-cases. For example, 30% of the data selected for by our filtering pipeline was local broadcast news (uploaded to YouTube). Including these news videos seems to perform better than filtering them and only using how-to videos (Table 4b). however, there are risks when training on them. Local broadcast news (at least in the US) dedicates significant time covering crime, sometimes in racist and sensationalized
manner 38 29 44 Indeed, running a topic model over our data identifies several 'crime" categories (Appendix B Past work has shown correlation between watching local news and having more explicit racialized beliefs about crime [28]: it seems likely therefore that training models on this data could teach them learn the same racist patterns.
Additionally, there are inherent social biases on YouTube and treating these videos as equivalent to 'the world' [11I] can embed hegemonic perspectives [42
114 13 Most popular YouTubers
are men [30] and video practices emerging on YouTube are often gendered [83]. YouTube also has problems with hate, including radical alt-right and 'alt-lite' content (90]. These problems with other problems in representation and power are themselves amplified by the 'You'Tuee algorithm'
15] that recommends content to users. Though we downloaded videos independently of YouTube's recommender system, by filtering based on what content has views, we are implicitly filtering based on this algorithm. The dynamics of YouTube (i.e., which videos get popular/monetizee) influence the style and content of videos that get made and uploaded to the platform; this in turn shapes and is shaped by culture more broadly [044.
5.3 Dual use.
The video QA tasks that we studied carry risk of dual use, through possible downstream applications like surveillance 191, 128] It seems unlikely that purely technological fixes and defenses which themselves can be problematic [40] could resolve these dynamics. Studying how well video-level pretraining enables surveillance applications might be an important avenue for future work, if only to inform stakeholders and policymakers about these risks.
5.4
Energy consumption.
The pretraining that we used in this work was expensive upfront [05]. Our results suggest that scaling up the amount of data and compute that we used might yield additional performance gains but at increased environmental cost. To pretrain more efficiently, we used a much more lightweight architecture (in terms of FLOPs) than is standard for today's vision and language models. We hope that our public release of the model (for research use) can further amortize this cost.
5.5 Synthesizing these risks.
With these issues in mind, we release MERIOT and -T-Temporal--188 180M for researchers. We view our work, and our research artifacts, to be part of larger conversation on the limits of pretrained 'foundation models' [17]. These models have broad impact to real-world areas like healthcare, law, and education. At the same time, these models have significant risks, including the harms that we outlined. We believe that further academic research into this video-and-language pretraining paradigm is important- - especially to probe its limits and possible harms. We hope that our paper, code, and data release can contribute to this direction.
10

---

Acknowledgements and Funding Transparency Statement
We thank the anonymous reviewers for their helpful feedback that improved this work, along with Oren Etzioni and Gabriel Ilharco. Thanks also to Zak Stone and the Google Cloud TPU team for providing access to the TPU machines used for conducting experiments, and for help with the computing infrastructure. Last, but not least, thanks to all the YouTubers who share interesting videos with the world. This work was funded by DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AL.
References
[1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici. Balakr- ishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint 160i 08675, 2016.
[2] Harsh Agrawal. Arjun Chandrasekaran, Dhruv Batra, Devi Parikh, and Mohit Bansal. Sort Story: Sorting Jumbled Images and Captions into Stories. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 925-931, 2016.
[3] Pulkit Agrawal, Ashvin Nair, Picter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: experiential learning of intuitive physics. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 5092-5100, 2016.
[4] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: transformers for multimodal self self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2 104.11178, 2021.
[5] Jean- Baan-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 45833, 2016.
[6] Jean-Bane Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Simon Lacoste-Julien. Joint discovery of object states and manipulation actions. In ICCV, 2017.
[7] -ean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Rama- puram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self- supervised multimodal versatile networks. arXiv preprint arXiv:2006. 102006.6625 /6228, 2020.
[8] Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. arXiv preprint (eXXv:/908.050000 2019.
[9] Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. Noise estimation using density estimation for self-supervised multimodal learning, arXiv preprint 2:X11:2003.03186 2020.
[10] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.
[11] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Proceedings of the 30th International Conference on Neurul Information Processing Systems, pages 4509 2016.
[12] Emily Bender and Batya Friedman. Data statements for nlp: Toward mitigating system bias and enabling better science. Transactions of the Association for Compulational Linguistics, 2019.
[13] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2 ACM Conference Fairness, Accountability, and Transparency, pages 610-623, 2021.

---

[14] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on patter analysis and machine intelligence. 35(8): 1798-1828, 2013.
[15] Sophie Bishop. Anxiety, panic and self-optimization: Inequalities and the youtube algorithm. Convergence, 24(1):69-88) 84, 2018.
[16] David Blei, Andrew Y Ng. and Michael Jordan, Latent dirichlet allocation. Journal of machine Learning research, 5(Jn)))93311022, 2003.
[17] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks foundation models. arXiv e-prints, pages arXiv-2108, 2021.
[18] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- inc Lec, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:2012.078033 2020.
[19] Chien-Ye Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. In CVPR, 2019.
[20] Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. Story Comprehension for Predicting What Happens Next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1603-1614, 2017.
[21] Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan. Rspnet: Relative speed perception for unsupervised video representation learning. In AAAI, 2021.
[22] Yen-Chur Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Learning universal image-text representations, arXiv preprint arXiv:1909.11177 2019.
[23] Seongho Choi, Kyoung-Wove Woon On, Yu-Jung Heo, Ahjeong Seo, Youwon Jang, Seungchan Lee, Minsu Lee, and Byoung-Tak Zhang. DramaQA: character-centered video story understanding with hierarchical qa. arXiv preprint arXiv:2005.03356 2020.
[24] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276 286, 2019.
[25] William Croft. Verbs: Aspect and causal structure. OUP Oxford, 2012.
[26] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009, CVPR 2009. IEEE Conference on, pages 248-255. Icee, 2009.
[27] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[28] Travis L Dixon. Crime news and racialized beliefs: Understanding the relationship between local news viewing and perceptions of african americans and crime. Journal of Communication, 58(1): 106-106....22 2008
[29] Travis I. Dixon and Daniel Linz. Overrepresentation and underrepressnntation of african americans and latinos as lawbreakers on television news. Journal of communication, 50(2): 131-154, 2000.
[30] Nicola Döring and M Rohangis Mohseni. Male dominance and sexism on youtube: results of three content analyses. Feminist Media Studies, 12255212-522, 2019.
12

---

[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[32] Dave Epstein, Jiajun Wu, Cordelia Schmid, and Chen Sun. Learning temporal dynamics from cycles in narrated video. arXiv preprint arXiv:2101.022 Stt:::2200.02337,, 2021.
[33] Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, Lawrencce Zitnick, et al. Visual storytelling, arXiv preprint arXiv:1604 104.0366 2016.
[34] Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Proceedings of the 30th International Conference Neural Information Processing Systems, pages 64-72, 2016.
[35] David F Fouhey, Wei-cheng Kuo, Alexei A Efros, and Jitendra Malik. From lifestyle vlogs to everyday interactions. In CVPR, 2018.
[36] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large- scale adversarial training for uisionaadd-languge representation learning. arXiv preprint arXiv:2006.061 06195, 2020.
[37] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. Datasheets for datasets. arXiv preprint WrXiv:/803.09011 2018.
[38] Franklin D Gilliam Jr, Shanto Iyengar, Adam Simon, and Oliver Wright. Crime in black and white: The violent, scary world of local news. Harvard International Journal of press/politics, 1(3):6-23, 1996.
[39] Jonathan Gordon and Benjamin Van Durme. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 25-30. ACM, 2013.
[40] Ben Green. Good" isn't good enough. Proceedings of the Al for Social Good workshop at NeurlPS, 2019.
[41] Herbert Grice. Logic and conversation. In Speech acts, pages 41-58. Brill, 1975.
[42] Donna Haraway. Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist studies, 14(3):575-533 5799 1988.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[44] Don Heider. White news: Why local news programs don't ton't cover people of color. Routledge. 2014.
[45] Fabian Caba Heilbron, Victor Escorcia, Bermard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
[46] Jack Hessel, Bo Pang, Zhenhai Zhu, and Radu Soricut. A case study on combining ASR and visual features for generating instructional video captions. In CoNLL, November 2019.
[47] Jack Hessel, Zhenhai Zhu, Bo Pang, and Radu Sorieut. Beyond instructional videos: Probing for more diverse visual-textual grounding on youtube. In EMNLP, 2020.
[48] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09755 2019.
13

---

[49] De-An Huang, Joseph J. Lim, Li Fei-Fei, and Juan Carlos Niebles. Unsupervised visual- linguistic reference resolution in instructional videos. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
[50] Ting-Hat Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1233-1239, San Diego, California, June 2016. Association for Computational Linguistics. doi: (118665/11/IGG 147. URL https://www. acwweb.wwb........ org/ http//anooooogg/1---11155
[51] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3543-3556, 2019.
[52] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). Honolulu, Hawaii, pages 2680-8, 2017.
[53] Huaizu Jiang, Ishan Misra, Mareus Rohrbach, Erik Learned- and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10267-10276, 2020.
[54] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
[55] Ruogu Kang, Laura Dabbish, Nathaniel Fruchter, and Sara Kiesler. "my data just goes everywhere:" user mental models of the internet and implications for privacy and security. In Eleventh Symposium On Usable Privacy and Security ((SOUPS) 2015), pages 39-52, 2015.
[56] Seonhoon Kim, Seohyeong Jeong, Eun-Byul Kim, Inho Kang, and Nojun Kwak. Self- supervised pre-training and contrastive representation learning for multiple-choiee video qa. ArXiv, abs/2009.08000 08044,, 2020.
[57] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. arXiv preprint arXiv:2/02.03334 2021.
[58] Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecast- ing. European Conference on Computer Vision, pages 201-214. Springer, 2012.
[59] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- Captioning Events in Videos. In International Conference on Computer Vision (ICCV), 2017.
[60] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 1 123(1):32-33, 2017.
[61] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. In Proceedings of the International Conference on Computer Vision (ICCV), 2011.
[62] Hilde Kuehne, Ahsan Iqbal, Alexander Richard, and Juergen Gall. Mining youtube-a dataset for learning fine-grained action concepts from webly supervised video data. arXiv preprint Ntte:1996600012.. 2019.
[63] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-9, 1955.
[64] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video question answering, In EMNLP, 2018.
14

---

[65] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. Tvqa+: Spatio-temporal grounding for video question answering. In Tech Report, arXiv, 2019.
[66] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. arXiv preprint wrXiv:2110.07999 2020.
[67] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. arXiv preprint arXiv:2102.06183, 2021.
[68] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vi: A universal encoder for vision and language by cross-modal pre-training. AAAI, pages 11336- 11344, 2020.
[69] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 19X8:0900000357, 2019.
[70] Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In Proceedings of the European Conference on Computer Vision (ECCV), pages 513-528, 2018.
[71] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[72] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 2019.
[73] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
[74] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In European Conference on Computer Vision, 2016.
[75] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic vi- siolinguistic representations for vision-and- and-language tasks. In Advances in Neural Information Processing Systems, pages 13-23, 2019.
[76] Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C Courville, and Christopher Joseph Pal. A dataset and exploration of models for understanding video data through fill- in-the-blank question-ansswring. In Computer Vision and Pottern Recognition (CVPR), 2017. URL.atttp://ppeaccee thecvf cww/ coon/content_cppp httpc/ppr...////pppr///aa/aa//a/ Bestaase__and_cvPP__000 -ww.pppppppppp.. .df
[77] Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy, What's cookin'? ?? interpreting cooking videos using text, speech and vision. In NAACL, 2015.
[78] Alice E Marwick and danah boyd. Networked privacy How teenagers negotiate context in social media. New media & society, 16(7):1051-106,,22 2014.
[79] Andrew Kachites McCallum. Mallet: A machine learning for language toolkit. 2002. URL
http://mallet.ce cs, umass.
[80] Antoine Miech, Dimitri Zhukov, Jean- Ban--Battstee Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning Text- Video Embedding by Watching Hundred Million Narrated Video Clips. In /CCV, 2019.
[81] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan I Laptev, Josef Sivie, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
15

---

[82] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and lear: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pages 527-544. Springer, 2016.
[83] Heather Molyneaux. Susan O' O'Donnell, Kerri Gibson, Janice Singer, et al. Exploring the gender divide on youtube: An analysis of the creation and reception of vlogs. American Communication Journal, 10(2):1-14, 2008.
[84] Yasufumi Moriya, Ramon Sanabria, Florian Metze, and Gareth JF Jones. Grounding object 06147, 2019.
detections with transcriptions. arXiv preprint
[85] Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pages 69-84, 84, 2007.
[86] Shruti Palaskar, Jindrich Libovický, Spandana Gella, and Florian Metze. Multimodal abstrac- tive summarization for how2 videos. arXiv preprint arXiv: 077901, 2019.
[87] Alessandro Prest, Christian Leistner, Javier Civera, Cordelia Schmid, and Vittorio Ferrari. Learning object class detectors from weakly annotated video. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3282 22899. IEEE, 2012.
[88] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
[89] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint (:XXv:2203.0000) 2021.
[90] Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgílio AF Almeida, and Wagner Meira Jr. Auditing radicalization pathways on youtube. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 131-141, 2020.
[91] Neil M Richards. The dangers of surveillance. Harv. L. Rev. 126:1934, 2012.
[92] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Jour- nal of Computer Vision, 2017. URL http://link, springer com wttp:///rriiieeee/eeaee 1007/ s11263- 016 0987- 17wt_ __- Internal Event 1. SEM. wwt.:aArrrriilletttooooonnieeee.
[93] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.
[94] Oleksandr Savsunenko. How tensorflow's tf.imagg.resize stole 60 days of my life. Technical report, Hacker Noon.
[95] Roger C. Schank and Robert P. Abelson. Scripts, plans, and knowledge. In Proceedings of the 4th International Joint Conference on Artificial Intelligence Volume 1, IJCAI'75, pages 151-157, San Francisco, CA, USA, 1975. Morgan Kaufmann Publishers Inc. URL arw..rg//a...............oo....e
http: acm. org/ citatiinn..../ wfw.id==mmmmm2m2m2 1624649
[96] Ozan Sener, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Unsupervised semantic parsing of video collections. In ICCV, 2015.
[97] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, 2016.
[98] Sofia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951, 2019.
[99] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2018
16

---

[100] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-lange and-language tasks? arXiv preprint arXiv:2
2021.
[101] Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming Zhou, Dense procedure captioning in narrated instructional videos. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6382-6391, 2019.
[102] Wei Shi and Vera Demberg. Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLE pages 5794-5800, 2019.
[103] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(1 2012.
[104] Michael Strangelove. Watching YouTube. University of Toronto press, 2020.
[105] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 3645-3650, 2019.
[106] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:
2019.
[107] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT:
A joint model for video and language representation learning. In ICCV, 2019.
[108) Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP, 2019.
[109] Zineng Tang, Jie Lei, and Mohit Bansal. Decembert: Learning from noisy instructional videos via dense captions and entropy minimization. In Proceedings of the 202 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 2415-2426, 2021.
[110] Atousa Torabi, Niket Tandon, and Leon Sigal. Learning language- visual embedding for movie understanding with natural-language. arXiv preprint, 2016. URL httt:://axii.orre org/pdf/ 1609. 0ww.4v1.2.v.ooo pdf
[111] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1521-1528. IEEE, 2011.
[112] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008 2017.
[113] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision, pages 835-85 851. Springer. 2016.
[114] Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. Disembodied machine learning: On the illusion of objectivity in nlp. arXiv preprint arX9v.2101.11944 2021.
[115] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 8052-8060, 2018
[116] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,) pages 1 11-20, 2019.
17

---

[117] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[118] Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learn- ing answer questions from millions of narrated videos. arXiv preprint arXiv:2012.00433 2020.
[119] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprini arXiv:2006. 16934, 2020.
[120] Shoou-I Yu, Lu Jiang, and Alexander Hauptmann. Instructional videos for unsupervised harvesting and learning of action examples. ACM MM, 2014.
[121] Youngjae Yu, Jongseok Kim, and Gunhee Kim. joint sequence fusion model for video question answering and retrieval. In Proceedings of the European Conference Computer Vision (ECCV), pages 471 471-487, 2018.
[122] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet-QAA: dataset for understanding complex web videos via question answering. In AAAI, pages 9127-9134, 2019.
[123] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6720-6731, 2019.
[124] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances Neural Information Processing Systems 32, 2019.
[125] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. arXiv preprint arXiv:2101. 201.09299...... 2021.
[126] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arX1v:2010.00747, 2020.
[127] Linchao Zhu and Yi Yang. ActBERT: Learning global-local video-text representations. In CVPR, 2020.
[128] Shoshana Zuboff. Big other: surveillance capitalism and the prospects of an information civilization. Journal of Information Technology, 30(1):75-89, 2015.
18


Reviews:
[{"review_id": "UepOD8JY0ZL", "reviewer": "Reviewer_CWuA", "summary": "This paper makes several contributions: a large scale video-language dataset obtained from 6M Youtube videos; a end-to-end VL model that learns multimodal representations, MERLOT; and large number of experiments including ablations, zero-shot transfer, and fine-tuning to achieve a new state of the art on a host of popular video language tasks, as well as challenging image-text tasks such as visual commonsense reasoning.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths:*\n1. A large new dataset to rival HowTo100M, YT-Temporal-180M, that is not limited primarily to instructional videos. I found the performance improvements somewhat surprising, considering the prevalent understanding that a generic Youtube video (non-instruction, non-vlog) need not have narration describing visual aspects shown on screen.\n2. Another pleasant surprise is that the model achieves the performance by simply using a mid-frame of a video segment, instead of needing to rely on complicated 3D CNNs.\n3. Thorough pre-processing and three important pretraining objectives probably contribute to much of the success. Unfortunately, it's not very clearly explained how much is gained from each of the three pretraining tasks (Weaknesses 1a).\n4. Thorough experiments on a variety of tasks on video/vision-language understanding, achieving new SOTA.\n\n*Weaknesses:*\n1. While this is a strong submission, it would be useful to include a few additional ablations / experiments:  \na) Impact of overall pre-processing steps vs. directly using all videos and the three pretraining tasks could be demonstrated perhaps on VCR and TVQA+ as done in Table 3. Seems to me that these are critical aspects of the contribution, and while I understand that they are costly experiments in terms of compute, are necessary to shed light onto what made the model actually work. Some combinations are already presented in Table 3a.  \nb) If similar dataset pre-processing steps are applied to HowTo100M, would the performance reported in Table 3b improve further?  \nc) The large jump in performance on VIST is probably owing to the \"temporal reordering\" task. Might be interesting to analyze performance without this objective (as indicated in ablation above).\n\n2. Dataset:  \na) The dataset filtering (Appendix A.2) seemed like it primarily focuses on videos of people showing a variety of common objects (filtered based on presence of at least 4 objects in COCO). This may filter out most documentary style videos from Youtube - but it doesn't affect performance since most of the downstream tasks are about people as well.  \nb) Some statistics such as number of seconds per segment, the distribution of videos by Youtube categories, etc. might be nice to include in the appendix.\n\n3. VCR experiments. L249-251, while it's very interesting that the model can see the colored bboxes in the image and reason about them, it might be interesting to also try a baseline model (any of the \"visual BERT\" models) with such a strategy to properly benchmark improvement.\n\n4. Minor:  \na) L106 says \"large body of work\" on action forecasting, but only cites one.  \nb) L124, I think there might be space to mention the MobileNet-V2.  \nc) Are segments of the video that do not have a speech transcript ignored?  \nd) I'm not convinced by the word \"knowledge\" in the title. It gives the impression of a potentially symbolic component or at least an evaluation on such related knowledge graphs - Comet, Atomic, etc.\n\n\nNote: Many experiments asked in this review will probably take time longer than that provided for the rebuttal. It's sufficient to include a brief plan instead.\n\n\n### Post-rebuttal\n\nThanks to the authors for their response. I found that it explains several questions about the implementation details satisfactorily. I also briefly read all other reviews and responses and we all seem to agree that this is a good paper to accept.\n\nI probably mis-framed the question about temporal ordering and VIST. My question was considering this scenario: if the pretraining did not involve the temporal ordering task, would that affect VIST as a downstream task significantly? Essentially, what is the importance of temporal ordering during pretraining as compared to only fine-tuning? I agree with the response that a 'zero-shot' setting would be complicated.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Paper discusses considerations of data collection and resulting social biases well in Section 6. Additionally, the Appendix includes a detailed \"Datasheet\" for the YT-Temporal-180M dataset.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "OQAfAlROLP", "reviewer": "Reviewer_aTvC", "summary": "This paper introduces a new approach to learn from large collections of uncurated video and text obtained from Automatic Speech Recognition Transcript. The authors introduce a dataset of 6M YouTube videos that lead to better results than existing large scale video and language datasets. The proposed architecture consists of separate backbones adapted to images (ResNet50) and text (Transformer) followed by a joint Vision and Language transformer. The model is trained with three different losses: \n* (i) a contrastive text-vision matching loss applied before the joint transformer, \n* (ii) a masked language modeling loss (with a modified masking mechanism to bias towards visual words), \n* (iii) a temporal ordering loss to learn to reorder vidoe frames.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Clearly written\n\n- A new dataset that can be impactful for the community (important for the authors to confirm that the dataset will be released at least in the form of YouTube ids + narrations).\n\n- State-of-the-art results on 12 different benchmarks\n\n- Interesting innovations on the model side: biased masking strategy, contrastive loss coupled with Joint Transformer based approach.\n\n## Weaknesses + clarification points\n\n- More ablation studies: importance of the different losses is not properly ablated such as the temporal ordering loss does not seem to be ablated in the main paper. Similarly the novel masking technique is not compared to the default random masking strategy. Please include these ablations in the rebuttal.\n\n- Confirm that the training dataset will be made available.\n\n- I found the data collection description quite interesting in the Appendix. It might be worth surfacing this more in the main paper. In particular how important was the procedure to clean the transcripts? Have you ablated that? \n\n- It would be nice to also evaluate the learned visual representations since they are trained from scratch. For example it would be nice to compare to [6, 56] on benchmarks such as UCF101 or HMDB51.\n\n- Related to the previous point, would it be possible to perform zero-shot text-to-video retrieval on benchmarks such as YouCook2 or MSRVTT to compare to previous work? In particular most benchmarks are evaluated via specific finetuning while it would be Interesting to be able to evaluate the model without any adaptation.\n\n- The model only takes as input single frame per segment, how can it then model dynamic scenes? Can the authors comment on that limitation and how to address it in future work?\n\n- Minor comments and questions:\n\n  * Typos: atttention -> attention (Table 3, (a)).\n  * Sentence boundaries: how are the text boundaries exacty obtained? In L138 BPE is mentionned with 32 tokens but it seems that a sentence separator is also used. Are the two things combined somehow?\n\n\n## Overall assessment\n\nOverall, MERLOT is a good paper that introduces a good large scale pretraining datasets, interesting architecture innovations to deal with the scale of the data and obtains strong results on several benchmarks. As of now, I lean towards acceptance but request the authors to address the points mentionned above.\n\n## Post rebuttal\n\nAfter reading other reviews and rebuttals, all my questions have been addressed. I believe MERLOT is a strong paper that will likely be influential in the vision and language community.\n\nOf particular interest I believe that showing that leveraging YouTube data can improve **image** tasks such as VCR is quite interesting. Also after the additional ablation requests, I believe this will help make the contributions in terms of architecture and losses clearer in the paper. I hope the authors will include this nicely in the final version.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact were properly addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns for that paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "M2xrjlExcq3", "reviewer": "Reviewer_w8dn", "summary": "The submission proposes a model and dataset for learning representations of multimodal events by training over YouTube videos with ASR captions. The dataset (YY-Temporal-180M) consists of 180 video segments pairs with words extracted from ASR.  The dataset is taken from 6 million YouTube videos and spans multiple domains and topics.  The model (MERLOT) is a transformer based model that is trained on image and language to 1) predict frame-caption match 2) recover masked words (masked language modeling) and 3) predict the temporal ordering of frames (given two frames f1, f2, does f1 come before or after f2).  For 1) a contrastive loss is used and the features for the caption for a segment is extracted not just from the segment but from the entire transcript (it is a contextualized representation for the segment). For 2), attention weights from a language-only transformer is used to favor masking of highly attended-to-tokens so to reduce masking of fillers such as 'umm' or 'yeah' that are common in spoken speech (SpanBERT masking is also applied for robustly against noise). \nExperiments on 14 downstream tasks (mostly video QA tasks) demonstrates the proposed model/dataset/training objective combination outperforms existing state-of-the-art results on these tasks.   There are also ablations comparing the number of epochs to train, the dataset used for training, and training setup (different amount of context, different loss terms).\n\nMain contributions\n- MERLOT: a Transformer-based \n- YY-Temporal-180M - dataset of frames and transcription from a filtered set of YouTube videos\n- Experiments and ablations showing the combination of the proposed model and dataset can outperform prior models on 14 downstream tasks\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Currently, there is many different models using transformer-based models for vision and language.  Despite some novel elements, the overall originality of this work is limited. \n\nQuality: The works looks to be well executed with reasonable rationales for the design choices, and a good set of experiments.\n\nClarity: Overall, the paper is well written and well organized.\n\nSignificance: The dataset (if it can be publicly released) will be useful to foster additional research in this area.  The proposed model and experiments are also useful.\n\nStrengths:\n- The model is well-designed and experiments are well-executed\n- The dataset (YY-Temporal-180M) would be useful for the community\n\nWeaknesses:\n- Poor description of how the proposed model (MERLOT) relates to prior work on pretraining with video and language.  It's unclear specifically where the novelty of the model lies.\n- Lack of comparison with prior models (ClipBERT, ActBERT) pretrainined on the same data.  It is unclear whether pretrained on the same dataset (YY-Temporal-180M), the proposed model performs better at the specified tasks than models from prior work. \n- The ablation studies is not that thorough.  For instance, while there is ablations on the contrastive VL loss and the attention masking, there is no ablation on the temporal ordering loss. \n- The dataset does not seem like it will be easy to download as only links to the associated YouTube videos will be provided.  This also means that the videos may disappear over time.\n\nOther comments:\n\nSome of the claims in the introduction about learning commonsense and script-knowledge representation is not really substantiated - the main evidence for this is that the model outperforms other models on two very specific tasks (the VCR and SortStory tasks).\n\nReferences should be proofread and updated so that the publication venue is clear and consistent.\n- [20],[21] is duplicated\n- [80]: \"Leon Sigal\" => \"Leonid Sigal\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact is discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Privacy and Security (e.g., consent)"], "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "The dataset contains videos involving people and potentially identifiable information.  There are concerns of privacy.  While the videos are uploaded to YouTube and public, it is unclear whether the individuals would be okay with videos of themselves and corresponding transcripts being used for research.   \n\nThe authors documented the concerns and their plan for how to release the data in the appendix.    "}}, {"review_id": "JAf9Ar4TPON", "reviewer": "Reviewer_eAHy", "summary": "In this paper, the authors propose a video-text based self-supervised learning objective to learn strong representations of temporal commonsense. They curate a large scale dataset of videos with transcribed captions (from ASR) from diverse everyday situations to learn stronger and diverse knowledge representations. They perform extensive experiments  on 12 downstream tasks of video question-answering and on the visual commonsense reasoning tasks and outperform state-of-the-art models. They also perform extensive ablation studies to show the importance of pre-training on video datasets compared to image based pre-training. \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contributions of the paper are an end to end vision and language model to learn powerful multimodal representations from videos and transcripts using no labeled data (no bounding boxes and or fine grained captions as in previous work) and a diverse large scale dataset of YouTube videos. They use three objectives to train the joint vision-language model - a contrastive frame caption matching, masked language modeling and temporal ordering of frames. The paper proposes a novel idea and is well written and shows extensive experiments. \n\n1. The contrastive frame-caption matching loss considers the CLS token to compute the loss. How would the model perform by taking the average embeddings of the image matches instead of the CLS level tokens ? How important are the CLS level tokens for the image encoders ?\n\n2. As both the contrastive and temporal re-ordering loss are over the image CLS token embeddings, is this supervision enough to learn good and meaningful image patch representations ([1,1] … [H,W]) ? What if we only have the image level embeddings from the image encoder?\n\n3. Does the video-pre training work better just because the dataset is much larger than  the image ones?  Is it still possible to achieve good performance (reasoning and understanding) with large scale image datasets with strong cross modal attention and architectures ?\n\n\n\n##Post Rebuttal\n\nI have carefully read the authors' response and they address my concerns. I maintain my original rating.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations and impact of their work. One thing to consider is the cost while training on large scale video datasets versus on image datasets and how potentially that can be curbed by not hurting performance and understanding of the visual world. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to jointly learn a visual-language model from a large amount of uncurated videos for (YT-Temporal-180M) with transcribed speech. The authors combine a set of pretraining tasks: image-text contrastive learning, MLM (with a special twist of using attention for selecting words), and temporal predictions of video frames. Reviewers unanimously this is good paper with strong SOTA results across many benchmarks and thoughtful analyses. The authors also thoughtfully respond to ethic reviewers' comments. On my side, I particularly appreciate the additional interesting ablation tests on data processing and ASR that the authors provided during rebuttal. The YT-Temporal-180M seems like a valuable resource for researchers and I hope that a careful release strategy of the dataset will work. I hope the authors can add more details to "Section 5.2 Qualitative analysis: MERLOT learns representations spanning distant frames" and make the figure more viewable. Last point, the title refers to "script knowledge" and to be convincing on that, it would be helpful to have a qualitative analysis on the model understanding of one scenario, e.g., restaurant in details. Given all the plus points of this paper, I recommend Accept.
Decision: Accept (Oral)