MERLOT: Multimodal Neural Script Knowledge Models

1 Introduction . The human capacity for commonsense reasoning is shaped by how we experience causes and effects over time . Consider the still image of people dining at a restaurant in the bottom right of Figure 1 : while a literal , concrete description like “ people sitting at a table eating '' might be technically correct for the static scene , it doesn ’ t capture the richer temporal , commonsense inferences that are nonetheless obvious : before sitting down , the people had to meet up , agree where to go , and enter the : Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . restaurant ; at present , the man is pointing because the server just came to the table , and she might want to know whose food is whose ; and after , it is likely the server will return to the kitchen to help another table . Teaching machines this type of script knowledge [ 95 ] is a significant challenge in no small part because enumerating all facts , inferences , and counterfactuals is prohibitive . As a result , the highest performing models on vision-and-language tasks , including Visual Commonsense Reasoning ( VCR ) ( where Figure 1 ’ s scene originates from ) , learn about the visual world exclusively through static images paired with literal captions [ 108 , 22 , 69 , 75 , 119 , 36 ] . Though some captions might hint at the past and future , it is not obvious that even training on , e.g. , 400M literal image/text pairs [ 89 ] will result in models capable of temporal reasoning . In this paper , we introduce MERLOT , short for Multimodal Event Representation Learning Over Time . MERLOT is a model that learns commonsense representations of multimodal events by selfsupervised pretraining over 6M unlabelled YouTube videos . With the goal of learning multimodal reasoning capacity beyond static images/literal captions , we train MERLOT to a ) match individual video frames with contextualized representations of the associated transcripts , and to b ) , contextualize those frame-level representations over time by “ unmasking '' distant word-level corruptions [ 27 ] and reordering scrambled video frames . We validate our model on a diverse suite of video tasks , requiring both recognition- and cognition-level reasoning across long and short timescales ; when finetuned , MERLOT achieves a new state-of-theart on 12 such tasks . Additionally , we show that our script-knowledge representations transfer to the single image domain . On Visual Commonsense Reasoning ( VCR ; [ 123 ] ) , our model achieves particularly strong performance , outperforming models that require heavy visual supervision ( in the form of object detection bounding boxes , or images paired with pristine captions ) . Beyond finetuning , we show both quantitatively and qualitatively that MERLOT has a strong outof-the-box understanding of everyday events and situations . Given a scrambled visual story , [ 50 , 2 ] , MERLOT can sort image sequences to match captions which tell a globally coherent narrative . Despite considerable domain shift from videos to static images , MERLOT outperforms strong baselines like CLIP [ 89 ] and UNITER [ 22 ] , which independently match images to text and thus can not reason over long-term contexts as effectively . This capacity for temporal coherence emerges during pretraining : analysis of MERLOT ’ s attention patterns ( Figure 11 ) show that regions attend to captions that are distant in time ( and vice versa ) , allowing it perform cross-modal coreference to piece together a holistic view of situations . Finally , ablations of MERLOT show that 1 ) pretraining works better when we train on videos rather than still images , aided crucially by our strategy of corrupting highly visual words in the masked language modeling task , 2 ) using a diverse set of videos covering many aspects of everyday situations improves downstream performance compared to curated instructional video corpora [ 107 , 80 ] which both cover a smaller slice of the visual world ( confirming hypotheses from past work [ 47 ] ) ; and 3 ) MERLOT ’ s performance does not saturate even after many epochs of training on the pretraining corpus we curated , YT-Temporal-180M , as it continues to improve performance simply with more pretraining . The combination of these results suggests that learning full-stack visual reasoning and multimodal world knowledge from video data is a promising path forward for future research . In summary , our main contributions are : 1 . MERLOT a performant end-to-end vision and language model , that learns powerful multimodal world representations from videos and their transcripts – using no labeled data . 2 . YT-Temporal-180M , a diverse corpus of frames/ASR derived from a filtered set of 6M diverse YouTube videos , which we show greatly aids performance , and 3 . A set of experiments/ablations demonstrating the strong performance of MERLOT on a set of 14 tasks , spanning finetuning and zero-shot transfer , and images and videos . At rowanzellers.com/merlot , we have released code , data , and models for public research use . 2 Related Work . 2.1 Joint representations of written text and images . There is a long history of work on learning joint text-image representations [ 14 ] . Recently , several papers have proposed “ Visual BERT ” models [ 108 , 22 , 8 , 69 , 75 , 119 , 36 ] , trained on image captioning datasets such as MSCOCO [ 71 ] . In general , features are extracted using Anderson et al . [ 10 ] ’ s frozen object detector , which was originally trained on Visual Genome [ 60 ] . Some exceptions are Zhang et al . [ 125 ] , who use an even larger object detector trained on more labeled data ; Kim et al . [ 57 ] , who use an ImageNet-pretrained backbone [ 26 ] , and Shen et al . [ 100 ] , who study a CLIP backbone [ 89 ] pretrained on web image-caption pairs . Overall , these approaches all learn visual representations of static images , and rely on significant human annotation in doing so ( e.g . through literal image descriptions ) . Instead , our approach learns dynamic visual representations purely from videos – their frames , and a transcript of what is said – thus using no human annotation . 2.2 Learning from videos , with automatic speech recognition ( ASR ) transcripts . Prior works have used web videos with ASR to build weakly-supervised object detectors [ 87 ] , action detectors/classifiers [ 120 , 6 , 62 , 84 ] , instruction aligners [ 77 , 5 , 19 ] , video captioners [ 96 , 46 , 86 , 101 ] , and visual reference resolvers [ 49 ] . Of late , works have sought to learn multimodal representations transferable to many tasks from uncurated sets of ( usually how-to ) videos [ 80 , 106 , 107 , 81 , 127 , 9 , 7 , 4 ] ; generally these are applied to video understanding tasks like activity recognition . One challenge is designing an appropriate objective for learning video-level representations . Lei et al . [ 67 ] ’ s ClipBERT model learns vision-language representations from image captions , which more literally describe image content versus the longer ASR transcripts we consider . Tang et al . [ 109 ] use a pretrained dense image captioner [ 59 ] to provide auxiliary labels for web how-to videos . Both approaches use ( supervised ) ResNets pretrained on ImageNet [ 43 ] as their visual backbones . MERLOT is trained using a combination of objectives requiring no manual supervision ; it nonetheless outperforms both prior approaches on downstream tasks . 2.3 Temporal ordering and forecasting . There has been a large body of work on analyzing ‘ what happens next ’ in videos [ 58 ] . Some modeling choices include using pixels [ 34 , 113 ] , graphs [ 11 ] , euclidean distance using sensors [ 3 ] , or studying cycle consistency across time [ 32 ] . In addition to extrapolation , past work has studied deshuffling objectives in videos [ 82 , 115 ] , though this has mostly been limited to the visual modality . In contrast to these papers , our goal is learning multimodal script knowledge representations : using both language and vision as complementary views into the world , instead of just tracking what changes on-screen . 3 MERLOT : Multimodal Event Representation Learning Over Time We now present our unified model for learning script knowledge through web videos ; including our pretraining dataset , architecture , and objectives . 3.1 YT-Temporal-180M . We collect YT-Temporal-180M , a dataset for learning multimodal script knowledge , derived from 6 million public YouTube videos . Our YT-Temporal-180M intentionally spans many domains , datasets , and topics . We began with 27 million candidate video IDs ( which we then filtered ) , including instructional videos from HowTo100M [ 80 ] , lifestyle vlogs of everyday events from the VLOG dataset [ 35 ] , and YouTube ’ s auto-suggested videos for popular topics like ‘ science ’ or ‘ home improvement. ’ Our intent ( in making the corpus as diverse as possible ) was to encourage the model to learn about a broad range of objects , actions , and scenes [ 47 ] : we will later show through an ablation that limiting our pretraining to only instructional videos indeed hurts performance downstream . We filtered videos using the YouTube API , which provides access to videos themselves , their ASR track ( automatically transcribed speech tokens ) , and other metadata . We discard videos 1 ) without an English ASR track ; 2 ) that are over 20 minutes long ; 3 ) that belong to visually “ ungrounded '' categories like video game commentaries ; and 4 ) that have thumbnails unlikely to contain objects , according to a lightweight image classifier . We add punctuation to the ASR by applying a sequenceto-sequence model trained to add punctuation to sentences/paragraphs from news articles . Full details of the scraping and filtering are in Appendix A . Each video V might contain thousands of frames . In this work , we represent a video V as a sequence of consecutive video segments { st } . Each segment st consists of : a. an image frame It , extracted from the middle timestep of the segment , b. the words wt spoken during the segment , with a total length of L tokens . To split the videos into segments , we byte-pair-encode ( BPE ; [ 97 , 88 ] ) each video transcript and align tokens with YouTube ’ s word-level timestamps . This enables us to split the videos into segments of L=32 BPE tokens each ( Appendix A.4 ) ; our final dataset has 180 million segments of this form . 3.2 MERLOT Architecture A diagram of MERLOT is given in Figure 2 . MERLOT takes a sequence of video frames { st } as input . We encode each frame It using an image encoder , embed the words wt using a learned embedding , and jointly encode both using a Transformer [ 112 ] . After pretraining , the architecture can be applied to a variety of vision-and-language tasks with minimal modification . For video QA , for example , we pass several video frames to the image encoder , the question to the text encoder , and extract a single vector representation from the CLS token position . For each task , we learn a lightweight classification head mapping from this hidden state to the task ’ s label space ; specific modeling/optimization details are given in Appendix E.2 . Image encoder . We train our image encoder end-to-end , alongside the rest of the model , from random initialization ( thus without learning from supervised data ) . While most performant visionand-language models pre-extract features from a ( supervised ) object detector [ 108 , 69 , 75 , 22 , 68 ] , for the sake of pre-training efficiency we use a grid-based hybrid ResNet/Vision Transformer.1 Specifically : our encoder uses a ResNet-50 backbone , followed by a 12-layer , 768-dimensional Vision Transformer [ 43 , 112 , 31 ] . We made additional modifications that improve efficiency , including : 1 ) we trained on smaller , widescreen images of size 192x352 ( because most YouTube videos are 1Standard object detectors have expensive operations for proposing regions , and extracting features from those regions ( RoI-pooling ) ; our grid approach avoids these . Recent work has proposed using ‘ grid features ’ broadly [ 53 ] , yet on tasks like VCR these approaches have so far underperformed the more expensive object detector backbones [ 123 ] ; our results suggest that ‘ grid features ’ can perform well broadly . widescreen ) using a patch size of 16x16 pixels ; 2 ) we mirror [ 31 ] ’ s alterations of removing the C5 block in ResNet-50 ; and 3 ) we save compute further by average-pooling the final-layer region cells using a kernel size of 2× 2 . With these modifications , our image encoder requires 40 gigaFLOPs for a forward pass , which is 2 % of the 2 teraFLOPs required for the Faster-RCNN . In summary : given an image of size W ×H , the image encoder will output a W/32×H/32 feature map , along with two CLS hidden states : one for pooling a global representation of the image , and another for pretraining ( Task 1. ) . Joint Vision-Language Encoder . The joint encoder is a 12-layer , 768-dimensional Transformer [ 112 ] , mirroring the RoBERTa base architecture [ 72 ] ; we initialize it with pretrained RoBERTa weights . To compute joint representations , we first embed the tokens { wt } via lookup , and then add position embeddings to both language and vision components ( i.e. , { It } ) . The position embeddings differ between different segments , so as to distinguish between images and captions at different timesteps . Finally , we pass the independent visual and textual feature maps to our joint encoder . The tokens wt in each segment begin with a CLS token ; recall that the feature maps for each frame It start with one as well . At those positions , we will later pool final-layer hidden-state representations , for use in pretraining along with downstream tasks .

Summary: The submission proposes a model and dataset for learning representations of multimodal events by training over YouTube videos with ASR captions. The dataset (YY-Temporal-180M) consists of 180 video segments pairs with words extracted from ASR.  The dataset is taken from 6 million YouTube videos and spans multiple domains and topics.  The model (MERLOT) is a transformer based model that is trained on image and language to 1) predict frame-caption match 2) recover masked words (masked language modeling) and 3) predict the temporal ordering of frames (given two frames f1, f2, does f1 come before or after f2).  For 1) a contrastive loss is used and the features for the caption for a segment is extracted not just from the segment but from the entire transcript (it is a contextualized representation for the segment). For 2), attention weights from a language-only transformer is used to favor masking of highly attended-to-tokens so to reduce masking of fillers such as 'umm' or 'yeah' that are common in spoken speech (SpanBERT masking is also applied for robustly against noise).  Experiments on 14 downstream tasks (mostly video QA tasks) demonstrates the proposed model/dataset/training objective combination outperforms existing state-of-the-art results on these tasks.   There are also ablations comparing the number of epochs to train, the dataset used for training, and training setup (different amount of context, different loss terms).  Main contributions - MERLOT: a Transformer-based  - YY-Temporal-180M - dataset of frames and transcription from a filtered set of YouTube videos - Experiments and ablations showing the combination of the proposed model and dataset can outperform prior models on 14 downstream tasks