VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

1 Introduction . Convolutional neural networks ( CNNs ) [ 53 , 51 ] have triumphed over various computer vision tasks . The inductive bias induced by convolutions , namely translation invariance and locality , are proven effective for the visual data . In the meantime , however , we witness in the natural language processing ( NLP ) community a paradigm shift from the models with strong inductive biases , such as recurrent neural networks [ 43 , 7 ] and CNNs [ 104 , 32 ] , to more general architectures constructed upon selfattention . Particularly , Transformers [ 88 ] have become the de facto model architecture for NLP ∗Work done during an internship at Google . 2https : //github.com/google-research/google-research/tree/master/vatt 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tasks [ 23 , 70 , 71 , 10 ] . Pre-training a Transformer on large text corpora followed by fine-tuning gives rise to state-of-the-art results for different downstream tasks . In view of the success of the attention mechanism in NLP , there has been a rich line of works exploring its potential in computer vision . Early work studied hybrid models consisting of both convolutions and attention modules [ 89 , 94 , 36 , 105 ] . Recent studies showed that convolution-free , specially designed all-attention models can match CNNs ’ performance on image recognition tasks [ 106 , 44 , 73 ] . Most recently , [ 25 ] achieved impressive performance on several image recognition tasks , including ImageNet [ 22 ] , using a pre-trained Transformer with minimal architecture changes . Their work delivered a compelling message that “ large scale ( supervised ) training trumps inductive bias ( for image classification ) . ” This conclusion was further extended to video recognition tasks by [ 9 , 5 ] . However , the large-scale supervised training of Transformers is essentially troubling for two main reasons . First , it rules out the much larger other part of “ big visual data , ” i.e , the vast amount of unlabeled , unstructured visual data . As a result , the supervised training strategy could produce biased systems that require even more labeled data to correct their biases . Second , this strategy fundamentally limits the application scope of Transformers in computer vision because it is costly and extremely time-consuming to collect enough labeled images or videos for training the millions of parameters , choosing hyper-parameters , and validating their expected generalization . Hence , this work poses another pressing question about the Transformers that take raw signals as input . How to empower them with large-scale , unlabeled visual data ? To answer this question , we draw insights from NLP . BERT [ 23 ] and GPT [ 70 , 71 , 10 ] use masked language modeling as their pre-training tasks . Natural languages are organic supervision for Transformers . They sequentially place words , phrases , and sentences into context , granting them semantics and syntax . For visual data , the most organic supervision is arguably the multimodal videos . They are abundantly available in the digital world , and their temporal , cross-modality regulation , and therefore supervision , requires no human annotation . The extreme scale of multimodal videos is potentially capable to teach Transformers necessary priors , as opposed to predefined inductive biases , to model the visual world . To this end , we study self-supervised , multimodal pre-training of three Transformers [ 88 ] , which take as input the raw RGB frames of internet videos , audio waveforms , and text transcripts of the speech audio , respectively . We call the video , audio , text Transformers VATT . Figure 1 illustrates the architecture . VATT borrows the exact architecture from BERT [ 23 ] and ViT [ 25 ] except the layer of tokenization and linear projection reserved for each modality separately . This design shares the same spirit as ViT that we make the minimal changes to the architecture so that the learned model can transfer its weights to various frameworks and tasks . Furthermore , the self-supervised , multimodal learning strategy resonates the spirit of BERT and GPT that the pre-training requires minimal human curated labels . We evaluate the pre-trained Transformers on a variety of downstream tasks : image classification , video action recognition , audio event classification , and zero-shot text-to-video retrieval . Fine-tuning the vision-modality Transformer on ImageNet [ 22 ] obtains the top-1 accuracy of 78.7 % , which is comparable to 79.9 % achieved by ViT . This result is especially appealing considering the domain gap between videos and images , and that ViT is pre-trained using a large-scale , human-curated image dataset . Furthermore , we set new records on Kinetics-400 [ 14 ] , Kinetics-600 [ 15 ] , Moments in Time [ 61 ] , and AudioSet [ 33 ] without supervised pre-training . Our VATT results , along with others reported for NLP tasks [ 23 , 10 ] , image recognition [ 25 ] , semantic segmentation [ 108 ] , point cloud classification [ 107 ] , and action recoginition [ 9 ] , demonstrate that Transformer is a versatile general-purpose architecture for different types of data . To move one step forward , we challenge the Transformers in VATT by a seemingly too strong constraint : sharing weights among the video , audio , and text modalities . The idea is to test whether there exists a single , general-purpose model for all the modalities — of course , they still have their own layers of tokenization and linear projection . Preliminary results are encouraging . This modality-agnostic Transformer is on par with three modality-specific ones of slightly smaller sizes . Finally , another contribution of this work is DropToken , a simple and yet effective technique to reduce the training complexity with a minor reduction of the end Transformers ’ performance . DropToken randomly drops a portion of the video and audio tokens from each input sequence during training , allowing for high-resolution inputs and leveraging their abundance . This is significant for Transformers because their computational complexity is quadratic with respect to the number of input tokens . 2 Related work . 2.1 Transformers in Vision . Transformer was originally built for NLP tasks [ 88 ] and the design of multi-head attention shows its effectiveness on modeling long-term correlation of words . A few attempts have been made to use Transformer for vision tasks like image super-resolution [ 99 ] , object detection [ 11 ] and multimodal video understanding [ 84 , 19 , 57 ] . However these methods still rely on the feature extracted by CNNs . Recently , [ 25 ] proposes a set of convolution-free vision Transformers which directly work on raw images and obtain competitive performance with CNNs . [ 86 ] improves the training data efficiency of [ 25 ] by using stronger data augmentations and knowledge distillation . Since then , the pure Transformer design has been adopted to various vision tasks including semantic segmentation [ 108 ] , point cloud classification [ 107 ] , action recoginition [ 9 , 78 , 5 ] . To the best of our knowledge , our VATT is the first Transformer model on raw multimodal inputs of video , audio and text . 2.2 Self-Supervised Learning . Single vision modality . Early work of self-supervised visual representation learning usually learns from unlabeled images via manually specified pretext tasks , like auto-encoding [ 64 , 102 , 103 ] , patch location prediction [ 24 ] , solving jigsaw puzzles [ 63 ] , and image rotation prediction [ 35 ] . [ 95 ] propose a novel instance discrimination objective . The recent trend of contrastive learning [ 40 , 17 , 100 , 37 , 41 , 85 ] integrates data augmentations and instance discrimination by maintaining relative consistency between representations of an image and its augmented view . Clustering can also provide an effective addition [ 12 ] . Recently , [ 18 ] conduct contrastive learning using ViT [ 25 ] and achieve impressive results . As for the video domain , it is natural to exploit the temporal signals as the pretext task . Examples include predicting the future frame [ 82 ] , motion and appearance statistics [ 90 ] , speed [ 8 , 91 ] and encodings [ 56 , 38 , 39 ] , sorting frames or video clips [ 54 , 97 , 45 , 31 ] . Recently , [ 68 ] apply contrastive learning to videos with a temporal sampling strategy and temporally consistent spatial augmentation . Multimodal video . Video is a natural source of multimodal data . Multimodal self-supervised learning can be achieved by predicting whether a video has correspondence with an audio stream [ 3 , 4 , 62 , 50 ] , cross-modality clustering [ 2 ] , and evolving losses [ 67 ] . Recently , [ 1 ] use contrastive loss to learn from video , audio and text ; [ 74 ] learn to predict a broad view that spans a longer temporal context from a narrow view . VATT serves as a first work combining the strength of convolution-free Transformer and multimodal contrastive learning . 3 Approach . In this section , we introduce our convolution-free VATT architecture and elaborate on the selfsupervised multimodal objectives for training VATT from scratch . Figure 1 is an overview of the architecture . We feed each modality to a tokenization layer , where the raw input is projected to an embedding vector followed by a Transformer . There are two major settings : 1 ) The backbone Transformers are separate and have specific weights for each modality , and 2 ) The Transformers share weights , namely , there is a single backbone Transformer applied to any of the modalities . In either setting , the backbone extracts modality-specific representations , which are then mapped to common spaces to be compared with each other by contrastive losses . We describe each module in the following . 3.1 Tokenization and Positional Encoding . VATT operates on raw signals . The vision-modality input consists of 3-channel RGB pixels of video frames , the audio input is in the form of air density amplitudes ( waveforms ) , and the text input is a sequence of words . We first define a modality-specific tokenization layer that takes as input the raw signals and returns a sequence of vectors to be fed to the Transformers . Besides , each modality has its own positional encoding , which injects the order of tokens into Transformers [ 88 ] . We partition an entire video clip of size T ×H ×W to a sequence of ⌈T/t⌉ · ⌈H/h⌉ · ⌈W/w⌉ patches , where each patch contains t × h × w × 3 voxels . We apply a linear projection on the entire voxels in each patch to get a d-dimensional vector representation . This projection is performed by a learnable weight Wvp ∈ Rt·h·w·3×d . This can be seen as a 3D extension of the patching mechanism proposed in [ 25 ] . To encode the position of these patches , we define a dimension-specific sequence of learnable embeddings as follows : ei , j , k = eTemporali+eHorizontalj + eVerticalk , ETemporal ∈ R⌈T/t⌉×d , EHorizontal ∈ R⌈H/h⌉×d , EVertical ∈ R⌈W/w⌉×d ( 1 ) where ei is the i-th row of E. This scheme allows us to use ⌈T/t⌉+ ⌈H/h⌉+ ⌈W/w⌉ positional embeddings to encode all the ⌈T/t⌉ · ⌈H/h⌉ · ⌈W/w⌉ patches in a video clip . The raw audio waveform is a 1D input with length T ′ , and we partition it to ⌈T ′/t′⌉ segments each containing t′ waveform amplitudes . Similar to video , we apply a linear projection with a learnable weight Wap ∈ Rt ′×d to all elements in a patch to get a d-dimensional vector representation . We use ⌈T ′/t′⌉ learnable embeddings to encode the position of each waveform segment . For text , we first construct a vocabulary of size v out of all words in our training dataset . For an input text sequence , we then map each word to a v-dimensional one-hot vector followed by a linear projection with a learnable weight Wtp ∈ Rv×d . This is equivalent to an embedding dictionary lookup , which has been widely used in natural language understanding [ 60 ] .

Summary: This submission is about multi-modal representation learning from unlabeled data (self-sup.) using Transformer. This work deals with video, audio, and text data as multi-modal data. The approach to tackle representation learning is based on Transformer + contrastive learning. The contrastive learning in this work is interesting in that 1) noisy association is considered in multiple instances learning way [64], and 2) the different level of semantic granularity is carefully considered. Also, the authors adopt a simple DropToken to improve efficiency.  The training requires very large-scale training using HowTo100M and AudioSet. The authors demonstrated its effectiveness on action recognition, audio event classification, zero-shot video retrieval, image classification.  Multi-modal data is encoded by independent feature extractors, which is a small difference from other Vision Transformers. Naturally, the authors extend the positional encoding to 3D positional encoding (horizontal, vertical, and temporal axes) with relativeness.  Overall, the technical contribution is a bit weak, but this submission contains interesting analyses that are worthwhile to report in the community and also achieve state-of-the-art in downstream tasks.