VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Authors: Hassan Akbari
Liangzhe Yuan
Rui Qian
Wei-Hong Chuang
Shih-Fu Chang
Yin Cui
Boqing Gong
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-supervised Learning
Multimodal Understanding
Transformer
Contrastive Learning
Video Recognition
Audio Recognition

Abstract: We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training.

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Hassan Akbari' Columbia University ha2436@columbiz edu
Liangzhe Yuan Google 1ayuan@google.com
Rui Qian"
Cornell University rq4 rg498corne11.co edu
Wei-Hong Chuang
Shih-Fu Chang Columbia University
Yin Cui Google
Google
whchuang8google.com con.
sc2508columble edu
y yincui@google.com
Boqing Gong Google bgong8goog1e e com
Abstract
We present a framework for learning multimodal representations from unlabeled data using convolution-frs Transformer architectures. Specifically, Video- Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multi- modal representations that are rich enough benefit variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study modality-agnostic, backoee Transformer by sharing weights among the three modalities. We show that the convolution-fre VATT outperforms state-of-the-ae art Codseee architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82. 1 % on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classifi- cation leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets new record on waveform-basee audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT's source code is publicly available.
Introduction
Convolutional neural networks (CNNs) [53, 51] have triumphed over various computer vision tasks. The inductive bias induced by convolutions, namely translation invariance and locality, are proven effective for the visual data. In the meantime, however,
witness the natural language processing (NLP) community paradigm shift from the models with strong inductive biases, such as recurrent neural networks [43, 7] and CNNs [104, 32], more general architectures constructed upon self- attention. Particularly, Transformers [88] have become the de facto model architecture for NLP
Work done during an internship at Google. http:///gihuub.oom/ooooo www.://wwwoo.eeeoooooooeeeeeeeee
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

VATT
Multimodal Projection
Transformer Encoder
Multimodal Projection Head
LX
audio
Transformer Encoder
MIPP
Modelly-Speciffe
OR ModelllyAAgnosic
Norm
NCE loss
Extra Loamable
[AGG] Embedding
Mult-Head Attention
Medalty-Specffic Patch Position Embedding
Linear Projection (3D RGB voxels)
Linear Projection (1D waveform)
Linear Projection (1 -hat word vactors)
MIL-NCE loss
Norm
"Sied dogs running the snow pulling the
Embedding
text
Input
Input Audio Waveform
Input Text
foature
Figure 1: Overview of the VATT architecture and the self-supervised, multimodal learning strategy. VATT linearly projects each modality into feature vector and feeds it into a Transformer encoder. We define semantically hierarchical common space to account for the granularity of different modalities and employ the Noise Contrastive Estimation (NCE) train the model.
tasks [23, 70, 71, 10]. Pre-training a Transformer large corpora followed by fine-tuning gives rise to statc-of-the-arr results for different downstream tasks.
In view of the success of the attention mechanism in NLP, there has been rich line of works exploring its potential in computer vision. Early work studied hybrid models consisting of both convolutions and attention modules [89, 94, 36, 105]. Recent studies showed that convolution-free specially designed all-attention models can match CNNs' performance on image recognition tasks [106, 44, 73]. Most recently, [25] achieved impressive performance
several image recognition tasks, including ImageNet [22]. using a pre-trained Transformer with minimal architecture changes. Their work delivered compelling message that "large scale (supervised) training trumps inductive bias (for image classification)." This conclusion was further extended to video recognition tasks by [9, 5].
However, the large-scale supervised training of Transformers is essentially troubling for two main reasons. First, it rules out the much larger other part of "big visual data," " i.c, the vast amount of unlabeled, unstructured visual data. As result, the supervised training strategy could produce biased systems that require even more labeled data to correct their biases. Second, this strategy fundamentally limits the application scope of Transformers in computer vision because it is costly and extremely consuming to collect enough labeled images or videos for training the millions of parameters, choosing
and validating their expected generalization.
Hence, this work poses another pressing question about the Transformers that take raw signals as input. How to empower them with large-scale, unlabeled visual data? To answer this question, draw insights from NLP. BERT [23] and GPT [70, 71, 10] use masked language modeling as their pre-training tasks. Natural languages place words, phrases, and sentences into context, granting them semantics and syntax. For visual data, the most organic supervision is arguably the multimodal videos. They abundantly available in the digital world, and their temporal, cross-modality regulation, and therefore supervision, requires no human annotation. The extreme scale of multimodal videos is potentially capable to teach Transformers necessary priors, as opposed to predefined inductive biases, to model the visual world.
organic supervision for Transformers. They sequentially
To this end, we study self-supervised multimodal pre-training of three Transformers [88], which take as input the raw RGB frames of internet videos, audio waveforms, and text transcripts of the speech audio, respectively. We call the video, audio, text Transformers VATT. Figure 1 illustrates the architecture. VATT borrows the exact architecture from BERT [23] and VIT [25] except the layer of tokenization and linear projection reserved for each modality separately. This design shares the same spirit as VIT that we make the minimal changes to the architecture so that the learned model can transfer its weights to various frameworks and tasks. Furthermore, the superviedd, multimodal learning strategy resonates the spirit of BERT and GPT that the pre-training requires minimal human curated labels.
We evaluate the pre-trained Transformers on a variety of downstream tasks: image classification, video action recognition, audio event classification, and zero-shot text-to-video retrieval. Fine-tuning

---

the Ussion-modalle Transformer on ImageNet [22] obtains the top-1 accuracy of 78.7%, which is comparable to 79.9% achieved by VYT. This result is especially appealing considering the domain gap between videos and images, and that VIT is pre-trained using large-scale, human-curated image dataset. Furthermore, we set new records on Kinetics-400 [14], Kinetics-600 [15], Moments in Time [61], and AudioSet [33] without supervised pre-training.
Our VATT results, along with others reported for NLP tasks [23, 10], image recognition [25], semantic segmentation [108], point cloud classification [107], and action recoginition [9], demonstrate that Transformer is a versatile general- purpose architecture for different types of data.
To move one step forward, we challenge the Transformers in VATT by a seemingly too strong constraint: sharing weights among the video, audio, and text modalities. The idea is to test whether there exists a single, general-purpose model for all the modalities - of course, they still have their own layers of tokenization and linear projection. Preliminary results are encouraging. This modality-agnostie Transformer is on par with three modality-specifff ones of slightly smaller sizes.
Finally, another contribution of this work is DropToken, a simple and yet effective technique to reduce the training complexity with a minor reduction of the end Transformers' performance. DropToken randomly drops a portion of the video and audio tokens from each input sequence during training, al- lowing for high-resouution inputs and leveraging their abundance. This is significant for Transformers because their computational complexity is quadratic with respect to the number of input tokens.
2 Related work
2.1 Transformers in Vision
Transformer was originally built for NLP tasks [88] and the design of multi-head attention shows its effectiveness on modeling long-term correlation of words. A few attempts have been made to use Transformer for vision tasks like image nuper-resolution [99]. object detection [11] and multimodal video understanding [84, 19, 57]. However these methods still rely on the feature extracted by CNNs. Recently, [25] proposes a set convolution-free vision Transformers which directly work on raw images and obtain competitive performance with CNNs. [86] improves the training data efficiency of [25] by using stronger data augmentations and knowledge distillation. Since then, the pure Transformer design has been adopted to various vision tasks including semantic segmentation [108], point cloud classification [107], action recoginition [9, 78, 5]. To the best of our knowledge, our VATT is the first Transformer model on raw multimodal inputs of video, audio and text.
2.2 Self-Supervised Learning
Single vision modality. Early work of self-supervised visual representation learning usually learns from unlabeled images via manually specified pretext tasks, like auto-encoding [64, 102, 103], patch location prediction [24], solving jigsaw puzzles [63]. and image rotation prediction [35]. [95] propose novel instance discrimination objective. The recent trend of contrastive learning [40, 17, 100, 37, 41, 85] integrates data augmentations and instance discrimination by maintaining relative consistency between representations of an image and its augmented view. Clustering can also provide an effective addition [12]. Recently, [18] conduct contrastive learning using ViT [25] and achieve impressive results. As for the video domain, it is natural exploit the temporal signals as the pretext task. Examples include predicting the future frame [82]. motion and appearance statistics [90], speed [8, 91] and encodings [56, 38, 39], sorting frames or video clips [54, 97, 45, 31J. Recently, [68] apply contrastive learning to videos with temporal sampling strategy and temporally consistent spatial augmentation.
Multimodal video. Video is natural source of multimodal data. Multimodal self-supervised learning can be achieved by predicting whether video has correspondence with an audio stream [3, 4, 62, 50], cross-modality clustering [2], and evolving losses [67]. Recently, [1] use contrastive loss to learn from video, audio and text; [74] learn to predict a broad view that spans a longer temporal context from a narrow view. VATT serves as a first work combining the strength of convolution-fies free Transformer and multimodal contrastive learning.
3

---

3 Approach
In this section, we introduce our convolution-free VATT architecture and elaborate on the self- supervised multimodal objectives for training VATT from scratch.
Figure 1 is an overview of the architecture. We feed each modality to tokenization layer, where the raw input is projected to an embedding vector followed by a Transformer. There are two major settings: 1) The backbone Transformers are separate and have specific weights for each modality, and 2) The Transformers share weights, namely, there is a single backbone Transformer applied to any of the modalities. In either setting, the backbone extracts modality-specifie representations, which are then mapped to common spaces to be compared with each other by contrastive losses. We describe each module in the following.
3.1 Tokenization and Positional Encoding
VATT operates on raw signals. The vision-modalie input consists of 3-channel RGB pixels of video frames, the audio input is in the form of air density amplitudes (waveforms). and the text input is sequence of words. We first define a modality-specifff tokenization layer that takes as input the raw signals and returns sequence of vectors be fed to the Transformers. Besides, each modality has its own positional encoding, which injects the order of tokens into Transformers [88]. We partition an entire video clip of size T x H x W to sequence of [T/t] [H/h] [W/w]] patches, where each patch contains t x h x w x 3 voxels. We apply a linear projection on the entire voxels in each patch to get d-dimensional vector representation. This projection is performed by a learnable weight W up E Rf---ww33dd This can be seen as a 3D extension of the patching mechanism proposed in [25]. To encode the position of these patches, we define a cemeesion-sppciife sequence of learnable embeddings as follows:
Ci,j. = @Temporal; + Cllrizomtaly + evertical
Temporal E IR [T/t]xdd Hoorizantl E RR [H/ Everiial E R [W/w]] xd
where e; is the i-th row of E. This scheme allows us to use [T/4] + [H/N]] + [W/w] positional embeddings to encode all the [T/[] [ [H/N]] [W/w]] patches in video clip. The audio waveform is a 1D input with length T", and we partition it to [T'/t'] segments each containing t' waveform amplitudes. Similar to video, we apply linear projection with learnable weight patch to get a d-dimensional vector representation. We use /t"
Wap E I xd all elements learnable embeddings to encode the position of each waveform segment. For text, we first construct a vocabulary of size a out of all words in our training dataset. For an input text sequence, we then map each word to e-dimensional one-hot vector followed by a linear projection with a learnable weight Wip E Rexd This is equivalent to an embedding dictionary lookup, which has been widely used in natural language understanding [60].
3.1.1 DropToken
We introduce DropToken, a simple and yet effective strategy to reduce the computational complexity during training. Once we get the token sequence for the video or audio modality, we randomly sample a portion of the tokens and then feed the sampled sequence, not the complete set of tokens, to the Transformer. This is crucial for reducing the computational cost because a Transformer's computation complexity is quadratic, O(N2), where is number of tokens in the input sequence. Any effort on reducing the input length would reduce the number of FLOPs quadratically. This has an immediate impact on the wall clock time for training these models and makes it possible to host large models in limited hardware. We argue that instead of reducing the resolution dimension of the raw inputs, it is better to keep a gigh-ficliity input and randomly sample the tokens via DropToken. DropToken is appealing especially with the raw video and audio inputs, which may contain high redundancies.
3.2 The Transformer Architecture
For simplicity, we adopt the most established Transformer architecture [23], which has been widely used in NLP. Similar to ViT [25], we do not tweak the architecture so that our weights can be easily transferred to any standard Transformer implementation. We will briefly elaborate on the pipeline (also illustrated in Figure I middle panel) and refer the reader to [25, 23] for more details of the

---

standard Transformer architecture. The sequence of input tokens to the Transformer follows the below formulation:
Zin (GGG Zo Wp; Wp: N Wpl + EPOS
(2)
where Tn is the input patches sequence and TAGG is the learnable embedding of a special aggregation
token whose corresponding output in the Transformer (=om) is used the aggregated representation
for the entire input sequence. This will be later used for classification and common space mapping. We standard self-attention [88] as the Bulli-Head-AAtenntion (MHA) module, and GeL.U [42] as the activation in the MIP layer. We also use Layer Normalization [6] before the MHA and MIP modules. In our text model, we remove the position encoding epos and add learnable relative bias to each attention score of the first layer in the MHA module. This simple change makes our text model's weights directly transferable the state-of-the-an text model T5 [72].
3.3 Common Space Projection
We use common space projection and contrastive learning in that common space to train our networks. More specifically, given vidco-audio-tex triplet, we define semantically hierarchical common space mapping that enables us to directly compare video-audio pairs as well as video-text pairs by the cosine similarity. As argued in [1], such comparison is more feasible if we assume there are different levels of semantic granularity for these modalities. To achieve this, we define multi-level projections follows:
20,00 = Du-vua (Fout
video )
2a. = 9o -vea (Zu
audio
(3)
2t, St- Fout
Zn,ot - So- (Zu,no)
where gu-rea and Jo-+ra are the projection heads to respectively map the video and audio Trans- formers' outputs to the video-audio common space Sea- Moreover, It-set and Ju-ret project the text Transformer's outputs and the video embedding in the Sea space video-text common space, Syt- This multi-level common space projection is depicted in Figure (the rightmost panel). The main intuition behind this hierarchy is that different modalities have different levels of semantic granularity, so we should impose this as an inductive bias in the common space projection. Similar to [1], we use linear projection for ga-vo (.), It-vet (-), and oe-) and a two-layer projection with ReLU in between for Je-vea (.). To ease the training, a batch normalization is used after each linear layer.
3.4 Multimodal Contrastive Learning
Inspired by [1, 3, 59], we use Noise Contrastive Estimation (NCE) to align video-audio pairs and Multiple Instance Learning NCE (MIL-NCE) to align video-text pairs. The pairs are composed from different temporal locations in the iideo-audio-tex" stream. Positive pairs from two modalities constructed by sampling their corresponding streams from the same location in the video, and negative pairs are constructed by sampling from any non-matching locations in the video [1]. Concretely, given the common space specified in Section 3, the loss objectives can be written as follows:
exp(z,, Za,vo/7)
NCE( Eo,eay Za,va) log
exp(zT (0 2a,va/ Even exp(2" v.va v.vaz'a,vel awv//t)
(4)
Ex.Ep exp (z. (tt
MIL-NCE( Zu, (2),nt)) = -log
Ex,..ep ftp
EveN exp(2' o (we//
(S)
where N contains all non-matching pairs in batch. In Equation 5, P contains five text clips that are nearest neighbors to the video clip time, т a temperature to adjust the softness of the objectives in distinguishing the positive pairs from the negative pairs.
The overall par-s objective for training the entire VATT model end-to-end is as follows:
L = NCE( (ze za,va) + AMIL-NCE (z
where balances the two losses. The model optimized based on the back paok-propagation of the average loss calculated over a batch of samples.
5

---

4 Experiments
In this section, we first briefly describe the experimental setup for the pre-training and downstream evaluation, and then present the results and analytic interpretation of VATT in different tasks. We refer the reader to the Appendix for a more detailed description of all experimental settings.
4.1 Experimental Setup
Pre-train: we use a combination of AudioSet [33] and HowTol00M [58] datasets to pre-train VATT- we use only a subset of the HowTo100M dataset in compliance with Youtube's policies. Following [1], we use video-audio-text triplets from NowTo100M clips while only using video-audio pairs from AudioSet. We sample 32 frames at 10 fps with a spatial size of 224 x 224 following a random crop, horizontal flip and color augmentation (details in A.2.1). Accordingly, we sample audio waveforms in sync at 48kHz. Both video and audio are normalized between [-1,1]. We use patch sizes of 4 x 16 x 16 and 128 for video and raw waveform tokenization, respectively (ablation in A.5). We use one-hot vectors to encode text sequences (capped to 16 tokens) with the vocabulary size of 216 In all pre-training experiments, we use Drop'Token with drop rate 50%. We train our models using the Adam optimizer [46] with a couuteer-perio cosine scheduled learning rate from le-4 to Se-5 and 10k warmup steps. Optimization is performed totally 500k steps with batch size 2048 (512 in exploration experiments). Following the previously established practice [1] for the projection to the common spaces S.. and S.t, we d,s 512 and dut 256. We also use the temperature of 7= 0.07 and the weight of À = in the loss in Equation 6. We use 4 network sizes in our experiments (details in A.2.2). We the Medium model (155M parameters) for (VATT-MA), and 3 variants for the modelity-specific video-audio-tex backbones: Base-Base-Small (BBS; 197M), Medium-Base Base--malll (MBS; 264M), and Large-Base-Smal (LBS; 415M). Pre-training an MBS VATT with batch size 2048 on 256 TPUs (v3) takes less than 3 days. Pre-training with batch size 512 takes less than 1 day.
moodality-agnostte variant
evaluate the pre-trained VATT models on 4 major downstream tasks using total of 10 datasets. We use UCFI01 [81], HMDB51 [52], Kinetics-400 [14], Kinetics-600 [15]. and Moments in Time [61] for video action recognition. We use ESC50 [66] and AudioSet [33] for audio event classification, and we evaluate the quality of our video-text common space representations by zero-shot text-to-video retrieval on YouCook2 [109] and MSR-VTT [98]. Finally, we evaluate the transferability of the vision backbone by fine-tuning it on ImageNet classification [22]. Since HMDB51, UCF101, and ESC50 are very small datasets compared the size of our networks, we only use them to train a linear classifier on top of the frozen pre-trained backbones. In our exploration experiments, we report linear classification accuracy and zero-shot video retrieval metrics. We refer to the Appendix for detailed description of the datasets and the experimental setup.
Downstream:
4.2 Results
4.2.1 Fine-tuning for video action recognition
We fine-tune VATT's vision Transformer on Kinetics-400, Kinetics-600, and Moments in Time, three of the arguably most established large-scale datasets for video action recognition. We use the final checkpoints of four pre-train settings for these experiments: three modality -spectyy- variations (LBS, (Medium). Table 1 shows the results compared with the
MBS, BBS), and one modality-agnosii state-of-the-art video models. On all three datasets, we achieve higher accuracy than previous works including TimeSFormer [9], recent effort in fine-tuning the VIT checkpoints obtained by supervised pre-training. In contrast, our pre- pre-training does not rely on any labels curated by humans. To the best of our knowledge, VATT provides the first vision Transformer backbone that is pre-trained from scratch using self-supervision on multimodal videos and achieves state-of-the-e results on video action recognition. It is also worth mentioning that fine-tuning VATT on the most recent Kinetics-700 dataset results in a top-1 accuracy of 72.7%, which outperforms the state-of saate-of-the-aa top-1 accuracy of 72.4% in [47].
To further quantify how much the multimodal self-supervised pre-training helps in achieving these numbers, we train variant from scratch without any pre-training and observe the top-1 and top-5 accuracies of 26.4% and 51. .8% on Kinetics-400, respectively. The low accuracies verify the efficacy of our pre-training strategy for VATT. Finally, we find that VATT-MA-Medium the modality-

---

Kinetics-400
Kinetics-600
ImageNet without Moments Time
METHOD
Top-1
Top-5
ToP-1
ToP-5
Top-1
Top-5
TFLOPS
I3D [13]
71.1
89.3
71.9
90.1
29.5
56.1
R(2+1)D [26]
72.0
90.0
17.5
bLVNet [27]
73.5
91.2
31.4
59.3
0.84
S3D-G [96]
74.7
93.4
Oct-I3D+NL [20]
75.7
76.0
0.84
D3D [83]
75.9
77.9
13D+NL [93]
77.7
93.3
10.8
ip-CSN JD-CNN-152 152 [87]
77.8
92.8
3.3
AttentionNAS [92]
79.8
94.4
32.5
60.3
1.0
AssembleNct-001 [77]
34.3
62.7
MoViNet-A5 [47]
78.2
82.7
0.29
LGD-3D-101 [69]
79.4
94.4
81.5
95.6
SlowFast-R101-N [30]
79.8
93.9
81.8
95.1
7.0
X3D-XL [29]
79.1
93.9
81.9
95.5
1.5
X3D-XXL [29]
80.4
94.6
5.8
TimeSFormer-L [9]
80.7
94.7
82.2
95.6
7.14
VATT-Base
79.6
94.9
80.5
95.5
67.5
9.09
VATT-Medium
81.1
95.6
82.4
96.1
39.5
68.2
15.02
VATT-Large
82.1
95.5
83.6
96.6
41.1
67.7
29.80
VATT-MA MATT-MA-Medinee Medium
79.9
94.9
80.8
95.5
37.8
65.9
15.02
Table 1:
Video action recognition accuracy on Kinetics-400, Kinetics-600 and Moments in Time.
backbone shared by the video, audio, and modalities, on par with the modality specific VATT- Base when fine-tuned for the video action recognition. This result is encouraging as it indicates the potential of unifying three data modalities by a single Transformer backbone.
4.2.2 Fine-tuning for audio event classification
We fine-tune VATT's audio Transformer on AudioSet, which benchmarks the task of multi-label audio event classification. We use the final checkpoints of two pre-train settings: one modality-specinf (BBS), and one modality-agnostic (Medium). Table shows the results compared to state-of-the-ar models. Following common practice [34, 48], we report mean Average Precision (mAP), Area Under Curve (AUC), and d-prime (based on AUC) [34]. Our audio Transformer consistently outperforms the existing ENNN-ased models in all metrics. More interestingly, fine-tuning the modality- backbone
one (VATT- -Bsse). To the best of our knowledge, VATT is the first Transformer that outperforms CNN-based models in audio event recognition. VATT operates on raw waveforms and does not utilize any handcrafted features.
is on par with fine-tuning the modality-spech
4.2.3 Fine-tuning for image classification
In this section, show that pipeline capable of transferring the learned knowledge into another domain by performing the image classification task, even though the models are pre-trained in the multimodal video domain. We fine-tune the vision Transformer VATT-BBS any modification to the backbone architecture. Instead, to satisfy the voxel-to-patch layer we replicate the input image 4 times and feed it to the network. The network sees the input as a single-frame video clip and performs spatial self-attention. Table shows the results for fine-tuning the vision Transformer end-to-end on ImageNet. We can see that our pre-training leads to significant boost in the accuracy compared to training from scratch. We also observe that even though the self-supervised pre-training happens in the video domain, we still achieve competitive results to the supervised pre-training using large-scale image data [25].
4.2.4 Zero-shot text-to-video retrieval
We feed video-text pairs to VATT-MBS, and extract representations in the S.r space. We then calculate the similarity between each video-text pair from YouCook2 and MSR-VTT. Given a text query, we rank the videos based their similarities to the text. We then measure the recall for the

---

METHOD
PRE-TRAINING DATA
ToP-1
ToP-5
iGPT-L [16]
ImageNet JFT
72.6
METHOD
mAP
AUC d-prime
ViT-Base [25]
79.9
DaiNet [21]
29.5
95.8
2.437
VATT-Base
64.7
83.9
LeeNetll [55]
26.6
95.3
2.371
VATT-Base
HowTo100M
78.7
93.9
LeeNer24 [55]
33.6
96.3
2.525
ResIdNet31 [49]
36.5
95.8
2.444
Table 3: Finetuning results for ImageNet classification.
ResldNet51 [49]
35.5
94.8
2.295
Wavegram-CNN [49]
38.9
96.8
2.612
YouCook2
MSR-VTI
METHOD
BATCH EPOCH
R 1O MedR R 1O MedR
VATT-Basc
39.4
97.1
2.895
MIL-NCE [59]
8192
27
51.2
10
32.4
30
VATT-MA-Medium
39.3 97.0
2.884
MMV [1]
4096
8
45,4
13
31.1
38
VATT-MBS
2048
4
45.5
13
29.7
49
Table 2: Finetuning results for AudioSet event classification.
VATT MATT-MA-NE MA-Medium 2048
4
40.6
17
23.6
67
Table 4: Zero-shot text-to-video retrieval.
correct video in the top-10 10 videos. We also measure the median of the rank of the correct video. Table 4 compares our video retrieval results to two baselines. In our experiments we observe that the zero-shot retrieval results are heavily affected by the batch size and number of epochs, confirming the observation made in [1]. That said, our model still delivers comparable results to MMV [1] while being pre-trained with a half number of epochs and a half batch size of theirs. We also experiment with a larger batch size 8192 and longer pre-traie pre-training for 6 epochs, arriving at exactly the same results as MIL-NCE [59] on YouCook2 and the R@ @ 10 of 29.2 and MedR of 42 on MSR- VTT. We also notice that, probably due to the noisy nature of text transcripts, a sophisticated language model like ours is underrated. As shown in [1], using a simple linear projection would still perform reasonably well. It is worth exploring other, higher-quality text sources in future work.
4.2.5 Feature visualization
We take our modality-specifff and modality-agnostic VATT fine-tuned on Kinetics-400 and visual- ize their output feature representations using t-SNE. For comparison, we also include the feature visualization the vision Transformer trained from scratch on Kinetics-400. From Figure 2, we observe that the fine-tuned VATT yields much better separation than the model trained from scratch. Furthermore, it is worth noting that there is no clear difference between the modality-agnostii features and the modality-speeiifii ones.
We further investigate the VATT backbones without any fine-tuning. We randomly choose Ik video clips from the YouCook2 dataset and store the representations from two points of a pre-trained VATT model. One after the tokenization layer (input space of the Transformer), and the other is after the common space projection (output space), where the loss is computed. Figure 3-top visualizes the representations, comparing modality-specifft VATT to modality- observe that the representations are slightly more mixed together in the modality- compared to the modality-specife ones, implying that the modality-agnostte backbone sees different modalities as different symbols describing the same concept. This is analogous to unified language model in NLP that supports multiple languages.
VATT. Interestingly, we setting
To see how well VATT distinguishes positive vidco-text pairs from randomly sampled pairs, we calculate pair-wise similarities for all possible pairs and perform a Kernel Density Estimation (KDE) to visualize the distributions of the similarities of the positive pairs vs. negative pairs. We perform this procedure for both input and output spaces of the -sseeiiit-specifiic and moddllity-ggnoste backbones. Figure 3-bottom shows the KDE curves of these similarities. We can see that VATT in both settings separates the positive and negative pairs in its output space. This verifies VATT's efficacy in learning semantic common space for different modalities, even if we share the backbone across modalities.
4.2.6 Model Activations
We measure the average activation of the modality-agnostic VATT when full multimodal input is fed to the model. More specifically, we sample 100k short video clips from the test split of HowTo100M along with their corresponding audio and text and feed them to the model separately. For each
8

---

From Scratch
Podalitt-Spediiin
Modality-Annosti
1s8
tain
Figure 2: t-SNE visualization of the feature representations extracted by the vision Transformer in different training settings. For better visualization, we show 100 random classes from Kinetics-400.
Input / (--Spfcific
Output / M-Spec ific
Input M- M-Agnostic
Output M- M-Agnastic
Modakiic
Modalic
bis
Nocultiss
Test
Text
Teat
Teat
P-SNE-
-11
E-NE-11
t:
t-SNE-1
t.SNE:
Pals
Pars
200
Takis Poalive
Pars tesliie
Pookive
10-
Denotty
Denillt
Denstty
Similariies
Similarities
Similarities
Figure 3: t-SNE visualization and distribution of pair-wise similarities of the input space output space for modality-specie -specifcc and modality- aggnosicc backbones when different modalities are fed.
modality, we calculate the average activation of each node at the output of the MLP module, before the residual addition (Figure 1-Transformer Encoder). Figure 4 shows the average activations across all nodes in a Medium-size model. We observe that earlier nodes in the model are activated with the text inputs, while the middle-to-later nodes activated with video and audio modalities. However, the nodes in the last layers of the network are activated with all modalities almost equally. This might suggest that the model allocates different nodes certain modalities while reaching the same level of semantic perception for all modalities in the later layers. Such observation encourages further studies on the possibility of utilizing Mfxture-of-Expee [79, 28, 76] to increase the model's capacity for simultancous multimodal perception. We leave this direction of research for future work.
4.2.7 Effect of DropToken
We introduced a new method reduce the redundancy in high-resolution data. To study the effect of the proposed DropTokke method on downstream applications and the pre-training computation, we perform pre-training by randomly dropping 75%, 50%, 25%, and 0% (no drop) of the tokens from the video and audio inputs. Table 5 shows the accuracy of linear classification on HMDB51, UCF101, ESC50 and @@ 10 on YouCook2 and MSR-VTT vs. the drop rate along with GFLOPs during forward call. We choose 50% sampling rate for our large-scale pre-training as it offers good trade-off between accuracy and computational costs. We then take the final checkpoint of the pre-trained VATT with 50% Drop'Token rate and perform fine-tuning on Kinetics-400 at different DropToken rates and different spatial and temporal resolutions see how high-resolution inputs coupled with DropToken compare to low-resolution inputs with no tokens dropped during fine-tuning. Table shows the top-1 accuracy on Kinetics-400. We argue against using low-resolution inputs, which is the most common approach to reduce the computational cost during training. Instead, we suggest using high-resolution inputs with DropTokenn whose accuracy and training cost are comparable or better than low-rese oowresooutionn counterparts.

---

Late Activation for Video & Audio
Modality
Audio
Input Modalities
Text
Text
0
2000
4000
6000
8000
10000
12000
14000
16000
Node #
Early Activation for Text
Modality-gnostic Activation
Figure 4: The average node activation across the Modality-AgnosticMMMciu VATT while feeding
multimodal video-audio-ten triplet to the model.
Token Drop
Resolution/
DropToken Drop Rate
75%
50%
25%
0%
FLOPs
75%
50%
25%
0%
Multimodal GFLOPss
188.1
375.4
574.2
784.8
32 x 224 x 224 Inference (GFLOPs)
79.9
548.1
HMDB51
62.5
64.8
65.6
UCF101
84.0
85.5
87.2
87.6
64 x 224 x 224 Inference (GFLOPs)
80.8
ESC50
78.9
84.1
84.6
84.9
1222.1
YouCookII MSR-VTT
17.9
20.7
24.2
23.1
32 320 x 320 Inference (GFLOPs)
79.3
80.2
80.7
81.1
14.1
14.6
15.1
15.2
279.8
572.5
898.9
1252.3
Table 5: Top-1 accuracy of linear classifica- tion and @ 10 of video retrieval vs. drop rate vs. inference GFLOPs in the VATT-MBS.
Table 6: Top-1 accuracy of video action recogni- tion on Kinetics400 using high-resolution inputs coupled with DropToken vs. low-resolution inputs.
5 Conclusion and Discussion
In this paper, we present self-supervised multimodal representation learning framework based on Transformers. Our study suggests that Transformers are effective for learning semantic even one model shared across modalities modal sule-super pre-training is promising for reducing their dependency on large-scale labeled data. We show that DropTooken can significantly reduce the pre-training complexity with video and audio modalities and have minor impact the models' generalization. We report records of results on video action recognition and audio event classification and competitive performance on image classification and video retrieval. Having these results, we still see some limitations in work. Firstly, not all videos have organic audio or speech, while our approach depends on meaningful multimodal correspondences. Besides, the text modality currently consists of speech transcripts, which are noisy and sometimes sparse. Potential negative Societal Impacts are mainly concerned with upplications. The models could be biased if one applies our approach to the multimodal videos that are not representative enough. Finally, our method is still demanding in computation, though we managed to avoid the need for human labels. Future work can improve upon these limitations.
video/audio/text representations
multi-
Acknowledgments and Disclosure of Funding
We would like to thank Min-Hsuan Tsai, Jean- Baan-Bapiis Alayrac, Andrew Audibert, Yeqing Li, Vidush Mukund, and the TensorFlow team for their help with codes, infrastructure, and insightful discussions.
10

---

References
[1] Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Rama- puram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self- supervised multimodal versatile networks. In NeurlPS, 2020. 3,5, 6, 8, 17, 18, 19,
[2] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and
Du Tran. Self-supervised learning by cross-modal audio-video clustering. arXiv preprint arXiv:1911.1266,, 2019. 3, 20
[3] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In CVPR, 2017. 3,5
[4] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018.
[5] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint arXiv:2103.1569) 2021. 2, 3
[6] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint 107.v:/607.064500 2016.
[7] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. 1
[8] Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubin- stein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness videos. CVPR, 2020. 3
[9] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2/02.05095, 2021. 2, 3, 6,
[10] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020. 2.
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[12] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
[13] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 7
[14] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 3, 6, 17
[15] Joao Carreira, Eric Noland, Andras Banki-Horvathe Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: 1808. 180..88....380 01340, 2018 6, 17
[16] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020. 8
[17] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020. 3
[18] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised visual transformers. arXiv preprint wxXv:2204.02007, 2021. 3
[19] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020. 3
[20] Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Jiashi Feng. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In ICCV, 2019. 7

---

[21] Wei Dai, Chia Dai, Shubui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In ICASSP, 2017.
[22] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 3, 6, 17
[23] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: tre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. 2, 3, 4
[24] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. 3
[25] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 2, 3, 4, 7,
[26] Heng Wang Du Tran, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. 2018 ieee. In CVPR, 2017. 7
[27] Quanfu Fan, Chun-Fu (Ricarhd) Chen, Hilde Kuchne, Marco Pistoia, and David Cox. More Is Less: Learning Efficient Video Representations by Temporal Aggregation Modules. In NeurIPS. 2019.
[28] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. 2021.
[29] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020. 7
[30] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In /CCV, 2019. 7, 18
[31] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould, Self-superviedd video representation learning with odd-one-out networks. In CVPR, 2017. 3
[32] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolu- tional sequence to sequence learning. In /CML, 2017.
[33] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Chan ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017. 3, 6, 17
[34] Jort Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017. 7
[35] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018. 3
[36] Rohit Girdhar and Deva Ramanan. Attentional pooling for action recognition. In NeurIPS, 2017. 2
[37] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bemardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. In NeurIPS, 2020.
[38] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCV Workshops, 2019. 3
[39] Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmenteed dense predictive coding for video representation learning. In ECCV, 2020. 3
12

---

[40] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3
[41] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: /9050992527 2019. 3
[42] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.084/ 2016. 5
[43] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997. 1
[44] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, 2019. 2
[45] Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, 2019. 3
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: ar.iv:/412.6600 /412.6980 2014. 6, 18
[47] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. CVPR, 2021. 7,
[48] Qiuqiang Kong, Changsong Yu, Yong Xu, Turab Iqbal, Wenwu Wang. and Mark D Plumbley. Weakly labelled audioset tagging with attention neural networks. TASLP, 2019. 7
[49] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. TASLP, 2020. 8, 19
[50] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. NeurIPS, 2018. 3, 20
[51] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
[52] Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: large video database for human motion recognition. In ICCV, 201 6, 17
[53] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[54] Hsin- Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised repre- sentation learning by sorting sequences. In /CCV, 2017. 3
[55] Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, and Juhan Nam. Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms. arXiv preprint arXiv:/70..07789, 2017.
[56] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08100 2016. 3
[57] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. Univilm: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint 20t:22002.0035533 2020.
[58] Antoine Miech, Dimitri Zhukov, dean-Baptiste Alayrac Makarand Tapaswi, Ivan Laptev, and Josef Sivie. 10wto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019. 6, 17
13

---

[59] Antoine Miech, Jean- Caan-Bpptsste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020. 5, 8, 17, 20
[60] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.378, 2013. 4, 18
[61] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. TPAMI, 2019. 3, 17 17
[62] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross- cross-modal agreement. arXiv preprint a1X1v:2004./2943, 2020. 3
[63] Mehdi Noroozi and Paolo Favaro, Unsupervised learning visual representations by solving
jigsaw puzzles. In ECCV, 2016. 3
[64] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016. 3
[65] Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, João Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transforma- tions. arXiv preprint arXiv:2003.088 04298, 2020. 20
[66] Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In ACM MM, 2015. 6,
17
[67] AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unsupervised video representation learning. In CVPR, 2020. 3, 20
[68] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsua Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, 2021. 3
[69] Zhaofan Qiu, Ting Yao, Chong-Wait Ngo. Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffusion. In CVPR, 2019. 7
[70] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[71] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[72] Colin Raffel, Noam Shazeer, Adam Roherts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
[73] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: arXiv: /906/099999 2019. 2
[74] Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac. Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, al. Broaden your views for self-supervised video learning, arXiv preprint 165559 2021. 3
[75] Steffen Rendle. Factorization machines. In ICDM, 2010. 19
[76] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. arXiv preprint arXiv:2106.05974 2021. 9
[77] Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity in video architectures. arXiv preprint 1rXiv:1905.13209, 2019. 7
14

---

[78] Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manor An image is worth 16x16 words, what is video worth? arXiv preprint arXhv:2103.13915 2021.
[79] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate -ixture-of-experts layer. arXiv preprint /rXI 7707006508, 2017.
[80] Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv: 191.::499220443. 2019. 20
[81] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, arXiv preprint arXiv: 12XXvv:2/22.402, 2012. 6, 17
[82] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In ICML, 2015.
[83] Jonathan Stroud, David Ross, Chen Sun, Jia Deng, and Rahul Sukthankar. D3d: Distilled 3d networks for video action recognition. In WACV, 2020. 7
[84] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video represen- tations using contrastive bidirectional transformer. arXiv preprint (00i::/900.0544555 2019.
3
[85] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020. 3
[86] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[87] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video elassification with channel- separated convolutional networks. In ICCV, 2019. 7
[88] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:/706.037665 2017. 1, 2, 3,4,5 4,5
[89] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In CVPR, 2017.
2
[90] Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self- supervised spatio-temporal representation learning for videos by predicting motion and ap- pearance statistics. In CVPR, 2019. 3
[91] Jiangliu Wang, Jianbo Jiao, and Yui-Hui Liu. Self-supervised video representation learning by pace prediction. In ECCV, 2020. 3
[92] Xiaofang Wang, Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael Ryoo, Anelia Angelova, Kris M Kitani, and Wei Hua. Attentionnas: Spatiotemporal attention cell search for video classification. In ECCV, 2020. 7
[93] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 7
[94] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, 2018. 2
Joon-Young
[95] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
[96] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. ECCV, 2018. 7
15

---

[97] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie. and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. CVPR, 2019.
[98] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 6, 17
[99] Fuzhi Yang, Huan Yang. Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In CVPR, 2020.
[100] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019. 3
[101] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez- Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017. 19
[102] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016. 3
[103] Richard Zhang, Phillip Isola, and Alexei Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, 2017. 3
[104] Xiang Zhang, Junbo Zhao, and Yann Lecun. Character-level convolutional networks for text classification. NeurIPS, 2015. 1
[105] Y Zhang, K Li, K Li, B Zhong, and Y Fu. Residual non-local attention networks for image restoration. In /CLR, 2019. 2
[106] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recogni- tion. In CVPR, 2020. 2
[107] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. arXiv preprint arXiv:2012.09164 2020.
[108] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei
Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-scquence perspective with transformers. arXiv preprint arXiv:2012.15840, 2020.
3
[109] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 6, 17
16


Reviews:
[{"review_id": "iKH7Daqp24d", "reviewer": "Reviewer_taBL", "summary": "This submission is about multi-modal representation learning from unlabeled data (self-sup.) using Transformer. This work deals with video, audio, and text data as multi-modal data. The approach to tackle representation learning is based on Transformer + contrastive learning.\nThe contrastive learning in this work is interesting in that 1) noisy association is considered in multiple instances learning way [64], and 2) the different level of semantic granularity is carefully considered.\nAlso, the authors adopt a simple DropToken to improve efficiency.\n\nThe training requires very large-scale training using HowTo100M and AudioSet. The authors demonstrated its effectiveness on action recognition, audio event classification, zero-shot video retrieval, image classification.\n\nMulti-modal data is encoded by independent feature extractors, which is a small difference from other Vision Transformers. Naturally, the authors extend the positional encoding to 3D positional encoding (horizontal, vertical, and temporal axes) with relativeness.\n\nOverall, the technical contribution is a bit weak, but this submission contains interesting analyses that are worthwhile to report in the community and also achieve state-of-the-art in downstream tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros\n- Trendy topic with good results\n- Pointing out different levels of semantic granularity\n- Proposing DropToken to improve training efficiency\n- Interesting experiment about the ImageNet finetuning results with the model pre-training video\n- Extensive evaluation\n\nCons\n- Weak technical contribution, and company-level large-scale training requirement\n- Composition of existing developments\n\n<Comments>\n\nThis work could be categorized as an analysis-oriented paper.\n\n* Technical contribution: This work can be viewed as a similar line of research to the contrastive learning-based Vision Transfomer representation learning, e.g., [18]. In this regard, this work also can be viewed as a straightforward extension to multi-modal-based self-supervised data. The consideration of semantic granularity difference and Droptoken is only different from the previous work, but this reviewer feels that technical contribution is not significant. \n\n* Semantic granularity: This reviewer concurs with the authors' argument on the different levels of semantic granularity in multi-modal learning. This reviewer thinks this must have been considered in all multi-modal learning research. At least, good to see the emphasis on it in this work.\n\n* Showing a possibility of the existence of a unified model in a multi-modal regime. Nice.\n\n\n** Question: Considering the out-of-context works on the recent MLP based backbones, what would be the true benefit of Transformer in this work? This reviewer does not see whether Transformer is a truly important component in the multi-modal learning regime like this work, other than large capacity. \nAlso, the submission does not include much about the necessity or motivation of the Transformer in multi-modal learning, but just mentioning no one did yet.\\\nWhat is the motivation of Transformer in the authors' opinion?\n\nIt would be more thorough to consider the motivation of the Transformer in this context.\n\n\n======================== After rebuttal ========================\n\nThe authors have addressed most of the comments. However, the concerns on the technical contribution and the position of this submission are not adequately addressed, which are shared across the majority of the reviewers.\n\nFor this reason, I downgrade the score. \nThis reviewer still thinks that the benefits of accepting this work would be slightly higher than that of rejecting it to the community.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuseed the requirement of large computation, and model bias due to the use of uncurated multi-modal videos, which are reasonble concerns.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ddoOwgYzH3d", "reviewer": "Reviewer_akAf", "summary": "This paper describes a self-supervised pure transformer-based multimodal representation learning including video-audio-text, i.e., VATT. VATT consists of  linear projection of input for each modality, modality-specific or agnostic transformer encoder, and multimodal projection head. VATT uses NCE loss for video-audio feature learning and MIL-NCE loss for text-video features. Also, the authors proposed DropToken as an efficient method for training. \nThey employ HowTo1M and AudioSet datasets for pretraining and evaluate their VATT for diverse finetuning tasks including video action recognition, audio event classification, and zero-shot text-to-video retrieval. With intensive experiments and ablation studies, they provide promising results and constructive analysis. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n#### Originality\n- It is novel to use audio for multimodal pretraining in addition to video and text.\n- Pure transformer architecture-based video-audio-text learning is novel considering NeurIPS submission deadline. \n\n#### Quality & Significance\n- This paper is well-organized to easy to understand.\n- Multimodal pretraining is a challenging but significant topic. \n\n#### Clarity & Experiments\n- Experiments are extensive and the results look promising on diverse downstream tasks.\n- Analysis including feature visualization is helpful.\n\n### Weakness\nAlthough I like this paper, there are some rooms to be improved.\n- Some core previous studies was missed in related work such as VideoBERT [Sun et al. 2019]. Also, The authors need to compare VideoBERT. Also, in related work, it is helpful to enhance the story by including recent studies on self-supervised learning from uncurated data in computer vision domains such as SEER [Goyal et al. 2021] and image-based self-supervised learning for videos [Feichtenhofer et al. 2021].\n- Why did the author use word-level linear projection as text input instead of BPE tokenization, which is de facto in training LM? Please clarify this reason. \n- The author employs multi-level common space projection with hierarchy. Is there any performance drop using pair-wise feature loss with plat-style? If then, how significant?\n- For negative pairs, how about making matching inter video clips? for example, frames from video A and audio from video B. \n- As the authors described in the paper, VATT model sizes are variable according to each modality encoder size and whether the encoder is modality-agnostic or not. At least, table 1, 2 and, 3 need to include model parameter sizes. \n- In table 1, What data are used as the pretraining or scratch-training for other baseline models? It need to be clarified. \n- How is the performance on Kinect-700? Including MMV, many recent work evaluated their model on Kinect-700. \n- For DropToken, how is the effect of DropToken on each modality? Did the authors try to apply DropToken to a specific modality?\n- Why VATT-MBS was compared instead of VATT-Base, unlike other tables? Considering the parameter size, I think that VATT-Base with the same batch-size and epochs is more suitable for comparison. \n- How is the performance when increasing batch-size and epochs similar to MMV or MIL-NCE in table 4? As the authors argued the importance of batch-size and epoch, the results on large batch size and epochs with the same model will help to clarify the argument. \n- Readability can be improved. For example, VATT-Base means VATT-BBS? In Table 4, \"VATT-MBS\" is used instead of \"VATT-Medium\". Consistent term usages are very helpful. Overall, the captions of Tables need to be self-contained with more explanation and term definition. Also, it is better to clarify the use of self-supervised ViT (79.9%) for better understanding. \n\n#### References\n- [Sun et al. 2019] A Joint Model for Video and Language Representation Learning. ICCV 2019.\n- [Goyal et al. 2021] Self-supervised Pretraining of Visual Features in the Wild. arXiv 2021.\n- [Feichtenhofer et al. 2021] A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning. CVPR 2021.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No special societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "V1pelnVD9TD", "reviewer": "Reviewer_GNEo", "summary": "The paper proposes a transformer-based architecture for learning representations from video-audio-text triplet data without manual data annotation. The paper studies modality-specific transformers (one transformer for one modality) and modality-agnostic transformers (a shared transformer for all modalities). The learned representation is evaluated on various downstream tasks on image, video, audio and video-text.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of the paper is limited. The standard VATT model is identical to [1] (Multimodal versatile network, Alayract et al.) except changing the ConvNet backbones into transformers. The novel point is the modality-agnostic shared transformer for all three modalities. The proposed DropToken is a useful technique to reduce the computation overhead for transformers. \n\nThe paper shows a large number of experimental results, especially for downstream tasks. The overall quality is good.\n\nThe paper is clearly written. \n\nThe significance of the results is moderate:\n1. The key method of the paper is a transformer-based architecture for multimodal learning. In detail, it uses identical architecture and loss design (sec 3.3, 3.4) except replacing the backbone with transformers. But when comparing with [1] (Table 4 and supp Table 2), the proposed transformer backbone underperforms or gives similar results as its ConvNet counterpart, also the discussion is limited on this point.\n2. The downstream performance of modality-agnostic transformers is not far from modality-specific transformers, which is encouraging for future works. \n3. Table 5 shows the proposed DropToken is useful for reducing computation while keeping the downstream performance.\n\n\n[1] Self-supervised multimodal versatile network, Alayrac et al.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As stated above, one of the key contributions is a transformer-based multimodal architecture, however it does not outperform the ConvNet counterpart when using the same loss. Discussion on this point would be valuable. \n\nThe proposed modality-agnostic transformer is novel, and the paper shows its performance on downstream tasks. But it’s still not clear to me what can be captured in that unified transformer. I am curious to see more qualitative results like the attention maps per encoder layer for each modality. \n", "review_text": "", "extra_scores": {"ethical_concerns": "No major ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "TeNim1n_Oei", "reviewer": "Reviewer_PWef", "summary": "This paper concerns multimodal self-supervised learning. It adopts the now ubiquitous contrastive learning framework to learn video, audio, text representation from each raw signal in joint embedding spaces. The proposed architecture heavily relies on MMV [1], with a twist on Visual/Audio Transformer encoders and modality-agnostic Transformers. Comprehensive experimental results are conducted on four diverse video(-text) tasks across ten datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is sending some mixed messages. On the one hand, it demonstrates Transformers' decent capability on learning from raw visual/audio signals, in the context of multimodal self-supervised learning. Also, the possibility of unifying encoder across modalities through a modality-agnostic framework. On the other hand, this unique capability is at the cost of larger models, more training resources, and sometimes performance degradation.\n\nSince the proposed methods **heavily** rely on the previous work MMV [1], missing this important baseline in many result tables is a major flaw (despite that MMV is not originally evaluated on these highly relevant datasets) and hinders a fair comparison between CNN-based methods and Transformer-based methods. On the few datasets that have MMV results reported (e.g., Tab. 4, Tab. 2 in Supp.), the proposed methods overall underperform the baseline, sometimes significantly, despite using much larger models (2-4x) and computation resources (over 2x TPU days). This puts a doubt on the effectiveness of the proposed method.\n\nBesides, the reported model variants are inconsistent throughout the paper (e.g., in Tab. 2 and 3, VATT-Base while in Tab. 4 VATT-Medium). In Tab. 1, columns indicating #params, #training data, modality should be added for better comparison.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Appear to be.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IDoZwO-T0Nu", "reviewer": "Reviewer_6L5v", "summary": "This paper provides a framework for self-supervised multimodal representations from unlabeled data with Transformer backbones. The authors study two styles of Transformer encoders: modality-specific or modality-agnostic.  Finally, they achieve good performance on various downstream tasks including action recognition, audio event classification, text-to-video retrieval and image classification.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- This paper is well written and the topic of this paper is valuable.\n\n- The novelty of this paper might be limited. The framework of self-supervised multimodal representation learning is almost as same as MMV [1] except replacing CNN backbones with Transformers.  DropToken technique does not show many advantages especially over training with lower resolution (79.9 acc. with 548.1 GFLOPS v.s. 80.2 acc. with 572.5 GFLOPS in Table 6). \n\n- It is interesting to study the modality-agnostic encoder and the results are encouraging. However, it is not the main part of this paper. It would be nice to take a more detailed analysis and prove that a modality-agnostic encoder can achieve competitive or better performance than a modality-specific one.\n\n- The performance comparisons with previous multimodal representation learning methods (MIL-NCE, MMV) are missing. Authors should following previous papers that evaluating on UCF and HMDB and report the results.\n\n- The detail of DropToken is confusing. Is the DropToken used in both the training and testing phases? Is it used when transferring to downstream tasks or only during pre-training?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The merits and weaknesses of this submission were discussed at length in back-and-forth conversations with the authors. The overall sense is that, although the technical novelty of the submission is limited, the paper provides a valuable large-scale empirical study on sharing a single transformer backbone architecture across different modalities. Several reviewers comment on the importance of sharing such results with the community. The ACs find that this is particularly relevant given that few labs are equipped with the resources to carry out such a study (thus here the ACs dissent from the opposite point made by some reviewers). The submission also introduces DropToken, a simple strategy to significantly reduce the computational cost of training large-capacity transformers. Based on these contributions, the ACs agree to accept this submission but encourage the authors to incorporate the feedback provided by reviewers, especially as it concerns the motivation for using transformers and the novelty claims. 
Decision: Accept (Poster)