TVLT: Textless Vision-Language Transformer
Authors: Zineng Tang
Jaemin Cho
Yixin Nie
Mohit Bansal
Venue: NeurIPS 2022 Accept (2022)
Keywords: textless vision-and-language modeling
audiovisual
TVLT

Abstract: In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR). TVLT is trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text. Our code and checkpoints are available at: https://github.com/zinengtang/TVLT

TVLT: Textless Vision-Language Transformer
Zineng Tang"
Jaemin Cho"
Yixin Nie" Mohit Bansal
UNC Chapel Hill
(terran, jmincho, yixinl, mbansal]@cs unc.edi edu
Abstract
In this work, we present the Textless VisionLLanguage Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and- language representation learning with minimal modelity-specific design, and do not use text-specific modules such tokenization automatic speech recognition (ASR). TVLT trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart various multimodal tasks, such visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster infer- ence speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text.'
Introduction
Humans perceive and learn the external world through signals from multiple modalities. To embody such human learning in machines, substantial research efforts are dedicated to developing vision- and-language (VL) models that can understand the joint semantics between visual and linguistic modalitics and solve tasks such as visual question answering [4]. Although most such VL models use written language rather than spoken language the main verbal communication channel, the default communication modality among humans has been speech, since circa 100,000 BCE [77]. Written language is relatively recent; cuneiform script, the earliest writing system, developed circa 3,200 BCE [65]. Moreover, we have witnessed an increasing usage AI models real- real-world products such virtual assistants and smart speakers [40], where perception-level signals such video and audio are the natural form of input. Intuitively, direct modeling of such signals will potentially yield more compact and efficient representations.
Transformers [80] have recently achieved great success in vision-language representation learning [75; 10; 48; 73; 86; 85] by using text-based modules [15] text-annotated images or videos. However, is non-trivial to learn VL representations using transformers that take only low-level visual and acoustic inputs without the prior existence of written language. The challenge lies in the difference between text and acoustic signals; text discrete and dense information, while acoustic signals continuous and sparse in information [26; 7]. Therefore, moodllitysspecinf architectures have been used to model data from different modalities. It is only recently that researchers started using modality-agnosti transformer architecture to learn representations of different unimodal [17; 19; 8], vision-text [32; 54], or vision-audio-text [2] data. However, to the best of our knowledge, no previous work has explored single homogeneous (modality-annostf representations directly from visual and acoustic input at the perception level (without relying on text), and also makes the textless VL model more compact and efficient than the existing text-based VL models (see Sec. 2 for details).
minimalist transformer that learns
equal contribution
"Our code and checkpoints are available at: http:://gihii...oo/iaiigggggg/ga
con//iinenttag/TVV..
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Language Encoding for Tasks
Efficiency: Inference Time #Parameters
Audio
Text
2690ms
26ms
Previous (wi ASR)
ASR
Pravious (we ASR)
Speech Recognition
2916ms 283M
Multimodal Encoder
Modality Interaction
TVLT (Ours)
TVLT (Curs)
103ms BUM
Fourier
Modality
Transform
Interaction (43mm)
(50ms)
Figure 1: Comparison of previous VL. architectures and proposed textless framework TVLT. The removal of automatic speech recognition (ASR) from the VI. pipeline brings efficiency improvement while maintains competitive performance. For inference time calculation, use 8 video frames and 20s audio (see Sec. 6.2 for detail). As shown in Table I, TVLT achieves competitive performance to text-counterpart on video retrieval and multimodal sentiment analysis tasks.
In this work, we propose Textless Vision-Language Transformer (TVLT) for rision-and-langee representation learning based on video data as the natural source of raw visual and audio input. As depicted in Fig. 2, TVLT accepts low-level video frames and audio spectrograms as input. We employ minimalist design for TVLT where homogeneous transformer blocks are used for both the encoder and decoder. TVLT trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. More importantly, TVLT makes no assumptions about the existence of written language and does not involve explicit modeling of text input, such as automatic speech recognition (ASR) or tokenization, which are crucial submodules in the success of existing VL models in aligning written concepts with visual clues.
Despite the removal of text-based modules and modality-specifie designs, TVLT achieves results comparable to its text-based counterparts in multimodal tasks (with either direct audio input, or text converted to audio input via TTS) such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, while being computationally efficient with 1/3 parameters and a 28x faster inference speed, as illustrated in Fig. 1. This indicates that the removal of text-specific modules such as ASR in vision-and and-language modeling helps reduce computational redundancy in existing pipelined learning paradigms, where text is first extracted through ASR and then further processed by a text-based VL model. Furthermore, we also show that TVLT can capture acoustic information beyond speech and is more effective multimodal emotion classification than its text- based counterpart. We hope that our findings spark further research in the realm of textless VL models that take raw signals input and seek to learn more compact and efficient vision-ande and-language representation.
2 Related Work
Text-based Representation Learning. Large-scale unsupervised pretraining of contextualized language models based on written texts has seen great success in recent years. ELMo [58] proposes to pretrain and finetune large recurrent language model, which improves performance on diverse set of downstream natural language processing tasks. BERT [15] improves the scalability of the pretrain- then-finetune paradigm by using a transformer [80] model with masked language modeling objective. Since then, the pre-training of transformers has been extensively explored for transfer learning in language [46; 82; 38; 16; 72; 60; 13]. these methods, learning focused on eliciting high-level linguistic semantics and structures from unlabeled written texts or natural sequences of words.
Audio-based Representation Learning. Pretraining methods on audio input involve transferring the continuous ID audio signal into dense vectors that can be input to speech or acoustic model. Early work mainly uses recurrent neural networks [12; 11; 69] and convolution networks [66] for audio encoding. To take advantage of the proven expressiveness and genericity of transformers, more recent work proposed using audio spectrograms [19; 20: 7] as image input and then encoding the patches of such images with transformer, following the same methodology in computer vision [17].

---

TVLT: Textless Vision-Language Transformer
GLL
Reconstructed Spectrogram 24
Spectrogram
Spectrogram
Decader
Encoder
Matched?
Encoder E
Shared Weights
Video Fromes
Video Frames
g
Decoder
D
Fa
(a) Vision-Audio Matching
(b) Masked Autoencoding
Figure 2: TVLT is pretrained with objectives: (a) vision-audio matching (Sec. 4.1) and (b) masked autoencoding (Sec. 4.2). The model takes video frames and audio spectrogram as inputs and does not use text input and completely removes text from the pipeline.
The pretraining objectives for transformers range from classification [19] to masked audio modeling [20; 7]. A line of work uses audio transformer with discrete audio units for pretraining [27] and speech tasks such as generative spoken language modeling [37; 31] and speech emotion conversion [35]. These works focus learning the acoustic and linguistic characteristics of language from raw audio or spectrogram.
Vision-and Sssion-and-Languagee Language Representation Learning. Following the success of pretraining of trans- former language models, pretraining of image+text [75; 48; 10; 43; 89; 41J, video+text [73; 52; 91; 42; 76; 86], and video+text+audio [78; 84; 61; 85; 2] multimodal transformers has recently achieved improvements in downstream VL tasks such as visual question answering [4; 28] and text-to-video retrieval [81; 90]. These methods text, such written captions ASR transcripts, as input into the language channel. There is another line of work on models taking video+audio input, where they can utilize naturally synchronized vision+audio pairs from videos. Audio-visual synchronization is often used for self superviedd learning [56; 5; 55; 34; 6; 53; 49], or for downstream tasks such automatic speech recognition [1; 71; 70] and video retrieval [74; 63; 64; 45]. Our work is different from these works, in that we focus on the design of homogeneous and modality-agnosst transformer (Sec. 3) achieve novel, unified, and minimalist textless lisuul-inguuisic representa- tion learning method directly from visual and acoustic signals (without relying on text), via masked autoencoding and contrastive modeling objectives (Sec. 4), which also makes the textless VL model more compact and efficient than the existing text-based VL models.
3 TVLT: Textless Vision-Language Transformer
We introduce TVLT: Textless Vision- Languuge Transformer, a minimal end-to-end vision-and- language transformer model that accepts list of embeddings obtained directly from perception-level video and audio input without text-specific modules, as depicted Fig. and Fig. 2.
3.1 Input Embeddings
The input embeddings of TVLT are the sum of (1) modality embedding, (2) temporal/spatial embedding for video, (3) temporal/frequency embedding for audio, and (4) vision/audio patch embedding. As illustrated by the red and blue boxes in Fig. 2, the modality embeddings trainable vectors added to the input embeddings and used to indicate whether the input is from vision or audio input. In what follows, we explain the details of vision and audio embeddings.
two
Vision Embeddings. We adopt VIT [17]-style vision embedding, where each video frame of 224 x 224 pixels is divided into list of 16 x 16-sized patches. Then, liner projection layer is

---

applied to the normalized pixel values of each patch, resulting in a 768-dimensional patch embedding. For video clip with N frame samples, the input tensor with shape N 224 224 (time height x width x channel) will result in N x 14 x 14 embeddings. The temporal and spatial embeddings are different trainable vectors added to the time, height, and width axis of the N x x 14 embeddings to incorporate the temporal and spatial information for each input patch. We treat image input as single frame video so that our model can handle both image and video tasks without modification of the architecture [9]. Temporal embedding is only added for video inputs; we do not use temporal embedding for images.
Audio Embeddings. To obtain audio embeddings, we first convert the ID waveform of the raw audio signal 128-dimensional log Mel-spectrogram having a dimension of T 128 (time axis x frequency axis). Then, we treat the audio spectrogram as an image, divide the spectrogram images into patches, and apply liner projection layer on each patch to obtain a 768-dimensional patch embedding. This follows the audio embedding methods in recent work [19; 20; 7], where similar modality-agee aggnosii transformer is used to model spectrogram patches. We experiment with two different patch sizes: 16 x 16 (square patches similar to the vision modality) and 2 x 128 (the same area as the first one but covers the entire frequency domain with a shorter time range) and use trainable temporal and frequency embeddings to indicate the temporal and frequency information of patches.' 3
3.2 Multimodal Encoder-Decoder
The main architecture of TVLT is a transformer [80] consisting of a 12-layer encoder (hidden size 768), E, and an 8-layer decoder (hidden size 512), D. We follow He et al. [26] and use a shallow decoder that only serves for masked autoencoding objective (Sec. 4.2) and has much fewer computations than the encoder. After pretraining, we only use the encoder representation for finetuning on downstream tasks.
Pretraining Objectives
By virtue of our minimal and aggalicy- design, TVLT is pretrained with two objectives: (1) vision-audio matching (Sec. 4.1) and (2) masked autoencoding (Sec. 4.2). For each training batch, we compute each objective through separate forward pass and use the weighted sum of them for the final loss, where VAM 1.0 and MA 0.3.
loss A VAM loss VAM + X MAE Loss MAE
(1)
4.1 Vudion-Audio Matching
We use the vision-audit matching (VAM) objective learn the global cross-modal representation, as illustrated in Fig. 2 (a). For each video input, we create a (positive) vision-audio pair (aV+, x²). Then, we construct half of the vision-audio pairs inside a batch mismatched (negative) pairs (.), by replacing video frames x: V+ with randomly sampled video frames x. from the training dataset.
Following previous vision-an dision-and-lee and- and-language transformers [75: 10; 48; 32], a linear layer with sigmoid activation is used as the classification head applied to the encoder output of the first [CLS] token to obtain the matching probability p. Then we compute the binary cross-entropy loss as:
loss VAM -y yog
where is when the input vision-audio pair (x\, AI is matched and 0 otherwise.
4.2 Masked Autoencoding
In addition to the VAM objective to learn cross-modal representation, we also use the masked autoencoding (MAB) objective to improve unimodal representations in the vision-and-language
We melspectrogram method of librosa [50] with arguments: sampling rate-44100, n_ fff==044,, hop 1ength=512, hindow="hann' pad_mode= 'cnnttan', n_me1s=128.
'Wit 16x16 patch, a 20-secoond audio will have spectrogram with shape 640 x 128 (time axis x frequency axis), resulting 40x x 8 320 patches.

---

settings, by masking random patches of visual frames and the audio spectrogram, and reconstruct missing imputs as shown in Fig. (b). Coneretely, we randomly drop portion of visual x and audio embeddings AA then feed the remaining patch embeddings to the encoder E. We create inputs for the decoder D by adding the dropped embeddings as trainable vectors [MASK] to the same location as the original input (gray boxes in Fig. 2 (b)). We also add the corresponding temporal, positional, and frequency embeddings to the decoder input. Note that the temporal, positional, and frequency embeddings of the encoder and decoder separately parameterized. We calculate the mean squared error between the reconstructed and original video frames and spectrograms:
loss MAE
1
Σ &Emasked
|a 22
1
Σ
1
NY
NA
jEmosked
where NY and N are the number of masked patches for vision and audio, respectively. We compute the loss only on masked patches, similar BERT [15].
To save computation, we slice the audio and video parts of the encoder output and feed them separately to the decoder, rather than decoding the video frames and the audio spectrogram jointly. In Sec. 6.6, we show that separate decoding achieves better finetuning performance, as well as better efficiency than joint decoding.
4.3 Masking Strategy
Vision Masking.
Following MAE [26], we randomly mask 75% of the visual patches, and the masking is applied for each video frame independently.
Audio Masking. Following MAE-AST [7], we randomly mask 75% of the spectrogram patches. To better capture speech-related audio representation, we emphasize audio masking on speech audios. We use Audiotok [3], an audio activity detection tool, to determine speech spans based on the detection of events in the energy of the audio signal. Then, we apply the masking only on those audio spans. We use a probability of 15%. We include the details of speech span detection in appendix.
Experimental Setup
To compare the andio-based and text-based language representations for vision-aue aadion-and-ae and---ndd-ddaan language tasks, we pretrain our TVLT and its text- counterpart on video datasets. Then, we finetune the models on set of downstream
language datasets for evaluation.
5.1 Text-based TVLT Counterpart
Our text-based TVLT counterpart has the same architecture as the vanilla TVLT with minor changes to accommodate text-based inputs. Firstly, we use sentence-piece [36] tokenizer and then map each token to trainable vectors to encode the raw text into embeddings, instead of converting the continuous input of frames or spectrograms into patch embeddings
in vanilla TVLT. Secondly, we follow the norm in mask language modeling [15] to use an affine layer as the decoder to recover masked words and set the mask ratio on text to be 15%, instead of using transformer decoder reconstruct 75% of the masked vidco and audio embeddings in vanilla TVLT.
5.2 Pretraining Datasets
HowTo100M, We used HowTo100M, [52], dataset containing 136M video clips of a total of
134,472 hours from 1.22M YouTube videos to pretrain our model. Our vanilla TVLT is pretrained directly using the frame and audio stream of the video clips. Our text-based TVLT is trained using the frame and caption stream of the video. The captions are automatically generated ASR provided in the dataset. We used 0.92M videos for pretraining, some links to the videos were invalid to download.
YTTemporal1800% YTTemporal 80M [86] includes 180M video segments from 6M YouTube videos that spans multiple domains, and topics, including instructional videos from How 100M [52], lifestyle vlogs of everyday events from the VI.GG dataset [29], and YouTube's 's auto-suggested videos for popular topics like 'science" or home improvement'. Each video segment consists of 1) image
5

---

frame extracted from the middle timestep of the segment, and 2) an ASR-based caption of L=32 BPE [18; 67] tokens. For each sample, we randomly sample 15s video clip from the entire video to form setting similar to HowTo100M dataset. Concretely, the original dataset provides 100 label files which are random split of the dataset. We sample 20% of YTTemporal 80M (0.93M videos) so that the resulting subset consists of a similar number of videos to How To100M (0.92M videos), and call it YTT-S. In appendix, we show that pretraining TVLT on YTT-S can improve the downstream task performance of over pretraining on HowTo100M.
5.3 Downstream Tasks
We evaluate models on video-based and image-based -anggaguaa tasks to compare the learned representation based audio and text. For video-based tasks, we experiment with video retrieval [81; 90; 92] and multimodal sentiment analysis [84]. For image-based tasks, we experiment with image retrieval [83] and visual question answering [4; 21]. Although audio comes naturally with video, image-based tasks, such as visual question answering, do not include audio. Thus, we obtain audio queries for visual question answering via the text-to-speech (TTS) synthesis method (Sec. 5.4).
Audio-to-Videe Retrieval. Following AVLnet [63]. we use MSR-VTT [81], Youcook2 [90], and CrossTask [92] for audio-to-video retrieval. We also follow the same data split in AVLnet [63] to finetune our models on their respective training set.
MSR-VTT is an open domain video dataset, consisting of 10,000 video clips from 20 categories such as music, movies or food. We follow AVLnet for the standard split, i.e., 6,783 training clips and 1000 test clips (where 32 videos do not have sound). We report the test split results.
Youcook2 is a video dataset on cooking tutorials that contains 2,000 long videos of 89 cooking recipes. Each recipe has on average 22 videos. It has 9,586 training clips and 3,350 validation clips. We report the validation split results.
CrossTask dataset contains instructional videos for 83 different tasks, divided into 18 primary tasks and 65 related tasks. Primary tasks are manually collected with temporal step human annotations and are the main focus of tasks such as cooking or repairing. Related tasks are automatically collected without any annotations and are tasks related to the primary tasks, such as masking latte (primary) vs. making machiato (related). The goal of related tasks is to assess whether they can improve primary tasks. It has 17,840 training clips and 2,819 validation clips. We report the validation split results. For all three tasks, we extract mp3 audio from videos with sample rate of 44. 1kHz. We also used the extracted audio or its corresponding ASR as retrieval queries for our experiment.
Multimodal Sentiment Emotion Analysis. We use CMU-MOSEI [84] for multimodal sentiment analysis. The dataset is made up of 23,454 movie review clips with more than 65.9 hours of YouTube video by 1000 speakers that cover 250 distinct topics. Each video clip also comes with a ground truth transcription written by the author of the video. Following previous studies, we use the 15,288/4,88 8300 train-test split and report the binary accuracy (A2) for sentiment analysis and weighted accuracy (WA) and Fl score on emotion classification over 6 emotion categories.
Pidii--to-Image Retrieval. We use Places-400k (The Places Audio Caption 400K Corpus) [25; 23; 24] for audio-to-image retrieval. The dataset contains approximately 1,000 hours of 400,000 spoken English captions for natural images drawn from the Places-205 [88] image dataset. The queries are conceptual descriptions of the image. The dataset also provides ASR of these audios. Places-205 is a lcage-scale scene dataset with 205 scene categorics such as forest, bedroom, and coast, which contains 2,500,000 images in total.
Visual Question Answering. We use VQAv1 [4] and VQAv2 [21] for visual question answering.
VQAv1 contains 204.721 images from COCO [44] and 430.725 questions. VQAv2 is newer version of VQAvI, with 265,016 images from COCO and 1,105,904 questions. For experiments with audio questions, we generate speech audio from textual questions using TTS (Sec. 5,4) and report test-dev results for both tasks.
Finetuning on Downstream tasks. For each of the downstream tasks, we add a task-specific head (two-layer MLP) on top of the encoder representation. For retrieval tasks, we use MLP to map

---

Table 1:
Comparison of TVLT and its text-based counterpart on audio-to-video retrieval
and video-based multimodal sentiment analysis benchmarks; HT/00M==Hw/T000MM, Y77:
S=YTTemporal 80M subset.
100M
Input Mod.
Pretrain Datasets
Audio-to- Midioo Retrieval (R@ 1)
Sentiment (A2)
Lateney
Method
V
T
A
MSR-VTT
Youcook2
CrossTask
CMU-MOSEI
(ms)
TVLT
V
2.2
68.1
2916
TVLT
4.3
4.7
2.7
65.7
103
TVLT
V
V
HT100M
17.1
24.9
11.1
76.5
2916
TVLT
HT100M
22.6
31.8
14.9
75.3
103
TVLT
/
YTT-S
19.3
12.2
76.6
2916
TVLT
YTT-S
23.8
32.8
15.3
76.8
103
encoder representation of [CLS] to matching scores E [0, 1]. which correspond match mismatch pairs, and train the model jointly with binary cross-entropy loss. For visual question answering tasks, use an MLP to map the encoder representation of [CLS] the answer probabilities with 3129 answer candidates, and train the model jointly with binary cross-entropy loss in multi-label classification setup. For multimodal sentiment analysis tasks, we use an MLP map the encoder representation of [CLS] token to the entiment scores, and train the model jointly with L.2 regression loss.
5.4 Other Details
Automatic Speech Recognition (ASR). For the text-based model mentioned above, we ob- tain text from audio with different automatic speech recognition (ASR) models. We use the asr-crdan- dectan-rnnnlln wamllmmmiisppeee
ASR model from the Speechbrain package [62]. The model
based on RNN language model and CRDNN encoder-CTTC/Attentio decoder architecture and trained on LibriSpeech [57]. We also experiment with the Google Cloud Speech-to-Tess API which uses Conformer [22] as the backend model.*
Text-to-Speech (TTS). We WaveNet [79] Google Cloud Text-to-Speech API to generate audio input for the questions in VQAv2. Since VQAv2 questions are written in English, we use en-US-neutral speaker. We follow the default pitch and speech configuration. We use the mp3 audio format with a sample rate of 44. 1kHz to match the audio configuration used in the pretraining.
Pretraining. We train TVLT and the text-based TVLT counterpart for 200k steps using Adam optimizer [33] with a learning rate of 1e-5, batch size 4096, and decay rate of 0.001 with cosine schedule [47]. We initialize the weights both models with the masked autoencoder transformer in He et al. [26] that is pretrained on ImageNet []4]. For the pretraining objectives in Eq. (l), we use VAM 1.0 and MAE
0.3. For each video clip, we uniformly sample frames. Pretraining takes 2 weeks with 4 NVIDIA RTX A6000 GPUs (each 49GB memory).
Finetuning on Downstream Tasks. We use learning rate of 1e-5, batch size 256, and a decay rate of 0.001 with cosine schedule for all tasks. For each video clip, uniformly sample 8 frames. We use 2 NVIDIA RTX A6000 GPUs.
6 Results and Analysis
6.1 Comparison to Text-based Counterpart
Table shows that TVLT outperforms the text-based counterpart in audio-to-video retrieval tasks when pretrained either HowTol00M YTT-S. On CMU-MOSEI sentiment analysis, TVLT also outperforms its text variant when pretrained on YTT-S. Table 2, although TVLT slightly underperforms the text-based counterpart audio-to-image retrieval and visual question answering, TVLT can still achieve decently comparable results and remain competitive while being 27x faster during inference due to the removal of ASR from the processing pipeline. More details
'nttpa://clond google. cow.specccc//pecheceeooooo to text 'nttps:/ccloud google. wowgle.com/tenn.eeoooooooooooooe teap/text-eeee ttwp://tttto-ssccc/ooo//////eae htc//
7

---

Table 2: Comparison of TVLT and its text-based counterpart on audio-to-image retrieval and visual
question answering benchmarks.
Input Mod.
Pretrain Datasets
Audio-to-Iman Image Retrieval
Visual QA (Acc.)
Latency +
Method
Places-400k @R@1 / R@
VQAv2
(ms)
TVLT
13.0 / 49.7
47.0
2010
TVLT
12.7 33.3 48.0
46.7
.52
TVLT
V.
HT100M HT
50.4 78.2 87.0
62.1
2010
TVLT
48.7 77.9 86.0
60.8
52
TVLT
YTT-S
54.3 78,9 88.8
63.2
2010
TVLT
YTT-S
49.0 7 86.8
61.0
52
efficiency analysis are given in Sec. 6.2. The results provide evidence of the possibility of learning more compact and efficient Wisionandd-laguaag representation from raw visual and audio signals compared to the prevailing VL learning paradigms with explicit text-based modules in the pipeline.
6.2 Efficiency Comparison
Table Latency of FFT, ASR and VL Models.
To test inference latency, we sample 100 videos in CMU-MOSEL. As the average video length in the CMU-MOSEI dataset is 12 seconds, we measure the latency with two sets of input video lengths: 10 and 20 seconds. For 10s and 20s videos, we also use and 8 video frames, respec- tively. Then calculate the processing time of Fast Fourier Transform (FFT), SpeechBrain (ASR-SpBr) [62], TVLT, text-based TVLT, and AVLNet on the sampled inputs. Speech- Brain is the default ASR module that we used our text-based counterpart pipeline (see Sec. 5.4).
Vidco laput
Latency (ms) 1
Length / Frames
FFT
ASR
VI.
Total
ASR-SpBr
195M
20s /8
10s 4
2890
2110
TVLT
SSM
20s /
10s /
40
60
40
43
103
80
TVLT + text
88M 195M
10s /4
2110
25
2135
88M
195M
206
2890
26
2916
158M
10s/4
208
AVI met text
158M 195M
10s /4
2110
206
2316
As shown in Table 3, we find that ASR dominates the inference time for text-based models. Although ASR helps reduce the input length in transformers (as indicated by the VI. module latency decrease), TVLT is more than 27x and 28x faster than text-based TVLT for inference with video input lengths of 10s and 20s, respectively, with only 1/3 of the parameters. The comparison is also shown in Fig. 1. In the bottom rows, we also show the inference latency of AVLnet and its text variant, where TVLT is 3x faster than AVLnet which contains audio-specific convolution modules.
6.3 Text Query vs. Speech Query for Language-based Video Retrieval
For text-to-video retrieval tasks, text cap- tions are commonly used for queries [81]. In Sec. 6.1, we show the experiment of audio-to-video retrieval tasks following AVLnet [63], where the audio queries are the sounds of the original videos. Since video sounds and text captions have dif- ferent information, the audio-to-video re- trieval results are not directly comparable to the results in other text-to-video retrieval papers. For better comparison, we experiment with video retrieval based on two language queries: 1) text captions and 2) speech audio obtained by TTS (see Sec. 5.4) from text captions. Table shows MSR-VTT video retrieval results of TVLT with text/audio queries and recent text-to-video retrieval models pre- trained with a similar scale of data. 6 Although TVLT with audio query slightly underperforms its text query counterpart due to TTS errors, it still outperforms other rext-to-video retrieval models (HERO [42] and DeCEMBERT (76]), showing promising possibilities of speech-based video retrieval.
Table 4: Text vs. Speech Query for Video Retrieval.
Mathed
Dirasets
Pretnin
Query
Video Retrieval (Re) 1)
MSR-VTT
TYLT TVLT
HT 100M HTI0IM
Captoon
22,0
Speech Audio (TTS)
20.1
HERO [43]
HT 1ODM
Capion
16.8
DcCEMBERT (26]
ITTLOOM TVQA
Caption
17.5
ChipERT [19]
COCO. VG
Caption
22.0
AVEnet [63]
HITIIMM
Caption
22,5
"We exclude the models pretrained on large-scale image captions such as Conceptual Captions [68] that has written annotation, or visual encoder pretrained on large-scale dataset beyond the scale of ImageNet [14], such CLIP [59]. as they not directly comparable our models.
8

---

Table 5: TVLT on CMU-MOSEI emotion analysis test set; WA=weighted accuracy, F/=weighted fl.
Mothod
Input Mod.
Happy
Sad
Angry
Fear
Disgust
Surprise
TVLT
V
64.7
63.9
70.2
66.0
68.9
71.8
66.2
84.4
70.7
82.9
58.4
86.2
TVLT
68.1
72.2
68.1
68.8
79.6
6.4 Emotion Analysis
Since TVLT takes raw visual and audio input instead of relying soiely on text as in text-based TVLT, we further investigate what type of information TVLT can learn beyond speech on CMU-MOSEI emotion classification task. As shown in Table 5, TVLT outperforms the text-based counterpart in most emotion categories, except for 'Disgust'. We conjecture that TVLT is capable of capturing speech-related acoustic information, such as tone and loudness, which is helpful in recognizing these emotions, while this ability is absent from text- ASR -SRedependent models.
Table 6: Finetuning performance on audio-to-vides retrieval and multimodal sentiment analysis benchmarks. For fair comparison, we gray out the models that use ground-truth text transcription as additional input for CMU-MOSEL.
Input Mod.
Pretrain Datasets
Audio-to- Video Retrieval (R@) 1)
Sentiment (A2) t
Method
V
T
MSR-VTT
Youcook2
CrossTask
CMU-MOSEI
Multilogue-Nert [69]
75.2
AVI.net [63]
HT100M
20.1
30.7
13.8
TVLT (Ours)
HTT00M
22.6
31.8
14.9
75.3
TVLT (Ours)
YTT-S
23.8
32.8
15.3
76.8
Table 7:
Finetuning performance on audio-to-image retrieval and visual question answering (Visual QA). For Visual QA, create spoken questions from text via TTS (Sec. 5.4). 1CSC (Conceptual Spoken Caption) is 3.3M image-speech pairs, where speech is obtained via TTS API from Conceptual Captions. The CSC dataset is not publicly available.
Input Mod.
Pretrain
Image Retrieval
Visual QA (Acc.) f
Method
Datasels
T
A
Places-400k (R@1 R@S RR100)
VQAv1 / VQAv2
TextMod [87]
V
56.7/-
SpeechMod [87]
47.0/-
AVLnet [63]
HT100M
44.8 /6.. 86.4
MILAN [64]
CSC
53.4 / 79.1 86.3
TVLT (Ours)
HTT00M
48.7 / 77.9 86.0
58.6/608
TVLT (Ours)
YTT-S
49.0 / 7 86.8
58.9 61.0
6.5
Comparison to State-of-the-art Textless Models
We compare our TVLT with recent models that also take raw visual and audio signals as input but involve audio-specific designs in their networks. As shown in Table 6, TVLL outperforms AVLnet [63] on three audio-to-video retrieval (MSR-VTT, Youcook2, CrossTask) tasks and out- perform Multilogue [69] multimodal sentiment analysis (CMU-MOSEI) task with simple modality-agnostic design. Similarly, Table shows that TVLT achieves competitive results with AVLnet [63] and MILAN [64] on audio-to-image retrieval (Places-400k). Note that MILAN is pretrained on Conceptual Spoken Caption [30] which contains 3.3M well-aligned image-speech pairs taken from Conceptual Captions [68] with TTS generated speech, whereas our TVLT is able to elicit effective representation from video inputs where vision-and and-language clues are only weakly aligned. TVLT also outperforms both variants of the VQA models (TextMod, SpeechMod) in Zhang et al. [87] on VQAv1.
6.6 Ablation Studies
In the following, we show the results of the ablation study on TVLT training details: the audio masking strategy, the encoder/decoder architectures, and the pretraining objectives.
The dataset is also publicly available.

---

Audio Masking Strategy. In Table 8, we show the result of finetuning performance with different au-
Table 8: Audio masking confi configurations.
dio masking configurations, described in Sec. 4.3. For patch sizes, masking audio patches on de- tected speech spans improves performance across the board. However, we did not observe strict supe- riority between the two patch sizes; 2 x 128 achieves higher scores on MSR-VTT, while 16 x 16 achieves
Masking on speech
MSR-VTT VTT
VQAv2
Patch Size
(R@1)
(Acc.)
16 16
21.7
57.8
16 x 16
22.3
58.6
128
21.0
58.8
128
21.2
59.2
higher scores VQAv2. For our default pretrain-
ing configuration, we use the 16 x 16 patch size and use speech span detection, since the 16 x 16 sized patch is also used in visual embedding (thus modality- agnosticc and speech span detection improves performance with minimal additional computation (see appendix).
Encoder Architecture. As described in Section 3.2, we use the joint encoder in TVLT. We compare this to modality- specific encoders for vision and audio. Table 9 below com- pares the separate encoders with the joint encoder for two tasks: VQAv2 and MSR-VTI To tackle VQAv2 with separate encoders, we learned a two-layer self-attention fusion layer over the concatenation of hidden states of the vision and audio encoder. Our joint encoder architecture achieves better accuracy on both tasks than a separate encoder architecture. The results show that although vision and audio spectrogram two different modalities, the single joint encoder could learn useful cross-modal representation for VL tasks without needing modality-specific encoders.
Table 9: Encoder variants.
MSR-VTT
VQAv2
Encoder
(R@1)
(Acc.)
Separate
9.6
53.1
Joint
10.2
54.6
Decoder Architecture. As described in Sec. 4.2, we use separate decoders (with shared weights) for the vision and audio MAE pretraining objectives. We compare this separate decoding with joint decoder, where we feed the concatenated encoder outputs to the decoder and jointly reconstruct the video frames and spectrogram. Table 10 shows that pretraining with separate decoder outperforms joint decoder on finetuning performance, while being more efficient well.
Table 10: Decoder variants.
MSR-VTT
VQAv2
Decoder
(R@) 1)
(Acc.)
Separate Joint
22.3 22.0
58.6
58.1
Pretraining Objectives. We measure the impact of each pre- training objective described in Sec. 4. Table 11 shows that each the pretraining objectives (MAE and VAM) improves finetuning performance over random weight initialization. The combination of VAM and MAE further improves the finetun- ing performance, and we use this configuration default for TVLT pretraining.
Table 11: Pretraining objectives.
MSR-VTI -TT
VQAv2
Objectives
(R@1)
(Acc.)
Random init
4.3
46.7
VAM
21.0
56.2
MAE
18.6
54.1
VAM MAE
22.3
58.6
Conclusion
In this work, we present TVLT, a simple end-to-end vision-and and-language transformer that can accept low-level visual and audio signals for saiion-addlanguuaa representation learning. Our TVLT achieves competitive performance with other state-of-the-an visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis. We also show that by eliminating the need for expensive ASR in the model pipeline, TVLT can be 28x faster than its text-based counterpart while achieving comparable performance. We comprehensively analyze the efficiency of our model and show ablation studies over different training variants. We hope that our research will inspire further exploration of simple and efficient vision-and-languaae frameworks with low-level signals.
audio-based sudion-and-languss models
Acknowledgments
We thank the reviewers for their helpful comments. This work was supported by ARO Award W91INF21 10220, DARPA KAIROS Grant FA8750-19-2-1004 ONR Grant N0001418 12711, and NSF-AI Engage Institute DRI-211263. The views, opinions, and/or findings contained in this article those of the authors and not of the funding agency.
10

---

References
[1] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2018. Deep Audio-visual Speech Recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-13.
[2] Hassan Akhari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34.
[3] Schili Amine. 2021. auditok: an audio/acoustic activity detection and audio segmentation tool.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV.
[5] Relja Arandjelovic and Andrew Zisserman. 2017. Look, listen and learn. In ICCV.
[6] Yuki M Asano, Mandela Patrick. Christian Rupprecht, and Andrea Vedaldi. 2020. Labelling unlabelled videos from scratch with multi-modal sele-supervision. In NewrlPS.
[7] Alan Baade, Puyuan Peng, and David Harwath. 2022. Mac-ast: Masked autoencoding audio spectrogram transforerr. arXiv preprint Xrxi:22003 16691.
[8] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli, 2022. data2vec: A general framework for self-supervised learning in speech, vision and language. CoRR, abs/2202.03555.
[9] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV. pages 1728-1738.
[10] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Learning universal image-text representations. In ECCV.
[11] Yu-An Chung and James Glass, 2018. Specch2vee: A souuence-to-sqquene framework for learning word embeddings from speech. arXiv preprint 17XXiv:803.089)
[12] Yu-An Chung, Chao-Chune Wu, Chia-Hao Shen, Hung-Yf Yi Lee, and Lin-Shan Lee. 2016. Audio vee:2vec: Unsupervised learning of audio segment representations using sequence-to-sequenee autoencoder. arXiv preprint /603::600000000
[13] Kevin Clark, Minh-Thang Luong, Quoe V Le. and Christopher D Manning. 2020. Electra: Pre-training text encoders discriminators rather than generators. In ICLR.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR.
[15] Jacob Devlin, Ming-Wo Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: bidirectional transformers for language understanding. In NAACL.
Pre-training of deep
[16] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wune Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NewrIPS.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021. An image is worth 16x words: Transformers for image recognition at scale. In ICLR.
[18] Philip Gage. 1994. new algorithm for data compression. C Users Journal, 12(2)223-8.
[19] Yuan Gong, Yu-An Chung, and James Glass. 2021. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778.
[20] Yuan Gong, Cheng-I Jeff Lai, Yu-An Chung, and James Glass. 2021. Ssast: Self-supervised audio spectrogram transformer. arXiv preprint 10.09421
[21] Yash Goyal. Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6904-6913.
[22] Anmol Gulati, James Qin, Chung-Chor Chiu, Niki Parmar, Yu Zhang. Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang. Yonghui Wu, et al. 2020. Conformer: Convolution augmented transformer for speech recognition. In Interspeech, pages

---

[23] David Harwath and James R Glass. 2017. Learning word-like units from joint audio-visual analysis. arXiv preprini arXiv:1701.07455
[24] David Harwath, Adria Recasens, Didac Surís, Galen Chuang, Antonio Torralba, and James Glass, 201 8. Jointly discovering visual objects and spoken words from raw sensory input. In Proceedings of the European conference on computer vision (ECCV), pages 649-665.
[25] David Harwath, Antonio Torralba, and James Glass. 2016. Unsupervised learning of spoken language with visual context. Advances in Neural Information Processing Systems, 29.
[26] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li. Piotr Dollár, and Ross Girshick. 2021. Masked autoencoders are scalable vision learners. arXiv preprint arXiv22111.03377.
[27] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Ab- deirahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3466
[28] Drew A. Hudson and Christopher D. Manning. 2019. GQA: new dataset for real-world visual reasoning and compositional question answering. In CVPR.
[29] Oana Ignat, I Laura Burdick, Jia Deng, and Rada Mihalcea. 2019, Identifying visible actions in lifestyle vlogs. arXiv preprint wrXivv1906.042:0
[30] Gabriel Ilharco, Yuan Zhang, and Jason Baldridge. 2019. Large-scale representation learning from visually grounded untranscribed speech. CoRR.
[31] Eugene Kharitonov, Ann Loc, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2022. Text-free prosody-aware generative spoken language modeling. In ACL, pages 8666-8681, Dublin, Ireland. Associa- tion for Computational Linguistics.
[32] Wonjae Kim, Bokyung Son, and Iodoo Kim. 2021. VILT: Vision-and-L Language Transformer Without Convolution or Region Supervision. In ICML.
[33] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR.
[34] Bruno Korbar, Du Tran, and Lorenzo Torresani. 2018. Cooperative learning of audio and video models from self-supprvised synchronization. In NeurlPS.
[35] Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdel rahman Mohamed, Emmanuel Dupoux, and Yossi Adi. 2021. Textless speech emotion conversion using decomposed and discrete representations. ArXiv, hts/2111.07400
[36] Taku Kudo and John Richardson. 2018. Sentencepicce: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, pages 66-71.
[37] Kushal Lakhotia, Eugene Kharitonov. Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte. Tu- Anh Nguyen, Jade Copet, Alexei Bacyski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On generative spoken language modeling from raw audio, Transactions of the Association for Computational Linguistics, 9:336-13554
[38] Zhenzhong Lan, Mingda Chen, Schastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite hert for self-supervised learning of language representations. In ICLRR
[39] Jie Lei, Linjic Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. 2021. Less more: Clipbert for vidco-and video-and-language learningvia sparse sampling. In CVPR.
[40] Gondy Leroy and David Kauchak. 2019. A comparison of text versus audio for information comprehension with future uses for smart speakers. JAMIA Open, 2(2)224-220.
[41] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, pages
[42] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. Hero: Hierarchical encoder for vidco+ language omni-ropresentaaionn pre-training. In EMNLP.
[43] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2020. Unimo: Towards unified-mood understanding and generation via cross modal contrastive learning. arXiv preprint arX4v:2012.15409
12

---

[44] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár. and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer.
[45] Yan-Bo Lin, Jie Lei, Mohit Bansal, and Gedas Bertasius. 2022. Eclipse: Efficient long-range video retrieval using sight and sound. In Proceedings of the Eumpean Conference on Computer Vision (ECCV).
[46] Yinhan Liu, Myle Out, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019, Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1901 1669..
[47] Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
[48] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision and language tasks. In NeurIPS.
[49] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. 2021. Active contrastive learning of audio-visual vidco representations. In ICLR.
[50] Brian McFee, Alexandros Metsai, Matt Mc Vicir,r, Stefan Balke, Carl Thomé, Colin Raffel, Frank Zalkow, Ayoub Malek, Dana, Kyungyun Lee, Oriol Nieto, Dan Ellis, Jack Mason, Eric Battenberg, Scott Seyfarth, Ryuichi Yamamoto, vitorradrreviccmoroooo Keunwoo Choi, Josh Moore, Rachel Bittner, Shunsuke Hidaka, Ziyao Wei, nullmightybofo, Adam Weiss, Darfo Hereñú, Fabian-Robert Robert Stöter, Pius Friesch, Matt Volirath, Taewoon Kim, and Thassilo. 2022. librosa/librosa: 0.9.1.
[51] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. In CVPR.
[52] Antoine Miech, Dimitri Zhukov, Jean-Baptian Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howtol00ms Learning text-viden embedding by watching hundred million narrated video clips. In ICCV, pages 2630-2640.
[53] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. 2021. Robust audio-visual instance discrimination. In CVPR.
[54] Yixin Nie, Linjie Li, Zhe Gan, Shuohang Wang, Chenguang Zhu, Michael Zeng, Zicheng Liu, Mohit Bansal, and Lijuan Wang. 2021. MLP architectures for vision-and-laan and-languae modeling: An empirical study. CoRR, abs/21122 12.04453..
[55] Andrew Owens and Alexci A. Efros. 2018. Audio-visual seene analysis with self self- supervised multisensory features. In ECCV.
[56] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. 2016. Ambient sound provides supervision for visual learning. In ECCV.
[57] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based public domain audio books. In ICASSP, pages 5206 5210. IEEE.
[58] Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. NAACL.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0000
[60] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
[61] Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis Chill-Phl Morency, Mohammed Ehsan Hoque, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philings Morency, and Ehsan Hoque. 2020. Integrating Multimodal Information Large Pretrained Transformers. In ACL, pages 2359-2369.
[62] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chich Chou. Sung-Lin Yeh, Szu-Wet Fu, Chien-Fena Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. 2021. SpeechBrain: A general general-purpose speech toolkit.
ArXiv:2106.04624
13

---

[63] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, al. 2020. Avinet: Learning audio-visual language representations from instructional videos. arXiv preprini urXiv:2000.09199
[64] Ramon Sanabria, Austin Waters, and Jason Baldridge. 2021. Talk, Don't write: A study of direct speech- based image retrieval. INTERSPEECH.
[65] Denise Schmand--esseratt 2014. The evolution of writing. International Encyclopedia of Social and Behavioral Sciences, pages 1-15.
[66] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint urXiv:1904.05800 090600.....
[67]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACI. pages 1715-1725, Berlin, Germany, Association for Computational Linguistics.
[68] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: cleaned, hypernymed, image alt-text dataset for automatic image captioning. ACL.
[69] Aman Shenoy and Ashish Sardana. 2020. Multilogue-net: A context-aware RNN for multi-modal emotion detection and sentiment analysis in conversation. In ACI. Workshop, pages 19-28, Seattle, USA, Association for Computational Linguistics.
[70] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. 2022. Learning audio-visual speech representation by masked multimodal cluster prediction. In ICLR.
[71] Brendan Shillingford, Yannis Assacl, Matthew W. Hoffman, Thomas Paine, Cian Hughes, Utsav Prabhu, Hank Liao, Hasim Sak, Kanishka Rao, Lorrayne Bennett, Marie Mulville, Misha Denil, Ben Coppin, Ben Laurie. Andrew Senior, Nando De Freitas, and Nando De Freitas. 2019. Large-scale visual speech recognition. In INTERSPEECHI
[72] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic- Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In ICML.
[73] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019, Videobert: A joint model for video and language representation learning. In ICCV.
[74] Didac Surís, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró i Nieto. 2018, Cross-modal embeddings for video and audio retrieval. In ECCV Workshop.
[75] Hao Tan and Mohit Bansal, 2019. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP.
[76] Zineng Tang, Jie Lei, and Mohit Bansal. 2021. Decembert: dense captions and entropy minimization. NAACL-HLT, pages 2415 22266.
Leaminng from noisy instructional videos via
[77] Ian Tattersall, A Sophie, Frederick L Coolidge, and Thomas Wynn. 2009. Cognitive Archaeology and Human Evolution. Cambridge UP.
[78] Yao- Mho-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis Louis- Philippe Morency, and Ruslan Salakhutdinov. 2019, Multimodal Transformer for Unaligned Multimodal Language Sequences. In ACL, pages 6558-6569.
[79] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. In Arxiv.
[80] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
[81] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. 2016. Mtr-vit: A large video description dataset for bridging video and language. In CVPR.
[82] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xinet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
[83] Peter Young. Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
14

---

[84] Amir Zadeh and Paul Pu. 2018, Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL.
[85] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound. In CVPR.
[86] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. Advances Neural Information Processing Systems, 34,
[87] Ted Zhang, Dengxin Dai, Timne Tuytelaars, Marie-Francine Moens, and Luc Van Gool. 2017. Speech-based visual question answering. arXiv preprint 1eoXiv:1005.000 704.00464.
[88] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27.
[89] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In AAAI.
[90] Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018. Towards automatic learning of procedures from web instructional videos. In AAAI.
[91] Linchao Zhu and Yi Yang. 2020. Acthert:
Learning global-local video-text representations. In CVPR.
[92] Dimitri Zhukov, Jean- Bann-Bapisste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. In CVPR, pages 3537-3545.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See supplementary material
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
supplementary material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen-
tal material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 5
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 5.4
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See supplementary material
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
15

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "yJqfYhsVV-z", "reviewer": "Reviewer_yMpw", "summary": "The paper proposes a transformer that takes both speech and images as input to learn a joint vision and language representation. The transformer is trained with two tasks, one is masked autoencoding and the other is whether an image matches an audio clip. The models are then fine tuned on a particular task, for example, audio-to-video retrieval, multimodal sentiment classification, and visual question answering with speech as input. Experiments show that the approach performs well compared to the state of the art.", "questions": "> Section 3.2\n\nIt is a bit unclear what architecture is used for the decoder. It is also unclear how a decoder could be used to generate both images and spectrogram patches. Images have 3 channels, while a spectrogram only has 1. How is this achieved exactly?\n\n> ... we construct half of the visio-audio pairs inside a batch as mismatched (negative) pairs (x^{V+}, x^A), by replacing video frames x^{V+} ...\n\nShould it be (x^{V-}, x^A)?\n\n> Section 5.3\n\nIt's unclear in general how the model is used for each downstream task. Are there any output layers added for each task? Are both the images and spectrograms present during testing? The model in principle should have the ability to take only a single modality instead of both.", "limitations": "The societal impact is inherited from the vision-language tasks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is easy to follow. The proposed architecture is simple, yet performs well. The ablation study and comparison of various input modalities are convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sheNnIUXy59", "reviewer": "Reviewer_WYuA", "summary": "This paper proposes a vision-language transformer representation learning model, denoted as TVLT. The proposed model is based on masked auto-encoders comprised of two encoders (one for vision and one for audio) and a single shared decoder. \nThe model was trained to predict the masked segments together with newly proposed Vision-Audio Matching (VAM) contrastive loss. The authors provide results for several tasks namely: Audio-to-Video Retrieval, Multimodal Emotion Analysis, Audio-to-Image Retrieval, and Speech-based Visual Question Answering. ", "questions": "1) The main issue with this submission is its novelty. By nature audio is pretty different than image, however, the authors used the same encoder arch for both domains where they consider the spectrogram as another image. Did the authors explore other encoders as well? \n2) Considering the synthetic speech data, I'm a bit surprised that synthesized speech from the text was more beneficial than just text considering the way this speech was generated. In other words, when speech was synthesized from the text it will contain the content and artificial prosodic feature, and I'm not sure why should it be better than just text? Can the authors provide more details about this process?\n3) There are some missing references the authors should pay attention to, especially when considering the textless approach: \n* Shi, Bowen, et al. \"Learning audio-visual speech representation by masked multimodal cluster prediction.\" arXiv preprint arXiv:2201.02184 (2022).\n* Owens, Andrew, and Alexei A. Efros. \"Audio-visual scene analysis with self-supervised multisensory features.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n* Surís, Didac, et al. \"Cross-modal embeddings for video and audio retrieval.\" Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018.\n* Lakhotia, Kushal, et al. \"On generative spoken language modeling from raw audio.\" Transactions of the Association for Computational Linguistics 9 (2021): 1336-1354.\n* Kreuk, Felix, et al. \"Textless speech emotion conversion using decomposed and discrete representations.\" arXiv preprint arXiv:2111.07402 (2021).\n* Kharitonov, Eugene, et al. \"Text-free prosody-aware generative spoken language modeling.\" arXiv preprint arXiv:2109.03264 (2021).", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1) The authors provide results for several tasks, demonstrating the effectiveness of the proposed approach. \n2) The authors provide a detailed analysis of their method together with an ablation study. \n\nWeaknesses: \n1) Novelty is limited. The proposed method is very similar to the masked auto-encoder which was recently proposed for image, where the authors additionally include another audio encoder getting the audio spectrogram \"image\" as another input. \n2) Missing references to previous work.\n3) Experimental description can be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ois6nQBN5y", "reviewer": "Reviewer_JbWb", "summary": "This paper presents an audio-visual multi-modal representation learning model called TVLT, and it shows the effectiveness of the learned representations on the downstream tasks that require the understanding of visual and linguistic information. Unlike the previous vision and language representation learning methods that take text input to help elicit the linguistic information, it takes raw audio input instead of the text input but still achieves comparable results with much more improved inference efficiency. In experiments, TVLT shows comparable performance to the text-based counterpart model on audio-to-video retrieval and video-based multimodal sentiment analysis. Although TLVT is slightly worse than the text-based counterpart model, it is still able to achieve comparable results while being 50x faster during inference. In addition, this paper conducts extensive experiments such as a comparison to the SOTA textless models and an ablation study on masking strategy and various model architectures.", "questions": "* Is it possible to use each of the representations for uni-modal subtasks (e.g. Can audio representations be used for ASR training)?\n* What if the text-based model uses all the modalities; text, image, and audio? This is because, as it is mentioned in the paper, the audio modality contains more information that is absent in the text.\n\n(I totally understand that it is almost impossible to conduct every experiment, but it is written somewhat as part of curiosity. So don't take these questions too negatively :D)", "limitations": "This paper adequately addressed the limitations and did considerations about the negative societal impact.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper is well-written and the flow of the writing is natural and neat so it was easy to read.\n- The proposed method is original and it presents an interesting research direction for future works.\n- I believe this work would give a significant impact on this field and its simple architecture is likely to be used by other people.\n- Especially, the minimalist design of TVLT allows to scale up the model so it is expected to give another possibility of large-scale multi-modal model as GPT-3 did in NLP.\n\n### Suggestions\n\n- As far as I know, the tables should be centered in the papers submitted to NeurIPS so I think it should be fixed.\n- There are several suggestions of the experiments that can be conducted in the future work. I listed them in the questions section below.\n- In line 128, I think it might be a typo, $(x^{V+} → x^{V-})$\n- It can give more insights if there are samples of the masked data instances.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept