Title: VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Venue: NeurIPS 2023
Accepted: yes
Keywords: Cross-Modality Foundation Model

Cross-Modality Pretraining Dataset
Abstract: Vision and text have been fully explored  in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including  vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks.

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

Abstract

Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including V ision, A udio, and S ubtitle, and T ext by exploring an automatically generated large-scale omni-modality video caption dataset called V AST-27M. Specifically, we first collect 27 million opendomain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST .

1 Introduction

A vast number of videos are recorded and uploaded to social media platforms every day, making the understanding of video content become a critical area of research in artificial intelligence. Simultaneously, the text serves as the most direct and efficient means of information propagation in our daily lives. Consequently, video-text cross-modality learning is crucial for AI systems to interact with humans, assisting them in understanding video content (video captioning), searching for desired videos (text-to-video retrieval), or answering questions about videos (video QA). With the advent of unsupervised pre-training techniques [1; 2] and large-scale video-text corpora [3; 4], video-text pre-training models have made rapid advancements and achieved relatively impressive performances on the aforementioned tasks. However, we contend that current pre-training models are far from perfect, as most of them are restricted to establishing connections solely between the text and visual content of videos, without incorporating other modality tracks such as audio and subtitles. Specifically, environmental audio can provide additional contexts overlapping with the visual information to reduce ambiguity and increase the prediction confidence of models, or

V-T  Pretraining

Model: Frozen, GIT, CoCa,...

Corpus: WebVid10M,CC14M

A-T Pretraining

Model: WavCaps

Corpus: WavCaps

Vision

VC

Audio

AC

VS-T Pretraining

Model: UniVL,  CoMVT, ...

Corpus: HowTo100M, HD_VILA_100M...

Vision

Subtitle

Subtitle

VA-T Pretraining

Model: AVLNet,

MERLOT-Reserve

,

VALOR...

Corpus: HowTo100M, VALOR-1M...

Vision

Subtitle/AVC

Audio

Omni-Modality VAS-T Pretraining

Model: VAST (Ours)

Corpus: VAST-27M (Ours)

Vision

Subtitle

Audio

Omni-Modality Caption

(OMC)

Omni-Modality Video

(OMV)

complement the visual information to supply more multi-modal cues. Human speech or transcriptions (subtitles) also contain valuable information about conversations, news, or instructional procedures, which can be jointly modeled to enhance the comprehensive understanding of videos. Given that both audio and subtitles are informative for video comprehension, it is necessary to integrate them into a more advanced omni-modality foundational model, as depicted in Figure 1.

The primary challenge lies in the absence of an appropriate training corpus. To train such a foundational model, we require an omni-modality video caption corpus in which captions exhibit correspondence with vision, audio, and subtitles simultaneously. Unfortunately, as shown in Table 2, current video-text corpora either utilize raw subtitles as captions [4; 5; 6], contain only vision captions (alt-texts) [3], or possess audiovisual captions but limited in scale [7]. On the other hand, manual annotation of an omni-modality corpus is not feasible due to the exorbitant cost associated with the high demand for descriptions.

To address this issue, we propose a two-stage automatic pipeline to generate omni-modality captions for video clips from a large-scale open-domain video corpus (HD_VILA_100M [5]). Specifically, as depicted in Figure 2, in the first stage, we train separate vision and audio captioners on publicly available vision and audio caption corpora, which are then utilized to generate high-quality singlemodality objective descriptions. In the second stage, we employ Vicuna-13b [8], an off-the-shelf Large Language Model (LLM), as a zero-shot omni-modality captioner. We feed the LLM with the generated single-modality captions, subtitles, and instructional prompts to encourage LLM to integrate different modality information and summarize them into a single long sentence, forming an omni-modality caption. Through this process, we create VAST-27M, a dataset that consists of 27 million video clips, each paired with 11 captions (including 5 vision, 5 audio, and 1 omni-modality caption). Given its comprehensive caption types, VAST-27M can contribute to various research communities, including vision-text, audio-text, and omni-modality pretraining. Extensive experiments have demonstrated that VAST-27M surpasses current video-text, audio-text, and audiovisual-text corpora by a significant margin.

Based on VAST-27M, we train a V isionA udioS ubtitleT ext omni-modality foundational model named VAST. As shown in Figure 2, VAST consists of three single modality encoders while text encoder can fulfill cross-modality fusion through cross-attention layers. V AST is trained with three objectives including OM-VCC, OM-VCM, and OM-VCG (detailed in section 4.2) to enhance omnimodality understanding and generation capabilities. As shown in Table 1, compared to existing cross-modality pre-training models, VAST is trained on omni-modality video caption corpus and is capable of perceiving and processing all four modalities and supports a wide range of downstream tasks, with various modalities (vision-text, audio-text, and multi-modal video-text), and multiple types (retrieval, captioning, and QA).

Our contributions are summarized as follows: ( i ) We introduce VAST-27M, the first large-scale omni-modality video caption dataset automatically generated by trained single modality captioners and Large Language Model; ( ii ) We train the first vision-audio-subtitle-text foundation model VAST which fulfills omni-modality perception and understanding; ( iii ) Extensive experiments verify the effectiveness of V AST and it outperforms state-of-the-art in a diverse range of cross-modality benchmarks.

2 Related Work

2.1 Cross-Modality Pretraining Corpus

Video-Text Pretraining Corpus. In the early stages, most video-text models were trained on the large-scale HowTo100M dataset [4]. This dataset consists of 136 million video clips from 1.22 million instructional YouTube videos, and employs automatically generated subtitles from ASR transcription tools as corresponding clip captions. Later, following similar schemes for collecting narrative videos-subtitles corpora, Zellers et al. proposed the YT-Temporal-180M corpus [6], which contains 180 million clips from 6 million YouTube videos. To overcome the constraint of the instructional domain, Xue et al. assembled the open-domain, high-resolution HD_VILA_100M corpus [5], consisting of 100 million clips from 3.3 million YouTube videos. However, while labeling subtitles is more cost-effective than annotating captions, subtitles often lack direct relevance to visual content, resulting in weak video-text correlations in model learning. Inspired by the collection schemes of the Conceptual Captions datasets (CC) [9; 10], Bain et al. compiled the WebVid2.5M and WebVid10M datasets [3], utilizing alt-texts as video captions, surpassing the aforementioned datasets that use subtitles as captions. However, alt-texts, although more related to video content than subtitles, differ in style from standard captions and still contain noise. Additionally, alt-texts typically describe visual content only, without incorporating other modalities. Taking a step further beyond vision, Chen et al. created the V ALOR-1M dataset [7], which contains 1 million video clips from AudioSet [11] paired with annotated audiovisual captions. However, subtitle contents are not reflected in these captions, and the dataset's scale is limited and challenging to expand due to the expensive cost of manual annotation. In comparison to the aforementioned training corpora, our VAST-27M is the first high-quality, large-scale omni-modality video caption dataset whose captions encompass all modalities of video including vision, audio, and subtitles. In addition, its expansion is more cost-effective due to the automated generation pipeline.

Audio-Text Pretraining Corpus. In contrast to vision-text pretraining, audio-text pretraining has progressed relatively slowly due to limitations in training corpora, both in terms of quantity and scale. Human-labeled datasets such as Clotho [12], AudioCaps [13], MCAS [14], and AudioCaption [15] all contain fewer than 50,000 audio clips, which fall far short of the requirements for large-scale pretraining. Wu et al. compiled LAION-Audio-630K [16] by crawling audios and corresponding alt-texts from multiple sources, with a majority of audios sourced from Freesound [17]. Mei et al. introduced WavCaps [18], which comprises 403,000 audios and descriptions collected from various sources. They utilized ChatGPT to filter and rewrite the raw descriptions into caption-like sentences. However, the quality of the captions is heavily influenced by the quality of the raw descriptions. In contrast, we generates audio captions through a trained high-quality caption model, and V AST-27M is nearly two orders of magnitude larger than LAION-Audio-630K or WavCaps.

2.2 Multi-Modality Learning

There have been early methods exploring multi-modality tracks for video-text understanding. For instance, MMT [21] leverages seven experts to enhance text-to-video retrieval, while SMPFF [22] introduces audio for video captioning. In the context of video-text pretraining, there are works that jointly model subtitles or audio together with vision and text. Notable examples of incorporating subtitles include UniVL [23], CoMVT [24], and VALUE [20]. However, the subtitle-text correlations established in these models are implicit and weak, achieved through masked prediction [23] or next utterance prediction [24] that uses raw subtitles as prediction targets, rather than abstract text. This approach introduces inconsistency between the pretraining and fine-tuning stages. In contrast, VAST fully harnesses the generalization capabilities of Large Language Models to extract the most crucial information from subtitles into language descriptions. On the other hand, notable examples of incorporating audio include AVLNet [25], MERLOT Reserve [26], i-Code [27], and VALOR [7]. However, AVLNet, MERLOT Reserve, and i-Code focus on learning audio-subtitle relations rather than audio-text, limiting their generalization to tasks such as text-to-audio retrieval and audio captioning. While VALOR jointly models audio, vision, and text, it primarily targets perceiving environmental sounds and pays less attention to human speech. In comparison to the aforementioned methods, VAST is the first omni-modality foundation model capable of perceiving four modalities, connecting vision, audio, and subtitle to caption.

3 Dataset

3.1 Data Collection of VAST-27M

Vision Captioner Training. To establish the general correspondence between objects and visual concepts, we draw inspiration from BLIP [28] and commence by training a vision caption model on large-scale image-text corpora, including CC4M, CC12M, and a randomly selected set of 100M image-text pairs from LAION-400M [29]. Subsequently, we fine-tune the model on a combination of manually labeled image and video caption datasets, such as MSCOCO [30], VATEX [31], MSRVTT [32], and MSVD [33]. Through this process, we achieve a high-quality captioner capable of perceiving both static objects and dynamic actions, thus generating smooth and accurate captions.

Audio Captioner Training. For the audio captioning task, we train a dedicated audio caption model using a combination of large-scale audio-text corpora, namely VALOR-1M and WavCaps datasets. Unlike its vision counterpart, we do not employ a second-stage fine-tuning due to the limited scale of downstream audio caption datasets, which could lead to overfitting to a narrow range of audio concepts. Additionally, VALOR-1M is a human-annotated corpus that provides a certain level of assurance regarding the accuracy of the generated captions.

Clip Selection. Due to the consideration about time and money cost of downloading, storage and computation, we select a 27M video clips from the HD_VILA_100M dataset instead of using all of them. The selection rules are introduced as follows: 1) Clips whose length are shorter than 5 seconds or longer than 30 seconds are dropped. 2) Clips which have missing modalities of vision, audio or subtitle are dropped. 3) Clips are evenly chosen from the original 3.3M long-form videos in HD_VILA_100M.

Text encoder

OM-VCC

Subtitle

Audio

OMC

Text encoder

Vision

Encoder

Audio

Encoder

Text encoder

Vision

OMC

OM-VCM

OM-VCG

Omni-Modality

Captioner (LLM)

VAST Training Framework

VAST-27M   Generation Pipeline

Vision

Audio

Text encoder

Subtitle

Vision

Captioner

Audio

Captioner

"She's in second place on this left opens up with a little snap, gets back into the lip to finish it off, rides through the

explosive closeout and look into better."

1. A person surfing in the ocean in front of the waves.

2. A man is surfing in the middle of the ocean and it

starts to wave in the ocean

。

3. A surfer on a surfboard rides in a wave, then jumps

up and starts surfing.

1. The ocean waves and men speaking are heard.

2. Someone is playing a surf - on.

3. Sea waves and people talking can be heard.

"Human:

About one video, here are 3 vision captions: {}, 3 audio captions: {}, and one speech subtitle: {}. You need to

understand and encode them into 1 sentence. Vision and audio captions maybe redundant, summarize them before encode

with speech subtitle. Do not simply concatenate them together. The weights of video/audio/speech are equaled.

Assistant:

The sentence is "

A surfer rides a

wave, jumping up

and surfing in the

ocean

with waves

and men speaking

in the background,

as they compete

for the top spot,

with a close-up of

the surfer finishing

the ride and

looking ahead.

Vision

Encoder

Audio

Encoder

Global representation

Local representation

Concatenation

f a

f s

f omv

f vg

f ag

f sg

f omcg

f omvg

Caption Generation. For each video clip of VAST-27M, we employ the trained vision and audio captioners to generate 5 captions each, using a Top-K sampling approach with K=10. Subsequently, we utilize the off-the-shelf Vicuna-13b [8] model as the omni-modality captioner. Vicuna-13b is an open-source auto-regressive Large Language Model (LLM) based on the Transformer architecture, trained through fine-tuning of LLaMA [34] on user-shared conversations collected from ShareGPT. For each video clip, we randomly select 3 vision captions and 3 audio captions, and feed them, along with the raw subtitle and designed instructional prompts, into the LLM. As depicted in Figure 2, the LLM generates omni-modality captions that effectively describe the visual, audio, and subtitle contents, while adhering to a natural human captioning style. More examples of VAST-27M dataset can be found in Appendix.

3.2 Statistics of V AST-27M

VAST-27M consists of a total of 27M video clips sampled from the large-scale HD_VILA_100M corpus. The dataset covers 15+ categories, including music, gaming, education, entertainment, animals, and more, as opposed to being limited to instructional domains like HowTo100M. Compared to other commonly used cross-modality training corpora, as shown in Table 2, V AST-27M contains a larger number of captions (in terms of both modalities and quantity). The average lengths of vision, audio, and omni-modality captions in VAST-27M are 12.5, 7.2, and 32.4, respectively.

4 Approach

4.1 Basic Framework

As shown in Figure 2, VAST employs a fully end-to-end Transformer architecture, comprising a vision encoder (ViT [35]), an audio encoder (BEATs [36]), and a text encoder (BERT [2]). This framework can accommodate multi-modal inputs such as images, videos, audios, subtitles, and captions. The text encoder is responsible for encoding single-modality captions or subtitles, as well as performing multi-modal encoding/decoding through cross-attention layers. The omni-modality captions and subtitles are tokenized using the WordPiece tokenizer [37] and then fed into the text encoder to obtain the output features f omc and f s , respectively. The vision encoder takes raw images

or sparsely sampled video frames as input and produces the output feature f v . For audio clips, they are first divided into multiple 10-second-long clips, padded with zeros, converted to 64-dimensional log Mel filterbank spectrograms using a 25ms Hamming window, and then fed into the audio encoder to obtain the output feature f a . The global representations ( [CLS] token feature) of these features are denoted as f omcg , f sg , f vg , and f ag , respectively.

4.2 Pretraining Objectives

Building upon conventional vision-text pretraining objectives, VAST employs the following three pretraining objectives:

Omni-Modality Video-Caption Contrastive Loss (OM-VCC). The global omni-modality representations of video clips, denoted as f omvg , are obtained by concatenating f sg , f vg , and f ag . Subsequently, f omvg and f omcg are projected into the same semantic space using two linear layers and are then normalized. The contrastive loss is employed to regularize the feature distance between omni-modality video (OMV) and caption (OMC). The contrastive loss is defined as follows, where sim ( · ) represents the dot product of f omcg and f omvg , and B and τ denote the batch size and a learnable parameter, respectively.

Omni-Modality Video-Caption Matching Loss (OM-VCM). This loss encourages the model to infer whether a pair of OMV and OMC is matched or not. Specifically, caption tokens are fed into the text encoder again, this time with cross-attention layers activated to attend to the condition features f omv which is obtained by concatenating those unpooled features f s , f v , and f a along the sequential dimension. Before concatenation, three independent linear layers are applied to adjust their hidden size to the same value. The output feature of the text encoder is then fed into a two-layer MLP to make binary predictions. To create informative negative pairs, we adopt a hard negative mining strategy following [38]. The loss function is formulated as follows, where y = 1 if OMV and OMC are matched, and 0 otherwise.

Omni-Modality Video Caption Generation Loss (OM-VCG). This loss employs conditional causal masked language modeling to enhance the model's ability to generate omni-modality captions. Specifically, 60% of the tokens in OMC are masked at the input of the text encoder. Cross-attention layers are activated and f omv is used as condition features as in OM-VCM. The self-attention layers in the text encoder utilize a single-directional causal attention mask to prevent information leakage, and the masked tokens are reconstructed at the output of the text encoder using BERT's vanilla prediction layer. The loss is defined as follows, where c m and c <m denote the masked tokens and the tokens before them, respectively.

The overall loss L OM is the sum of the three losses, with equal weights assigned for simplicity.

4.3 Modality Grouping

While VAST establishes omni-modality video-caption correspondence during pretraining, it is important to address the potential absence of modalities in downstream benchmarks and practical applications, due to that inconsistencies between the modalities used in pretraining and adaptation can have negative effects. Inspired by the modality grouping strategy proposed by VALOR [7], we uniformly model the relations of V-T, A-T, VA-T, VS-T, and VAS-T (previously introduced as L OM ). Specifically, vision and audio captions are used in V-T and A-T modeling, respectively, while omni-modality captions are employed in VA-T, VS-T, and VAS-T modeling. The final loss is formulated as follows:

Vision-Text

benchmark

T-to-V Retrieval

Captioning

QA

COCO[30]

Flickr[42]

Flickr(ZS)[42]

COCO[30]

TGIF[43]

MSVD[44]

VQAv2[45]

SOTA

68.3

[46]

90.3[47]

89.7[46]

154.9

*[48]

78.7[7]

60.2[49]

84.3

[50]

VAST

68.0

91.0

(+0.7)

90.4

(+0.7)

149.0*

79.1

(+0.4)

60.2

80.2

Audio-Text

benchmark

T-to-A Retrieval

Captioning

Clothov1[12]

Clothov2[12]

AudioCaps[13]

Clothov1[12]

Clothov2[12]

AudioCaps[13]

SOTA

17.5[7]

21.5[18]

42.2[18]

42.3[7]

48.8[18]

78.7

[18]

VAST

25.1

(+7.6)

26.9

(+5.4)

52.0

(+9.8)

50.7

(+8.4)

51.9

(+3.1)

78.2

MMVideo-Text

benchmark

T-to-V Retrieval

MSRVTT[32]

YouCook2[51]

VALOR-32K[7]

VATEX[31]

DiDeMo[52]

ANET[53]

SOTA_Vis

58.8[54]

33.7[55]

43.4[56]

71.1[57]

70.4[54]

66.8[54]

SOTA_MM

54.4[7]

31.3[20]

73.2[7]

76.9[7]

57.6[7]

63.4[7]

VAST

63.9

(+5.1)

50.4

(+16.7)

80.0

(+6.8)

83.0

(+6.1)

72.0

(+1.6)

70.5

(+3.7)

MMVideo-Text

benchmark

Captioning

QA

MSRVTT[32]

YouCook2[51]

VALOR-32K[7]

VATEX[31]

TVC[58]

MSRVTT[44]

MUSIC[59]

ANET[60]

SOTA_Vis

75.9[19]

131.2[19]

27.3[61]

94.5*[19]

66.1[19]

49.5[49]

-

47.9[54]

SOTA_MM

74.0[7]

190.0[55]

61.5[7]

95.8*[7]

66.0[62]

49.2[7]

78.9[7]

48.6[7]

VAST

78.0

(+2.1)

198.8

(+8.8)

62.0

(+0.5)

99.5*

(+3.7)

74.1

(+8.0)

50.1

(+0.6)

80.7

(+1.8)

50.4

(+1.8)

5 Experiments

5.1 Implementation Details

VAST is trained using the PyTorch framework on 64 Tesla V100 cards. The vision, audio, and text encoders are initialized from EV AClip-ViT-G [39], BEATs, and BERT-B, respectively, resulting in a total parameter size of 1.3B. The training is conducted on a combination corpus consisting of V AST27M, VALOR-1M, WavCaps, CC14M, and 110M randomly sampled pairs from LAION-400M, for a total of 200K training steps. At each training step, one corpus is sampled for training. In addition, the raw captions of CC14M and LAION are replaced with captions generated by the trained vision captioner. The initial learning rate is set to 1e-4, and a linear decay schedule is used. The batch size is set to 1024. During the pertaining stage, one video frame and two 10s long audio clips are randomly sampled for each video clip. For ablation studies, CLIP-ViT-B [40] is used as the vision encoder, with its parameters frozen to increase efficiency. Further details such as the mixture ratio of the pretraining dataset and the downstream finetuning configurations can be found in the Appendix.

Regarding the adaptation to downstream tasks, for retrieval tasks, all candidates are ranked using VCC, and then the Top-50 candidates are reranked using VCM. For captioning tasks, beam search with a beam size of 3 is employed. For QA tasks, they are formulated as open-ended generative problems, where questions serve as prefixes, and answers are predicted without any constraints. For SOTA comparison and ablation studies in the main paper, Recall@1, CIDEr, and Acc are used as evaluation metrics for retrieval, captioning, and QA tasks, respectively.

5.2 Comparison to State-of-the-Art Models

We present a comparison of the VAST foundation model with state-of-the-art models across various vision-text, audio-text, and multi-modal video-text benchmarks. The corresponding results are summarized in Table 3. Although the primary focus of VAST lies in enhancing omni-modality understanding and generation capabilities, it also demonstrates remarkable performance in image-text,

vision-only video-text, and audio-text benchmarks. Specifically, V AST surpasses BEiT-3 [47] and BLIP-2 [46] on the text-to-image retrieval benchmark of Flickr30K under finetuning and zero-shot settings, respectively, establishing new state-of-the-art results in TGIF-QA and MSVD-QA benchmarks. Furthermore, VAST exhibits exceptional performance in audio-text benchmarks, achieving five new state-of-the-art results with significant improvements, due to the abundant generated captions in the V AST-27M corpus.

In the context of multi-modal video-text benchmarks, they can be classified into three categories: subtitle-oriented (YouCook2, TVC) benchmarks that require a comprehensive understanding of subtitles for reasoning, audio-oriented (V ALOR-32K) benchmarks that necessitates careful attention to audio cues, and even-oriented benchmarks (others) where both audio and subtitles provide supportive roles to vision. Our results demonstrate that V AST excels across all types of benchmarks, outperforming previous both vision-only and multi-modal state-of-the-art models. Notably, we surpass the GIT2 [19] foundation model, specialized in captioning tasks, by 2.1, 67.6, 5.0, and 8.0 CIDEr points on MSRVTT, YouCook2, VATEX, and TVC captioning benchmarks, respectively, while utilizing only 22.5% of its parameters and 3.4% of its training data size. Furthermore, we achieve significant margins over the vision-audio-text foundation model VALOR [7] on 11 benchmarks. More detailed comparison results can be found in Appendix.

5.3 Comparison to Open-Source Cross-Modality Training Corpus

In this subsection, we quantitatively compare the quality of the proposed VAST-27M to current open-source video, audio, and audiovisual pretraining corpora. We achieve this by training models on these corpora and fine-tuning them on different types of downstream tasks, including video retrieval (RET), captioning (CAP), and question answering (QA).

V-T Quality. We train multiple V-T models on different corpora and fine-tune them on the MSVD and MSRVTT datasets. It is important to note that only the visual contents in videos are utilized for both pretraining and fine-tuning, and all models are trained for 50K steps. As shown in Table 4, the model trained with vision captions in V AST-27M (h) achieves the best results on all six benchmarks, outperforming the baselines and models trained with other corpora by a significant margin. In contrast, model (f), which treats subtitles as captions and can be viewed as raw HD_VILA_100M dataset, performs the worst due to the weak vision-text relations. Model trained on V AST-27M (g) outperforms other corpora but is still inferior to model (h), as the multi-modal captions introduce noise when only vision contents are fed into the model.

A-T Quality. We train multiple A-T models on different corpora and fine-tune them on audio-text benchmarks, including Clothov2 and AudioCaps. We observe that VALOR-1M and WavCaps are prone to overfitting, so we adjust the training iterations to obtain the best results for models trained on each corpus instead of training for the same number of iterations. As shown in Table 5, model (d) surpasses the baseline and models trained on both V ALOR-1M and WavCaps on all four benchmarks.

OMV-OMC Quality. We further investigate the correspondence quality between OMV and OMC in the VAST-27M dataset and compare it primarily to the VALOR-1M dataset. All models undergo 50K training steps with L OM as the objective, utilizing all modalities in both pretraining and finetuning. The results are presented in Table 6. From the table, we observe that omni-modality pretraining on VAST-27M significantly improves performance across all 7 benchmarks compared to the baseline. Additionally, V AST-27M outperforms VALOR-1M on 5 benchmarks in the MSRVTT and YouCook2 datasets, but lags behind on the V ALOR-32K benchmarks due to the consistent video distribution and caption style between VALOR-1M and VALOR-32K. As an additional experiment, we train model (c) on VAST-27M but replace the omni-modality captions generated by LLM with a simple concatenation of vision, audio captions, and subtitles. Model (d) outperforms model (c) on all benchmarks, showcasing the necessity and effectiveness of leveraging the powerful capabilities of LLM to integrate single-modality captions into omni-modality ones.

5.4 Ablation Study

We conduct comprehensive ablation studies on pretraining and finetuning models using different modalities for video representations to demonstrate the strength and necessity of omni-modality foundation model pretraining. Models are evaluated on the MSRVTT (open-domain), YouCook2 (subtitle-important), and VALOR-32K (audio-important) benchmarks. The results are presented in Table 7, when no pretraining is applied, models trained with audio and subtitles incorporated (d) can improve the vision-only baseline on all benchmarks. When omni-modality pretraining is applied, the improvement becomes more evidently on most benchmarks, which can be reflected by the comparison between the green values in model (f) and model (d), demonstrating the effectiveness of proposed method. In addition, model (i) outperforms model (f) on all benchmarks, indicating that, similar to the V-T capability, the OMV-OMC capability also benefits from large-scale pretraining. It is worth noting that model (h) and (f) exhibit inferior results on V ALOR-32K benchmarks compared to model (g) and even the vision baseline (e). This degradation can be attributed to the inconsistency between pretraining and finetuning, where all clips have subtitles in V AST-27M while most videos in VALOR-32K lack subtitles. When the modality grouping strategy is applied, model (j) demonstrates

good generalization across all types of tasks, as the correlations between text and every modality group have been explored during pretraining.

6 Conclusion, Broader Impact and Limitation

In this paper, we introduce the V AST-27M corpus, a large-scale omni-modality video caption dataset, aimed at advancing research in multi-modality pretraining. Each video clip in the dataset is accompanied by automatically generated vision and audio captions, as well as an omni-modality caption that integrates information from vision, audio, and subtitles using a pre-existing Large Language Model (LLM). We also train a unified foundation model called VAST, capable of understanding and connecting the various modalities in videos and captions. Through extensive evaluations, VAST demonstrates its effectiveness in a wide range of vision-text, audio-text, and multi-modal video-text tasks, including retrieval, captioning, and question answering, surpassing existing state-of-the-art methods on public cross-modality benchmarks.

The advancements in multi-modal understanding and generation of video content can have a significant impact on various domains such as entertainment, education, security, transportation, and healthcare, among others. The development of omni-modality foundation models, like VAST-27M and VAST, can contribute to the progress and practical applications in these areas. However, it is important to acknowledge the limitations of our method. For instance, there is a need for more diverse and larger-scale omni-modality corpora beyond VAST-27M. Furthermore, although VAST already supports a wide range of downstream tasks, the integration of LLM is necessary to further enhance its generalization capabilities. Additionally, due to the reason that the data collection process of VAST-27M have used LLM, and the training process of video and audio captioners have utilized some open-sourced cross-modality corpus, so VAST-27M dataset and VAST model may suffer the same bias of those datasets and models.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (No. 2022ZD0118801), National Natural Science Foundation of China (U21B2043, 62102416, 62206279).

References

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Advances in neural information processing systems , vol. 33, pp. 1877-1901, 2020.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'Bert: Pre-training of deep bidirectional transformers for language understanding,' arXiv preprint arXiv:1810.04805 , 2018.

M. Bain, A. Nagrani, G. Varol, and A. Zisserman, 'Frozen in time: A joint video and image encoder for end-to-end retrieval,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1728-1738.

A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, 'Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 2630-2640.

H. Xue, T. Hang, Y. Zeng, Y. Sun, B. Liu, H. Yang, J. Fu, and B. Guo, 'Advancing high-resolution video-language representation with large-scale video transcriptions,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5036-5045.

R. Zellers, X. Lu, J. Hessel, Y. Yu, J. S. Park, J. Cao, A. Farhadi, and Y. Choi, 'Merlot: Multimodal neural script knowledge models,' Advances in Neural Information Processing Systems , vol. 34, pp. 23 634-23 651, 2021.

S. Chen, X. He, L. Guo, X. Zhu, W. Wang, J. Tang, and J. Liu, 'Valor: Vision-audio-language omniperception pretraining model and dataset,' arXiv preprint arXiv:2304.08345 , 2023.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, 'Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,' March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/

S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, 'Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 3558-3568.

P. Sharma, N. Ding, S. Goodman, and R. Soricut, 'Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,' in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2018, pp. 2556-2565.

J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, 'Audio set: An ontology and human-labeled dataset for audio events,' in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2017, pp. 776-780.

K. Drossos, S. Lipping, and T. Virtanen, 'Clotho: An audio captioning dataset,' in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 736-740.

C. D. Kim, B. Kim, H. Lee, and G. Kim, 'Audiocaps: Generating captions for audios in the wild,' in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019, pp. 119-132.

I. Martin Morato and A. Mesaros, 'Diversity and bias in audio captioning datasets,' 2021.

M. Wu, H. Dinkel, and K. Yu, 'Audio caption: Listen and tell,' in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 830-834.

Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, 'Large-scale contrastive languageaudio pretraining with feature fusion and keyword-to-caption augmentation,' in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1-5.

F. Font, G. Roma, and X. Serra, 'Freesound technical demo,' in Proceedings of the 21st ACM international conference on Multimedia , 2013, pp. 411-412.

X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang, 'Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,' arXiv preprint arXiv:2303.17395 , 2023.

J. Wang, Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang, 'Git: A generative image-to-text transformer for vision and language,' arXiv preprint arXiv:2205.14100 , 2022.

L. Li, J. Lei, Z. Gan, L. Yu, Y.-C. Chen, R. Pillai, Y. Cheng, L. Zhou, X. E. Wang, W. Y. Wang et al. , 'Value: A multi-task benchmark for video-and-language understanding evaluation,' arXiv preprint arXiv:2106.04632 , 2021.

V. Gabeur, C. Sun, K. Alahari, and C. Schmid, 'Multi-modal transformer for video retrieval,' in European Conference on Computer Vision . Springer, 2020, pp. 214-229.

S. Chen, X. Zhu, D. Hao, W. Liu, J. Liu, Z. Zhao, L. Guo, and J. Liu, 'Mm21 pre-training for video understanding challenge: Video captioning with pretraining techniques,' in Proceedings of the 29th ACM International Conference on Multimedia , 2021, pp. 4853-4857.

H. Luo, L. Ji, B. Shi, H. Huang, N. Duan, T. Li, J. Li, T. Bharti, and M. Zhou, 'Univl: A unified video and language pre-training model for multimodal understanding and generation,' arXiv preprint arXiv:2002.06353 , 2020.

P. H. Seo, A. Nagrani, and C. Schmid, 'Look before you speak: Visually contextualized utterances,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 16 877-16 887.

A. Rouditchenko, A. Boggust, D. Harwath, B. Chen, D. Joshi, S. Thomas, K. Audhkhasi, H. Kuehne, R. Panda, R. Feris et al. , 'Avlnet: Learning audio-visual language representations from instructional videos,' arXiv preprint arXiv:2006.09199 , 2020.

R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi, 'Merlot reserve: Neural script knowledge through vision and language and sound,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 375-16 387.

Z. Yang, Y. Fang, C. Zhu, R. Pryzant, D. Chen, Y. Shi, Y. Xu, Y. Qian, M. Gao, Y.-L. Chen et al. , 'i-code: An integrative and composable multimodal learning framework,' arXiv preprint arXiv:2205.01818 , 2022.

J. Li, D. Li, C. Xiong, and S. Hoi, 'Blip: Bootstrapping language-image pre-training for unified visionlanguage understanding and generation,' arXiv preprint arXiv:2201.12086 , 2022.

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, 'Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,' arXiv preprint arXiv:2111.02114 , 2021.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, 'Microsoft coco: Common objects in context,' in European conference on computer vision . Springer, 2014, pp. 740-755.

X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang, 'Vatex: A large-scale, high-quality multilingual dataset for video-and-language research,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 4581-4591.

J. Xu, T. Mei, T. Yao, and Y. Rui, 'Msr-vtt: A large video description dataset for bridging video and language,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 5288-5296.

D. Chen and W. B. Dolan, 'Collecting highly parallel data for paraphrase evaluation,' in Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies , 2011, pp. 190-200.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , 'Llama: Open and efficient foundation language models,' arXiv preprint arXiv:2302.13971 , 2023.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020.

S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, 'Beats: Audio pre-training with acoustic tokenizers,' arXiv preprint arXiv:2212.09058 , 2022.

Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al. , 'Google's neural machine translation system: Bridging the gap between human and machine translation,' arXiv preprint arXiv:1609.08144 , 2016.

J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, 'Align before fuse: Vision and language representation learning with momentum distillation,' Advances in neural information processing systems , vol. 34, pp. 9694-9705, 2021.

Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao, 'Eva-clip: Improved training techniques for clip at scale,' arXiv preprint arXiv:2303.15389 , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in International Conference on Machine Learning . PMLR, 2021, pp. 8748-8763.

S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, 'Self-critical sequence training for image captioning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 7008-7024.

B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, 'Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,' in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2641-2649.

Y. Li, Y. Song, L. Cao, J. Tetreault, L. Goldberg, A. Jaimes, and J. Luo, 'Tgif: A new dataset and benchmark on animated gif description,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 4641-4650.

D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang, 'Video question answering via gradually refined attention over appearance and motion,' in Proceedings of the 25th ACM international conference on Multimedia , 2017, pp. 1645-1653.

Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, 'Making the v in vqa matter: Elevating the role of image understanding in visual question answering,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 6904-6913.

J. Li, D. Li, S. Savarese, and S. Hoi, 'Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,' arXiv preprint arXiv:2301.12597 , 2023.

W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som et al. , 'Image as a foreign language: Beit pretraining for all vision and vision-language tasks,' arXiv preprint arXiv:2208.10442 , 2022.

P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang, 'Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,' in International Conference on Machine Learning . PMLR, 2022, pp. 23 318-23 340.

W. Kuo, A. Piergiovanni, D. Kim, X. Luo, B. Caine, W. Li, A. Ogale, L. Zhou, A. Dai, Z. Chen et al. , 'Mammut: A simple architecture for joint learning for multimodal tasks,' arXiv preprint arXiv:2303.16839 , 2023.

X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer et al. , 'Pali: A jointly-scaled multilingual language-image model,' arXiv preprint arXiv:2209.06794 , 2022.

L. Zhou, C. Xu, and J. J. Corso, 'Towards automatic learning of procedures from web instructional videos,' in Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

L. Anne Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell, 'Localizing moments in video with natural language,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 5803-5812.

R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles, 'Dense-captioning events in videos,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 706-715.

K. Li, Y. Wang, Y. Li, Y. Wang, Y. He, L. Wang, and Y. Qiao, 'Unmasked teacher: Towards trainingefficient video foundation models,' arXiv preprint arXiv:2303.16058 , 2023.

D. Ko, J. Choi, H. K. Choi, K.-W. On, B. Roh, and H. J. Kim, 'Meltr: Meta loss transformer for learning to fine-tune video foundation models,' arXiv preprint arXiv:2303.13009 , 2023.

H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li, 'Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning,' Neurocomputing , vol. 508, pp. 293-304, 2022.

Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang et al. , 'Internvideo: General video foundation models via generative and discriminative learning,' arXiv preprint arXiv:2212.03191 , 2022.

J. Lei, L. Yu, T. L. Berg, and M. Bansal, 'Tvr: A large-scale dataset for video-subtitle moment retrieval,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16 . Springer, 2020, pp. 447-463.

G. Li, Y. Wei, Y. Tian, C. Xu, J.-R. Wen, and D. Hu, 'Learning to answer questions in dynamic audio-visual scenarios,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 19 108-19 118.

Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao, 'Activitynet-qa: A dataset for understanding complex web videos via question answering,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, no. 01, 2019, pp. 9127-9134.

K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y. Lu, and L. Wang, 'Swinbert: End-to-end transformers with sparse attention for video captioning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 17 949-17 958.

M. Tang, Z. Wang, Z. Zeng, F. Rao, and D. Li, 'Clip4caption++: Multi-clip for video caption,' arXiv preprint arXiv:2110.05204 , 2021.

W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev et al. , 'The kinetics human action video dataset,' arXiv preprint arXiv:1705.06950 , 2017.

S. Chen, Y. Zhao, Q. Jin, and Q. Wu, 'Fine-grained video-text retrieval with hierarchical graph reasoning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 638-10 647.

A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , vol. 123, no. 1, pp. 94-120, 2017.

Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim, 'Tgif-qa: Toward spatio-temporal reasoning in visual question answering,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 2758-2766.

A.-M. Oncescu, A. Koepke, J. F. Henriques, Z. Akata, and S. Albanie, 'Audio retrieval with natural language queries,' arXiv preprint arXiv:2105.02192 , 2021.

A. Karpathy and L. Fei-Fei, 'Deep visual-semantic alignments for generating image descriptions,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3128-3137.

X. Cheng, H. Lin, X. Wu, F. Yang, and D. Shen, 'Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss,' arXiv preprint arXiv:2109.04290 , 2021.

J. Lei, T. L. Berg, and M. Bansal, 'Revealing single frame bias for video-and-language learning,' arXiv preprint arXiv:2206.03428 , 2022.

J. Wang, D. Chen, Z. Wu, C. Luo, L. Zhou, Y. Zhao, Y. Xie, C. Liu, Y.-G. Jiang, and L. Yuan, 'Omnivl: One foundation model for image-language and video-language tasks,' arXiv preprint arXiv:2209.07526 , 2022.

Q. Ye, G. Xu, M. Yan, H. Xu, Q. Qian, J. Zhang, and F. Huang, 'Hitea: Hierarchical temporal-aware video-language pre-training,' arXiv preprint arXiv:2212.14546 , 2022.

F. Cheng, X. Wang, J. Lei, D. Crandall, M. Bansal, and G. Bertasius, 'Vindlu: A recipe for effective video-and-language pretraining,' arXiv preprint arXiv:2212.05051 , 2022.

L. Li, Z. Gan, K. Lin, C.-C. Lin, Z. Liu, C. Liu, and L. Wang, 'Lavender: Unifying video-language understanding as masked language modeling,' arXiv preprint arXiv:2206.07160 , 2022.

A. J. Wang, Y. Ge, R. Yan, Y. Ge, X. Lin, G. Cai, J. Wu, Y. Shan, X. Qie, and M. Z. Shou, 'All in one: Exploring unified video-language pre-training,' arXiv preprint arXiv:2203.07303 , 2022.

Y. Ma, G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji, 'X-clip: End-to-end multi-grained contrastive learning for video-text retrieval,' in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 638-647.

H. Xu, Q. Ye, M. Yan, Y. Shi, J. Ye, Y . Xu, C. Li, B. Bi, Q. Qian, W. Wang et al. , 'mplug-2: A modularized multi-modal foundation model across text, image and video,' arXiv preprint arXiv:2302.00402 , 2023.

H. Xue, Y. Sun, B. Liu, J. Fu, R. Song, H. Li, and J. Luo, 'Clip-vip: Adapting pre-trained image-text model to video-language representation alignment,' arXiv preprint arXiv:2209.06430 , 2022.

V. Gabeur, A. Nagrani, C. Sun, K. Alahari, and C. Schmid, 'Masking modalities for cross-modal video retrieval,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2022, pp. 1766-1775.

Y.-B. Lin, J. Lei, M. Bansal, and G. Bertasius, 'Eclipse: Efficient long-range video retrieval using sight and sound,' arXiv preprint arXiv:2204.02874 , 2022.

M. Patrick, P.-Y. Huang, Y. Asano, F. Metze, A. Hauptmann, J. Henriques, and A. Vedaldi, 'Support-set bottlenecks for video-text representation learning,' arXiv preprint arXiv:2010.02824 , 2020.

Q. Wang, Y. Zhang, Y. Zheng, P. Pan, and X.-S. Hua, 'Disentangled representation learning for text-video retrieval,' arXiv preprint arXiv:2203.07111 , 2022.

H. Xu, G. Ghosh, P.-Y. Huang, P. Arora, M. Aminzadeh, C. Feichtenhofer, F. Metze, and L. Zettlemoyer, 'Vlm: Task-agnostic video-language model pre-training for video understanding,' arXiv preprint arXiv:2105.09996 , 2021.

D. Li, J. Li, H. Li, J. C. Niebles, and S. C. Hoi, 'Align and prompt: Video-and-language pre-training with entity prompts,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 4953-4963.

T.-J. Fu, L. Li, Z. Gan, K. Lin, W. Y. Wang, L. Wang, and Z. Liu, 'Violet: End-to-end video-language transformers with masked visual-token modeling,' arXiv preprint arXiv:2111.12681 , 2021.

L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li et al. , 'Florence: A new foundation model for computer vision,' arXiv preprint arXiv:2111.11432 , 2021.

J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu, 'Less is more: Clipbert for video-andlanguage learning via sparse sampling,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 7331-7341.

T.-J. Fu, L. Li, Z. Gan, K. Lin, W. Y. Wang, L. Wang, and Z. Liu, 'An empirical study of end-to-end video-language transformers with masked visual modeling,' arXiv preprint arXiv:2209.01540 , 2022.

J. Huang, Y. Li, J. Feng, X. Sun, and R. Ji, 'Clover: Towards a unified video-language alignment and fusion model,' arXiv preprint arXiv:2207.07885 , 2022.

A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, 'Just ask: Learning to answer questions from millions of narrated videos,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1686-1697.

--, 'Zero-shot video question answering via frozen bidirectional language models,' arXiv preprint arXiv:2206.08155 , 2022.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , 'Flamingo: a visual language model for few-shot learning,' arXiv preprint arXiv:2204.14198 , 2022.

S. Yan, T. Zhu, Z. Wang, Y. Cao, M. Zhang, S. Ghosh, Y. Wu, and J. Yu, 'Video-text modeling with zero-shot transfer from contrastive captioners,' arXiv preprint arXiv:2212.04979 , 2022.

A. Nagrani, P. H. Seo, B. Seybold, A. Hauth, S. Manen, C. Sun, and C. Schmid, 'Learning audio-video modalities from image captions,' arXiv preprint arXiv:2204.00679 , 2022.

P. Wang, S. Wang, J. Lin, S. Bai, X. Zhou, J. Zhou, X. Wang, and C. Zhou, 'One-peace: Exploring one general representation model toward unlimited modalities,' arXiv preprint arXiv:2305.11172 , 2023.

X. Xu, H. Dinkel, M. Wu, Z. Xie, and K. Yu, 'Investigating local and global information for automated audio captioning with transfer learning,' in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 905-909.

Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao, 'Simvlm: Simple visual language model pretraining with weak supervision,' arXiv preprint arXiv:2108.10904 , 2021.

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, 'Scaling up visual and vision-language representation learning with noisy text supervision,' in International Conference on Machine Learning . PMLR, 2021, pp. 4904-4916.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, 'Coca: Contrastive captioners are image-text foundation models,' arXiv preprint arXiv:2205.01917 , 2022.

M. Ding, B. Xiao, N. Codella, P. Luo, J. Wang, and L. Yuan, 'Davit: Dual attention vision transformers,' arXiv preprint arXiv:2204.03645 , 2022.

Appendix

A More about VAST Foundation Model

A.1 Pretraining Settings

Specific pretraining configurations of V AST including training corpora, training steps for each corpus (i.e., dataset mix ratio), and training objectives on each corpus are presented in Table 8. To enhance data quality, we use trained vision captioner to generate new captions for CC12M and LAION datasets and replace original captions with them. It is noted that V AST have been trained for relatively small steps (205K steps), but have already shown excellent performances on various types of downstream tasks, and we believe that training by more steps can further increase the model capabilities.

Table 8: Model configurations and pretraining settings of V AST. It is noted that 400M Web data used in CLIP [40] and LAION-400M [29] used in EVAClip [39] are also counted for training samples statics. LAION-102M and LAION-110M are both random sampled subsets from LAION-400M. Regarding training objectives, 'ret' represents for the combination of VCC and VCM, while 'cap' denotes VCG, and differnet modality groups are separeted by '%'.

A.2 Downstream Datasets Descriptions

We evaluate VAST on multiple popular domnstream datasets, including MSRVTT, VATEX, YouCook2, VALOR-32K, MSVD, LSMDC, DiDeMo, ActivityNet Caption, TGIF, MUSIC-AVQA, TVC, Clotho, AudioCaps, MSCOCO, Flickr30K and VQAv2. Specific train/val/test splits of those benchmarks can be found in Table 9 and specific descriptions of them are as follows.

MSRVTT [32] contains 10K video clips and 200K captions. The videos cover a wide range of topics and scenes, including human activities, sports, natural landscapes, and more. We evaluate text-to-video retrieval, video captioning and video QA on this dataset. Following methods presented in Table 11, we use the '1K-A split' for retrieval evaluation. For captioning and QA, we use the standard split.

VATEX [31] contains 41,250 video clips sourced from Kinetics-600 dataset [63] and 825,000 sentence-level descriptions. We evaluate text-to-video retrieval and video captioning on this dataset. For captioning, we use the official split. For retrieval. we follow the HGR [64] split protocol.

YouCook2 [51] consists of 14K video clips from 2K instructional cooking videos from YouTube. Each video includes multiple actions performed by the chef, along with corresponding textual descriptions and temporal annotations. We evaluate text-to-video retrieval and video captioning on this dataset with official splits.

VALOR-32K [7] is an audiovisual video-language benchmark that contains 32K 10 seconds long audible video clips sourced from AudioSet [11]. Each video clip is annotated with an audiovisual caption which simultaneously describes both visual and audio contents in videos. We evaluate text-to-video retrieval and video captioning on this dataset with official splits.

MSVD [33] contains 1,970 videos, each of which is paired with around 40 captions. We evaluate video QA on this dataset and use the split proposed by Xu et al. [44].

LSMDC [65] consists of 118K clips form 202 movies, each of which is paird with one caption. We evaluate text-to-video retrieval on this dataset with official split.

DiDeMo [52] contains 10K long-form videos from Flickr and for each video, four short sentences are annotated in temporal order. We follow methods in Table 11 to concatenate those short sentences and evaluate 'paragraph-to-video' retrieval on this benchmark. The official split is used.

ActivityNet Caption [53] contains 20K long-form videos (180s as average length) from YouTube and 100K captions. We evaluate text-to-video retrieval and video QA on this dataset. For retrieval we use official split and for video QA, split proposed by Yu et al. [60] is used.

TGIF [66] contains three video QA benchmarks including TGIF-Action, TGIF-transition and TGIFFrame, and the first two are multiple-choice QA while the last is open-ended QA. We evaluate V AST on TGIF-frame benchmark with official split.

MUSIC-AVQA [59] is a audiovisual video QA benchmark containing more than 45K Q-A pairs covering 33 different question templates spanning over different modalities and question types. The offcial split is used.

TVC [58] is a multi-channel video captioning dataset containing 108K video moments and 262K paired captions. Video subtitles can be used as additional input. We evaluate video captioning on this benchmark with official split.

Clotho [12] contains 15-30 second audio clips and has two versions. The original (v1) has 4981 audios, while an expanded version (v2) includes 6974 audios, enlarging solely the training set. We evaluate text-to-audio retrieval and audio captioning on those benchmarks with official split.

AudioCaps [13] contains 51K 10-second clips, with one caption in the training set and five in the validation and test sets. We evaluate text-to-audio retrieval and audio captioning on it. For captioning, we use the official split, and for retrieval we follow the Sophia et al. [67] split protocol.

MSCOCO [30] contains 123K images each of which is paired with 5 annotated captions, We evaluate text-to-image retrieval and image captioning on this dataset with Karpathy split [68].

Flickr30K [42] contains 31K images each of which is paired with 5 annotated captions, We evaluate text-to-image retrieval on this dataset with Karpathy split [68].

VQAv2 [45] was used as the basis of the 2017 VQA Challenge2, it contains 1.1M questions with 11.1M answers relating to MSCOCO images. The official split is used.

A.3 Finetuning Settings

Specific finetuning hyperparameters of VAST for different benchmarks are presented in Table 10.

A.4 Detailed Comparisons to State-of-the-Art Methods

Text-to-Video Retrieval . We compare VAST to SOTA methods on six multi-modal text-to-video retrieval benchmarks. As shown in Table 4, VAST improves previous SOTA methods by 5.1, 1.6, 3.7, 6.1 points on MSRVTT, DiDeMo, ActivityNet, VATEX benchmarks, respectively. Besides above mentioned vision-oriented benchmarks, VAST outperforms VALOR-L [7] by 6.8 points on the audiooriented benchmark VALOR-32K, and surpass MELTR [55] by 16.7 points on the subtitle-oriented benchmark YouCook2, which demonstrate the strong generalization capabilities of VAST towards different types of downstream datasets. In addition, the zero-shot retrieval performance comparison is shown in Table 12, VAST achieves 49.3 and 55.5 zero-shot R@1 performance that surpasses previous SOTA by 7.3 and 6.9 points, respectively.

Video QA . We evaluate VAST on five open-ended video QA benchmarks. As shown in Table 13, VAST have achieved new SOTA performances on all benchmarks, and outperform recent proposed large-scale foundation models such as GIT [19], MaMMUT [49], Flamingo [92] and CoCa [93]. In addition, on the audiovisual video QA benchmark MUSIC-AVQA, VAST surpasses VALOR by 1.8 points, demonstrating its better capabilities to answer both visual and audio questions.

Video Captioning . In Table 14, we compare VAST to state-of-the-art methods on five multi-modal video captioning benchmarks. According to the results, VAST have achieved new state-of-the-art CIDEr score on all five benchmarks with evident margins. Compared to previous vision-language modal SOTA method GIT [19] which takes a 5.1B DaViT [100] as vision encoder and conduct pretraining on 12.9B private image-text corpus, VAST surpass it with only 22.5% parameters and 3.4% training data, demonstrating the high efficiency of our method. Compared to previous multimodal video-language SOTA method VALOR [19], VAST can additionally process subtitle-oriented benchmarks such as YouCook2 and TVC, and achieves better results due to that it jointly models the relations between text and omni-modalities in videos.

Text-to-Audio Retrieval and Audio Captioning . As shown in Table 15, VAST have largely improved previous SOTA methods on three text-to-audio retrieval benchmarks, by 7.6, 5.4 and 9.8 R@1 points, respectively. and for audio captioning task, VAST achieves new SOTA performances on Clotho benchmark (both V1 and V2), and comparable performance on AudioCaps benchmark to WavCaps [18]. It is noted that WavCaps explored four model architectures with different audio encoder and text encoders targeting at different benchmarks, while V AST takes a unified architecture without targeted optimizations for specific downstream benchmarks.

Image-Text Benchmarks . We evaluate VAST on text-to-image retrieval, image captioning and image QA benchmarks. The results are presented in Table 17, from which we can find that even though VAST is designed as a omni-modality video-language understanding and generation model, it also shows strong capabilities on image-text benchmarks, demonstrating its generalization capabilities towards tasks of various modality types. Specifically, V AST achieves new SOTA performance on R@1

score of Flicker30K and R@5, R@10 scores of MSCOCO dataset, which outperforms image-text pretrained foundation models such as BLIP-2 [46] and BEiT-3 [47]. On COCO caption benchmark, VAST achieves 27.0 SPICE score which outperforms all previous methods such as OFA [48] and GIT2 [19]. On image QA benchmark, VAST achieves better performance than GIT [19], which is also a generative methods and predicts answers in a fully open way without any constraints.

B More about VAST-27M Dataset

B.1 Word cloud distribution

Audio Caption

Omni-modality Caption

Subtitle

Vision Caption

camera

hair

televl

fish

wanotherroom

sit

bo

girl

lasses

phone

mater

red

suit

plarge

report

S

e

one

stands

new

e11on

blue

paysomeonentyoungpiece

players

ta

screen

=

E

different

speaks

food language

face

shirtinterviewed

otpnas

box

speech

top

guy

paper

moving,

dog

ho

peopleca

n

smal1foreln

field

ining

wearing

tdx

use

woman

6.0

Peo

hands

interview

sitting

gives

walks

913

showing

open

background

soccer

women.

going

st

shown

news

computer

glrls

white

monstrates

crowd

shows

boy

holds

show

inside

sits

handstanding

next

game

menusing

audience

Video

two

tching

front

playerblack

talkgroup

t0ok #

demonstrating

giving

three

Kol

1onpod

music

making

g

riding

SUTPTTN

clip

chi1d

words

iling

Somethingasking

footballstreet created

man

background

crowd

male

opening

song

game

may

funny

ned

PTO

girl

question

<ing

playing

persongo

driving

noise

things

report

product

Sututerdx

message

sayingLpeople

sound

xa

event

time

short

phrase

microphone

train

dog

good

七

oroadd

using

singing

played

hometelling

STA

Aa

cooking

gace

recorded

says

place

-giving

Sutuaddeu

vojceoyer

conwersation

speech

present

car

sounds

introduc

50

local

food

品

spoicen

park

snall

freesound

female

trying

used

bus

thank

deyice

1

U

makingmadewomanorng

kitchenbgyideo

erfect

restaur ant

projectannouncementaudio

announcer

water interviewed

studio

v01ce

radio

machine

sampleg

goingg

we11

video

usaop

feel

people

find

keep

guys

H

man

see

go

needdidnt youreyears

music

point

thatstime

oh

quite

since

work

gonw

prettyreaoy

first

lets

peq

okay

better

ves

getl

new

next

theyre

think

using

th

life

year

day

lve

dontwould

M

gamelastnice

W

rea

11

1ooking

still

ylnous

end

place

case

1ly

might

back

S

@

工

give

try

y

sKeMTe

actually

tWOcant

anysay

100gaid

maybe

bit

water

gt

fun

na

take

inside

even

ve

put

Auew

sht

big

team

adyeah

enever

mean

thing

Waytodayalsohes

Mnt

little

sure

start

voiceplayingSomeone

mentions

Video

sound

help

three

water

mll

showrirstl

ca

e

wearing

front

eatwres

anotherput

walking

like

speech

foodcurstergiving

smal1

great

spunot

S

e

O

ogking

make

city

things

around

guy

heard

explain

one

making

talk

going

cunning

looking

audio

movie

WC

getpiece

na

ediscussing

subt

cquld

oman

work

g

side

ame

news

standing

inside

oid ahite

report

explains

dog,

made

A3

play

use

coeputer

stage

good

nale

S

.time

speaks

二

speaking

recordedyoung

Music

phone

driving

men

need

red

thinks

want

back

team

radio

interviewed

mentioning

showing

talks

mention

sitting

two

saying

bey

sJeas.

stalT

working

saysbest

taking

shirt

better

shows

pafetd

B.2 Prompts for Omni-Modality Caption Generation

Single Modality Captions

1. A person surfing in the ocean in front of the waves.

2. A man is surfing in the middle of the ocean and it

starts to wave in the ocean

。

3. A surfer on a surfboard rides in a wave, then jumps up

and starts surfing.

"She's in second place on this left opens up with a

little snap, gets back into the lip to finish it off, rides

through the explosive closeout and look into better."

A surfer rides a wave, jumping up and surfing in the

ocean

with waves and men speaking in the

background, as they compete for the top spot,

with a close-up of the surfer finishing the ride and

looking ahead.

1. The ocean waves and men speaking are heard.

2. Someone is playing a surf - on.

3. Sea waves and people talking can be heard.

"Human:

About one video, here are 3 vision captions: {}, 3

audio captions: {}, and one speech subtitle: {}. You need to

understand and encode them into 1 sentence. Vision and

audio captions maybe redundant, summarize them before

encode with speech subtitle. Do not simply concatenate them

together. The weights of video/audio/speech are equaled.

Assistant:

The sentence is "

A person surfing in the ocean in front of the waves with

the sound of the ocean waves and men speaking, a man is

surfing in the middle of the ocean with the sound of

someone playing a surfboard, and a surfer on a surfboard

rides in a wave, then jumps up and starts surfing with the

sound of sea waves and people talking ...

"Human:

About one video, here are 3 vision captions: {}, 3

audio captions: {}, and one speech subtitle: {}. You need to

understand and encode them into 1 sentence. Vision and

audio captions maybe redundant, summarize them before

encode with speech subtitle. Do not simply concatenate them

together. The weights of video/audio/speech are equaled.

Assistant:

The sentence is "

A surfer rides a wave, performing a snap and finishing it

off, before navigating through an explosive closeout and

looking for a better spot.

"Human:

About one video, here are 3 vision captions: {}, 3

audio captions: {}, and one speech subtitle: {}. You need to

understand and encode them into 1 sentence. Vision and

audio captions maybe redundant, summarize them before

encode with speech subtitle. Do not simply concatenate them

together. The weights of video/audio/speech are equaled.

Assistant:

The sentence is "

Omni-Modality Captioner

(Vicuna-13b)

Omni-Modality Captioner

(Vicuna-13b)

Omni-Modality Captioner

(Vicuna-13b)

Onmi-Modality Caption Generation with Different Instructional Prompts

B.3 More Examples

1. a young boy riding a skateboard at a skate park.

2. a man is skateboarding.

3. a person riding a skateboard on a skate park course.

This is to use your hands once you swing it above through

your legs. Just grab just grab your nose a little bit and help

get it in there.

A young boy rides

a skateboard at a

skate park, while

talking about

skateboarding

moves and

explaining how to

do a trick with

your hands and

grab your nose to

help get it in

there.

1. skateboarder is explaining skateboarding moves.

2. skateboarder talking about skateboard.

3. skateboarders are skating.

1. cartoon characters are getting ready to go with a suitcase.

2. a group of piggy characters are in their house with a

suitcase.

3. a group of cartoon characters have their suitcase open in a

bedroom.

Are you sure we need it all? Yes, its all very important.

A group of cartoon

characters,

including piggy

ones, are preparing

to go on a trip, with

one character

checking if they

need to bring

everything and

another character

confirming the

importance of all

the items.

1. a cartoon character is speaking.

2. an animated character is talking.

3. someone is making a conversation in a cartoon.

1. a car driver is driving down a high hill on the road.

2. a man in sunglasses is talking while driving a car on a road.

3. a man is driving down the road.

Milk is readily available in market nowadays. So you can

get that condensed Milk. so here I put the condensed

milk in a big bowl.

A man in

sunglasses is

talking while

driving a car with

a high speed

engine sound in

the background,

on a road, with

the desire to have

unlimited fuel and

just drive off into

the sunset

forever.

1. a man is speaking about driving a car with a high

speed engine sound in the background.

2. a person is speaking.

3. a car engine is heard.

1. a woman is pouring a liquid into a large glass bowl.

2. a person is putting yellow cheese in a bowl.

3. the person is pouring butter into a bowl.

Milk is readily available in market nowadays. So you can get

that condensed Milk. so here I put the condensed milk in a

big bowl.

A woman is

introducing and

showing her

ingredients,

including

condensed milk,

which is readily

available in the

market, and is

putting it in a big

bowl.

1. a woman is introducing and showing her ingredients.

2. a woman is introducing her favorite foods.

3. a person is cooking.

00

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an automatic pipeline to create captions for videos, where multiple modalities are involved in the generation process of the captions. Authors initially train separate audio and video-based captioners. Then, those audiovisual-driven captions, alongside subtitles are restructured into a single caption by prompting a pertained LLM that is responsible for aggregating multiple text/caption sources. Next, such a dataset is used to train a multimodal model capable of addressing a variety of text-to-video/audio retrieval, generation, and question-answering tasks.\n\nSTRENGTHS:\n1. This work confirms that today’s unimodal caption generation models (datasets) are very strong baselines (resources) that diminish the need for a multimodal data collection process which could be costly and non-scaleable, thanks to summarization and aggregation capabilities of the latest LLMs.\n2. The downstream experiments are comprehensive and show promising results.\n\nWEAKNESSES:\n1. Paper is not easy to follow, especially due to the poor choice of naming for components (i.e features, losses etc) and abbreviations. \n2. Paper is highly incremental. It is heavily built on top of numerous existing pertained models and offers almost no technical novelty even in the aggregation step. \n3. Authors propose multiple components to the loss function (Eq 1-5). Still, there is almost no discussion on how these different components interact with one another, and impact the quality of learned representations. The experimental section is only focused on downstream tasks and the generated dataset.\n4. The dataset generation step is shaped by the power of the pertained models which authors have adopted. They are also combining multiple datasets during pertaining. Hence, outperforming SOTA is not only expected but one wonders whether such comparisons are even fair. At the end of the day, VAST stands on the shoulders of numerous pertained models developed by others.\n\nQUESTIONS:\n1. On OM-VCM: $f_{omv}$ which is fed to the text encoder plays the role of condition, however, a part of  $f_{omv}$ is $f_{s}$ which itself is the output of text encoder. I am not quite sure how this makes sense in terms of the space of inputs to the text encoder. The authors should clarify and elaborate.\n2. Line 206: what does “we uniformly model the relations of V-T, A-T, VA-T, VS-T, and VAS-T ” mean?\n3. Eq 2: why this cannot be captured via contrastive loss? Is it only due to maintaining temporal resolution? If so, authors should try sequences of length 1 which is [cls] token to confirm that.\n4. Line 219: using a single video frame is quite strange. Considering the length of video clips (e.g 5-30 seconds for VAST) does it make sense to pick a single frame, especially if we are aiming to create captions for videos? Did authors try more than 1 frame during pertaining to study the effect of such choice? \n5. Line 223-224: What is the contribution of VCM. In other words, what if you equally weigh the losses or reverse them, how do results vary?\n\nLIMITATIONS:\nThis work relies on multiple pretrained models and datasets. The bias-fairness concerns hence scale considerably due to such a design. On top of it is the choice of leveraging LLMs whose hidden biases are not quite yet clear for the research community. This is not to discourage using such models, but I believe authors can further improve the limitation and broader impact section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper introduces a large-scale dataset called VAST-27M, which contains audios, videos, subtitles, and generated captions. The authors employ an LLM to combine all forms of textual information to create omni-modality captions. They train an omni-modality video-text foundational model on this dataset. Extensive experiments conducted on various downstream tasks demonstrate the effectiveness of both their dataset and model.

STRENGTHS:
- They provide a valuable dataset, and the experiments demonstrate the superiority of this dataset. 
- The paper is well-written with clear explanations. 
- The experiments are thorough, including the validation of the methodology, the effectiveness of the dataset, and comprehensive comparisons with the state-of-the-art (SOTA) approaches. 
- The model achieves state-of-the-art (SOTA) performance on 22 downstream tasks.

WEAKNESSES:
- The authors utilized the Vicuna model instead of ChatGPT, which might have an impact on the quality of the data. Could you provide a comparison of the quality of data generated by ChatGPT and Vicuna?
- Why wasn't there consideration for multimodal LLMs like LLaVa or MiniGPT-4, and instead, but using a separate trained captioning model?
- The methodology lacks technical contributions and doesn't differ significantly from previous works in terms of model architecture and training. When dealing with multiple forms of text simultaneously, the current models simply aggregate multiple losses, which may not fully leverage the available information.

QUESTIONS:
I'd like to inquire about the authors' plans for sharing the data and code, including the captioning model, VAST model, and the dataset.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed to build a new multi-modal foundation model that includes vision, audio, subtitle and text modalities. The designed architecture of the model is very similar to previous works, such as BLIP. The paper also made another contribution, a new multi-modal video dataset VAST-27M. VAST-27M reuse videos from another dataset HD_VILA_100M. The paper uses off-the-shelf video captioning models and audio captioning models to generate text descriptions for the input video. Large language models are then applied to summarize the text descriptions together with the original subtitles of the video using customized prompts. The paper presents experimental results on 22 downstream datasets to demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS:\nHigh-quality pre-training datasets that are openly accessible to the research community is very important for us to experiment with new research ideas and push the research boundaries. The new VAST-27M dataset could be very valuable for researchers working on related topics. The experimental results also demonstrate the advantages of the trained model.\n\nWEAKNESSES:\nThe following are more detailed comments and suggestions about the paper.\n\n1, What is the difference between subtitles and text? The paper seems to treat them as different modalities. Also does the audio modality already contain all the information from subtitles?\n\n2, Based on the implementation  details in Section 5.1, the final model is initialized with other pretrained encoders, and pretrained with lots of other datasets as well. It is not clear how much the improvement is from the proposed VAST-27M dataset. It will be better if the paper can show the difference for with or without using the VAST-27M dataset. This can help us better understand the quality of VAST-27M.\n\n3, How to select a subset of 27M videos from HD_VILA_100M? Please elaborate the details why the paper selected a subset from HD_VILA_100M.\n\n4, In Line 60, why does the cross-attention layer only involve the text modality? Do we have cross-attention between video and audio?\n\n5, The paper seems to use BLIP for vision captioning. Why not use BLIP-2, which can generate better captions?\n\n6, The overall design of the learning objectives is very similar to BLIP, which also uses three losses, i.e., Image-Text Contrastive Loss (ITC), Image-Text Matching Loss (ITM), and Language Modeling Loss (LM). The novelty of the proposed learning framework seems to be very limited.\n\n7, All the captions from VAST-27M are generated by models. Does the paper look into the bias of the used models and their generated captions? Will they limit the quality of the dataset? \n\n8, A lot of details in the paper are similar to reference [7]. It is unclear what is the real contribution of the current submission. Many sections are very similar to [7]. I recommend clarifying the difference with [7], e.g., what are the contributions from the current submission, what has been done in [7].\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a large-scale omni-modality video caption dataset based on an existing video-text dataset, namely VAST-27M. Specifically, the authors utilized the VL models to generate captions and LLM to generate omni-captions for videos. Then a vision-audio-subtitle-text model is trained for a range of video tasks and establishes SOTA in several benchmarks.\n\nSTRENGTHS:\nThe paper is clearly written and easy to read. \nThe idea of combining text, video, and audio in one caption is interesting and novel. I believe this will contribute to the multimodal learning community. \nThe performances over a wide range of video-language-audio tasks are strong.\n\nWEAKNESSES:\n1. Despite the dataset contribution, the technical contribution of the pre-training model is somewhat weak and lacks novelty.\n2. I am confused by some details in the approach. (1) In Lines 185-186, why do you need to feed the caption tokens again instead of using the textual embeddings from the text encoder above? (2) In OM-VCG, do you use the masking strategy like BERT or generative training like GPT? Based on my understanding, you are adopting the former one, for which I think the name of “video caption generation loss” is improper, masking language modeling (MLM) may be better.\n3. Although the experimental results are very strong, I am unsure where the performance gain comes from. Several points that might be attributed to this are not discussed. (1) Combination of different datasets. The model combines several different datasets, including VAST-27M while the dataset scale is not compared with SOTA methods. (2) Strong backbones. As indicated in Line 213, EVAClip-ViT-G is adopted as a visual backbone which is different from SOTA models, which makes the comparison in Table 3 unfair. (3) More modalities. Although Table 7 provides an extensive comparison of using different modalities, there are no results that could be directly compared in Table 3 for a fair comparison. \n4. Some details are missing. (1) How do you select the 27M video clips from 100M clips in HD-VILA-100M? What is the selection standard? (2) Do you use audio in the HD-VILA-100M dataset for audio captioning and as input for the audio encoder? It is not clearly explained in the paper. (3) How do you sample frames in videos and how do you deal with video embedding from frame features in ViT (i.e., how to get f_v)? By Meanpooling?\n5. Lack of discussion on related works. As an extension of HD-VILA, CLIP-ViP used an off-the-shelf image captioner to generate captions for the HD-VILA-100M dataset. The authors should also compare that one and your advantage over CLIP-ViP. It seems that only audio information is not used in the CLIP-ViP model.\n6. It would be helpful to show some examples in the dataset for illustration.\n7. Some typos: Row 1 in Table 5.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nThe authors have addressed the scale limitation of the current dataset. It would be great if they could share more insights into the potential limitations of using their dataset and model, such as biases it could contain.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}