VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Authors: Sihan Chen
Handong Li
Qunbo Wang
Zijia Zhao
Mingzhen Sun
Xinxin Zhu
Jing Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Cross-Modality Foundation Model
Cross-Modality Pretraining Dataset

Abstract: Vision and text have been fully explored  in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including  vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks.

VAST: A Vision-Audioo-Subiille---eee Omni-Mooality Foundation Model and Dataset
Sihan Chen Handong Li Qunbo Wang", Zijia Zhao?!, Mingzhen Sun Xinxin Zhu?, Jing Liu
School of Artificial Intelligence, University of Chinese Academy of Sciences
2 Institute Automation, Chinese Academy of Science {sihan. chen, xinxin. jliu] (aluu)mlprriie .i..prri..a. ac en, (lihandong2023, qunbo. sang, zhaozijian a22221, summingzhen22200)iaa.... cn
Abstract
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open- domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off the--hh-hhe the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new ofate-of-the-ar results on various cross modality benchmarks. Code, model and dataset will be released at btttp://giihhe com/TXH mercury/VAST.
Introduction
vast number of videos recorded and uploaded to social media platforms every day, making the understanding of video content become a critical area of research in artificial intelligence. Simultaneously, the text serves the most direct and efficient means of information propagation in our daily lives. Consequently, video-text cross-modality learning is crucial for Al systems to interact with humans, assisting them in understanding video content (video captioning), searching for desired videos (text-to-video retrieval), or answering questions about videos (video QA). With the advent of unsupervised pre-training techniques [1: 2] and large-scale video-text corpora [3; 4], video-text pre- pre-training models have made rapid advancements and achieved relatively impressive performances on the aforementioned tasks. However, we contend that current pre- gre-training models are far from perfect, as most of them are restricted to establishing connections solely between the text and visual content of videos, without incorporating other modality tracks such as audio and subtitles. Specifically, environmental audio can provide additional contexts overlapping with the visual information to reduce ambiguity and increase the prediction confidence of models, or
"Equal Contribution. 'Coresponning author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Pretraining
Pretraining
Modet: Freoee, CaCito..
Audio
Medei: Wavt ape
Cerpus: Webvid1OM.CCIMM
Corpus: WorCops
VS-T Pretraining Modeli Taiv3. CWMVT...
VA-T Pretraining
Modet: AVT MERI (T-RoA OI...
Subritte
Corpus: HowTolDOM, HD. VILA 100M.......
Audio
Corpus: HoNToLOOM, VALOR-1M.
Vision
Audio
Omni-Modaity Video (CNN)
Ommi-Moddllty VAS-T Protraining VAST-27M (Ours)
Ommi-Madalty Caption (OMC)
Subtitle
Figure Illustration of the difference between conventional cross- mros--ty pretraining and the proposed omni-modality pretraining. Thanks to the proposed omni-modality video caption corpus VAST-27M, the VAST foundation model can perceive videos from multiple information sources, including vision, audio, and subtitles, and enhance the connections between ommi-mooallin (OMV) and omni-modality captions (OMC) through large-scale pretraining. A, V, S, and T represent audio, vision, subtitle, and text, respectively. AC, VC, and AVC are abbreviations for audio, vision, and audiovisual captions.
videos
complement the visual information to supply more multi-modal cues. Human speech or transcriptions (subtitles) also contain valuable information about conversations, news, or instructional procedures, which can be jointly modeled to enhance the comprehensive understanding of videos. Given that both audio and subtitles are informative for video comprehension, it is necessary integrate them into a more advanced omni-modality foundational model, as depicted in Figure 1.
The primary challenge lies in the absence of appropriate training corpus. To train such foun- dational model, we require an omni- modality video caption corpus in which captions exhibit cor- respondence with vision, audio, and subtitles simultancously. Unfortunately. as shown in Table current video-text corpora either utilize raw subtitles as captions [4; 5; 6], contain only vision captions (alt-texts) [3], or possess audiovisual captions but limited in scale [7]. On the other hand, manual annotation of an omni-modality corpus is not feasible due to the exorbitant cost associated with the high demand for deseriptions.
To address this issue, we propose a two-stage automatic pipeline to generate omni-modality captions for video clips from a large-scale open-domain video corpus (HD (HD_VILA 100M [5]). Specifically,
as depicted in Figure 2, in the first stage, we train separate vision and audio captioners on publicly available vision and audio caption corpora, which are then utilized to generate high-quality single- modality objective descriptions. In the second stage, we employ Vicuna- 13b [8], an off off-the-sheif Large Language Model (LLM), as a zero-shot omni-modality captioner. We feed the LLM with the generated single-modalle captions, subtitles, and instructional prompts to encourage LLM to integrate different modality information and summarize them into single long sentence, forming an omni-modality caption. Through this process, we create VAST-27M, a dataset that consists of 27 million video clips, each paired with 11 captions (including 5 vision, 5 audio, and omni-modality caption). Given its comprehensive caption types, VAST-27M can contribute various research communities, including vision-text, audio-text, and omni-modality pretraining. Extensive experiments have demonstrated that VAST-27M surpasses current video-text, audio-text, and audiovisual-texx corpora by a significant margin.
Based on VAST-27M. we train a named VAST. As shown in Figure 2. VAST consists of three single modality encoders while text encoder can fulfill mods-ity fusion through cross-attention layers. VAST is trained with three objectives including OM-VCC, OM-VCM, and OM-VCG (detailed in section 4.2) to enhance omni- modality understanding and generation capabilities. As shown in Table 1, compared to existing cross-modality pre-training models, VAST trained on omni-modality video caption corpus and capable of perceiving and processing all four modalities and supports wide range of downstream tasks, with various modalities (vision-text, audio-text, and multi-modal video-text), and multiple types (retrieval, captioning, and QA).
omni-modality foundational model

---

Table 1:
Comparisons between current mross-modality pretraining models and the proposed VAST. VAST is the first foundation model capable of perceiving four modalities and generalizing to various types of downstream tasks. V, A, S, and T represent vision, audio, subtitle, and text, respectively.
Task Type
Task Modality
Model
Retrieval
Discriminative
Generative
V-T
A-T
VA-T
VS-T
VAS-T
GIT [19]
VALUE [20]
V
v
VALOR [7]
V
v
VAST
V
V
v
Our contributions are summarized as follows: (i) We introduce VAST-27M, the first large-scale omni-modality video caption dataset automatically generated by trained single modality captioners and Large Language Model; (ii) We train the first ission-addo-suubitleeee
foundation model VAST which fulfills omni-modality perception and understanding; (iii) Extensive experiments verify the effectiveness of VAST and it outperforms state-of-the-ar a diverse range of cross-modality benchmarks.
2 Related Work
2.1 Cross-Modality Pretraining Corpus
Video-Text Pretraining Corpus. In the early stages, most video-text models were trained on the large-scale
dataset [4]. This dataset consists of 136 million video clips from 1.22 million instructional YouTube videos, and employs automatically generated subtitles from ASR transcription tools corresponding clip captions. Later, following similar schemes for collecting narrative iidoss-ubbitle corpora, Zellers et al. proposed the YT-Temporal-18061 corpus [6]. which contains 180 million clips from 6 million YouTube videos. To overcome the constraint of the instructional domain, Xue et al. assembled the open-domain, high-resolution HD_ VILA VILA_100M corpus [5), consisting of 100 million clips from 3.3 million YouTube videos. However, while labeling subtitles is more cost-effective than annotating captions, subtitles often lack direct relevance to visual content, resulting in weak video-text correlations in model learning. Inspired by the collection schemes of the Conceptual Captions datasets (CC) [9; 10J. Bain et al. compiled the Web Vid..id..5 and Vid10M datasets [3], utilizing alt-texts as video captions, surpassing the aforementioned datasets that use subtitles as captions. However, alt-texts, although more related to video content than subtitles, differ in style from standard captions and still contain noise. Additionally, alt-texts typically describe visual content only, without incorporating other modalities. Taking a step further beyond vision. Chen et al. created the VALOR-IM 1M dataset [7], which contains I million video clips from AudioSet [1 l] paired with annotated audiovisual captions. However, subtitle contents are not reflected in these captions, and the dataset's scale limited and challenging to expand due to the expensive cost of manual annotation. In comparison to the aforementioned training corpora, our VAST-27M is the first high-quality, large-scall scale moni-modality video caption dataset whose captions encompass all modalities of video including vision, audio, and subtitles. In addition, its expansion is more coft-effective due to the automated generation pipeline.
Audio-Text Pretraining Corpus. In contrast to vision-text pretraining, audio-text pretraining has progressed relatively slowly due to limitations in training corpora, both in terms of quantity and scale. Human-labeled datasets such as Clotho [12], AudioCaps [13]. MCAS [14], and AudioCaption [15] all contain fewer than 50,000 audio clips, which fall far short of the requirements for large-scale pretraining. Wu et al. compiled LAION-Audio- 660K [16] by crawling audios and corresponding alt-texts from multiple sources, with a majority of audios sourced from Freesound [17]. Mei et al. introduced WavCaps [18], which comprises 403,000 audios and descriptions collected from various sources. They utilized ChatGPT to filter and rewrite the raw descriptions into caption-like sentences. However, the quality of the captions is heavily influenced by the quality of the raw descriptions. In contrast, we generates audio captions through trained high-quality caption model, and VAST-27M is nearly two orders of magnitude larger than AAION-Audio-6300 or WavCaps.
3

---

Table 2: Comparisons between current video-text and audio-text pretraining corpora and the proposed VAST-27M. VAST-27M the first large-scale omni-modality video-caption corpus containing various types of captions generated automatically. S. VC, AC, AVC, and OMC stand for subtitle, vision, audio, audiovisual, and onni-modality captions, respectively.
Dataset
Domain
#Clips
#Captions
Text Source
S
VC AC
AVC
OMC
HowTo100M [4]
Instructional
136M
136M
ASR transcription
YT Temporal 180M [6]
Instructional
180M
180M
ASR transcription
HD_ VII.A 100M [5]
Open
103M
103M
ASR transcription
WebVid-2.5M [3]
Open
2.5M
2.5M
Ali-texts
WebVid- [3]
Open
10M
10M
Alt-texts
VALOR-IM [7]
Open
IM
IM
Manual
JAION-Audio-600 [16]
Open
634K
634K
Alt-texts
WayCaps [18]
Open
403K
403K
Generated
VAST-27M (Ours)
Open
27M
297M
Generated
2.2
Multi-Modality Learning
There have been early methods exploring multi-modality tracks for video-text understanding. For instance, MMT [21] leverages seven experts enhance text-to-video retrieval, while SMPFF [22] introduces audio for video captioning. In the context of video-text pretraining, there are works that jointly model subtitles or audio together with vision and text. Notable examples of incorporating subtitles include UniVL [23]. CoMVT [24], and VALUE [20]. However, the subtitle-text correlations established in these models are implicit and weak, achieved through masked prediction [23] or next utterance prediction [24] that uses raw subtitles as prediction targets, rather than abstract text. This approach introduces inconsistency between the pretraining and fine-tuning stages. In contrast, VAST fully harnesses the generalization capabilities of Large Language Models to extract the most crucial information from subtitles into language descriptions. On the other hand, notable examples of incorporating audio include AVLNet [25], MERLOT Reserve [26], i-Code [27], and VALOR [7]. However, AVLNet, MERLOT Reserve, and i-Code focus on learning audio-subtitle relations rather than audio-text, limiting their generalization to tasks such as text-to-audio retrieval and audio captioning. While VAL.OR jointly models audio, vision, and text, it primarily targets perceiving environmental sounds and pays less attention to human speech. In comparison to the aforementioned methods, VAST is the first omni- modalityy foundation model capable of perceiving four modalities, connecting vision, audio, and subtitle to caption.
3 Dataset
3.1 Data Collection of VAST-27M
Vision Captioner Training. To establish the general correspondence between objects and visual concepts, we draw inspiration from BLIP [28] and commence by training a vision caption model on large-scale image-text corpora, including CC4M, CC12M, and a randomly selected set of 100M image-text pairs from LAION-400M [29]. Subsequently, we fine-tune the model on a combination of manually labeled image and video caption datasets, such as MSCOCO [30], VATEX [31]. MSRVTT [32], and MSVD [33]. Through this process, we achieve a high-quality captioner capable of perceiving both static objects and dynamic actions, thus generating smooth and accurate captions.
For the audio captioning task, we train a dedicated audio caption model using a combination of large-scale audio-text corpora, namely VALOR- IM and WayCaps datasets. Unlike its vision counterpart, we do not employ a second-sttee stage fine-tuning due to the limited scale of downstream audio caption datasets, which could lead to overfitting to narrow range of audio concepts. Additionally, VALOR-IM is a numan-annotaan corpus that provides a certain level of assurance regarding the accuracy of the generated captions.
Audio Captioner Training.
Clip Selection. Due to the consideration about time and money cost of downloading, storage and computation, we select 27M video clips from the HD_ DD_V VILA_100M dataset instead of using all of them. The selection rules are introduced as follows: 1) Clips whose length are shorter than seconds or longer than 30 seconds are dropped. 2) Clips which have missing modalities of vision, audio or subtitle are dropped. 3) Clips are evenly chosen from the original 3.3M long-form videos in HD_ VIDA 100M.

---

VAST-27M
Generation Pipeline
persoon surting
middle of the wawes.
Vision Captioner
tatats wave the occan,
3. surfer on a suriboard rides in ware, then jumps up and starts surting.
surfer rides ware,, jumping and surting the ocsan baskground, they compete for the top apot, with a close-up of the wurter finishing looking
Audio
Someone playing men speaking
Sea waves and people talkinng can heard.
Ommi-Modality Captioner (LLM)
rides through
eeplosive clossoud look better""
"Human: About one video, here vision captions: 9 audio captions: O. and one speech subeite:: You need understend and encode them into sentence. Vision and audio captions maybe redundent, sodetntttt, aummeriza them before encode with speach subcitic. Do not simply concatenate them together. The weights of vidrolausio/spechh are equaled. Assisiatt:
VAST Training Framework
repostataaioo
Concalmaation
OM-VCM
CM-VCG
Encoder
Encoder
Text
Encoder
Vision
Audio
Text enooder
Text encader
Viston
Subritie
OMC
Vision
Audio
Subttlle
OMC
Figure 2: Illustration of the caption generation process of VAST-27M (top) and training framework of VAST (bottom). The vision and audio captioners generate captions based on the input video clip, and the Omni-Modality Captioner (Vicuna-13b) integrates them along with the raw subtitle and instructional prompts, to generate the omni-modality caption. The VAST model consists of three encoders and is trained under three objectives including OM-VCC, OM-VCM. and OM-VCG.
Caption Generation. For each video clip of VAST-27M, we employ the trained vision and audio captioners to generate 5 captions each, using Top-K sampling approach with K=10. Subsequently, we utilize the off-the-shelf Vicuna-133 13b [8] model as the omni-modality captioner. Vicuna- 13b an open-source auto-regred retterrereee Large Language Model (LLM) based the Transformer architecture, trained through fine-tuning of LLaMA [34] on user-shared conversations collected from ShareGPT. For each video clip, randomly select vision captions and 3 audio captions, and feed them, along with the raw subtitle and designed instructional prompts, into the LLM. As depicted Figure 2, the LLM generates omni-modality captions that effectively deseribe the visual, audio, and subtitle contents, while adhering to natural human captioning style. More examples of VAST-27M dataset can be found in Appendix.
3.2 Statistics of VAST-27M
VAST-27M consists of a total of 27M video clips sampled from the large-scale HD 1O0M corpus. The dataset covers 15+ categories, including music, gaming. education, entertainment, animals, and more, opposed to being limited to instructional domains like HowT Compared to other commonly used mross-modality training corpora, as shown in Table 2, VAST-27M contains a larger number of captions (in terms of both modalities and quantity). The average lengths of vision, audio, and omni-modality captions in VAST-27M are 12.5, 7.2, and 32.4, respectively.
100M
4 Approach
4.1
Basic Framework
As shown in Figure 2, VAST employs fully end-to-end Transformer architecture, comprising vision encoder (ViT [35]), an audio encoder (BEATs [36]), and text encoder (BERT [2]). This framework can accommodate multi-modis inputs such as images, videos, audios, subtitles, and captions. The text encoder responsible for encoding single-modality captions or subtitles, as well as performing multi-modal encoding/decoding through cross-attention layers. The omni-modality captions and subtitles are tokenized using the WordPiece tokenizer [37] and then fed into the text encoder obtain the output features fome and fe, respectively. The vision encoder takes raw images
5

---

or sparsely sampled video frames as input and produces the output feature f.. For audio clips, they are first divided into multiple lo-second-long clips, padded with zeros, converted 64-dimensional log Mel filterbank spectrograms using a 25ms Hamming window, and then fed into the audio encoder to obtain the output feature fa- The global representations (CCLS] token feature) of these features are denoted as Someg. Jeg Seg. and fag. respectively.
4.2 Pretraining Objectives
Building upon conventional vision-text pretraining objectives, VAST employs the following three pretraining objectives:
Omni-Mouality Video-Caption Contrastive Loss (OM-VCC) The global moni-mooallity representa- tions of video clips, denoted as fomvg. are obtained by concatenating fsg. feg. and fag- Subsequently, Somug and Jomey are projected into the same semantic space using two linear layers and are then nor- malized. The contrastive loss is employed to regularize the feature distance between omni-modality video (OMV) and caption (OMC). The contrastive loss is defined follows, where sim(.) represents the dot product of fomeg and fomug. and B and T denote the batch size and learnable parameter, respectively.
exp(r sim((s, G:)) log Ej=1 exp(r sim(us,
1 2
exp(T sim((s (i))
LOM-VCC
log Dj=1 exp(r sim(yy, c:))
Omni-Modality Video-Caption Matching Loss (OM-VCM). This loss encourages the model to infer whether pair of OMV and OMC is matched or not. Specifically, caption tokens are fed into the text encoder again, this time with cross-attention layers activated to attend to the condition features fomv which is obtained by concatenating those unpooled features fa. fu. and fa along the sequential dimension. Before concatenation, three independent linear layers are applied to adjust their hidden size to the same value. The output feature of the text encoder is then fed into two-layer MLP to make binary predictions. To create informative negative pairs, we adopt a hard negative mining strategy following [38]. The loss function is formulated as follows, where y = 1 if OMV and OMC are matched, and 0 otherwise.
LOM-VCM ((r,a)((V.c) (y log Peem + (1 y) log (1 Poem)]
(2)
Omni-Modality Video Caption Generation Loss (OM-VCG). This loss employs conditional causal masked language modeling to enhance the model's ability to generate omni-modality captions. Specifically, 60% of the tokens in OMC are masked at the input the text encoder. Cross-attention layers are activated and Some is used as condition features as in OM-VCM. The self-attention layers in the text encoder utilize a single- single-directional causal attention mask to prevent information leakage, and the masked tokens are reconstructed at the output of the text encoder using BERT's s vanilla prediction layer. The loss is defined as follows, where en and cam denote the masked tokens and the tokens before them, respectively.
Lom-vcc = - (V1 log P (cm (cm. v)
(3)
The overall loss Lom is the sum of the three losses, with equal weights assigned for simplicity.
Lom = LoM-voc LoM-VCM + LoM-voG
4.3
Modality Grouping
While VAST establishes omni-modalit, video-caption correspondence during pretraining, it is im- portant to address the potential absence of modalities in downstream benchmarks and practical applications, due to that inconsistencies between the modalities used in pretraining and adaptation can have negative effects. Inspired by the modality grouping strategy proposed by VALOR [7]. we uniformly model the relations of V-T, A-T, VA-T, VS-T, and VAS-T (previously introduced as LoM). Specifically, vision and audio captions are used in V-T and A-T modeling, respectively. while omni-modality captions are employed in VA-T, VS-T, and VAS-T modeling. The final loss is formulated as follows:
L Lom + Lv-T + CA-T LVA-T + Lvs-T
(5)

---

Table 3: Performance comparison between VAST and state-of-the-ar methods. VAST has achieved 22 new SOTA results. Recall 1, CIDEr, and Ace are used as evaluation metrics for retrieval, captioning, and QA tasks, respectively. For captioning tasks, results marked with *e' means that SCST finetuning [41] is employed. "MM benchmark' means that either audio or subtitle tracks are available in those benchmarks, and SOTA_ Vis and SOTA _MM are the best methods with vision track used only or employing multi-modal tracks. More detailed benchmark introductions, model references and comparison results can be found in Appendix.
Vision-Text
T-to-V Retrieval
Captioning COCO[30]
QA
benchmark
COCO
Flickr[42]
[lickr(ZS)(2]]
TGIF[43]
MSVD[44]
VQAv2[4S]
SOTA
68.3[46]
90.3[47]
89.7[46]
154.9"[88]
78.7[7]
60.2[49]
84.3[50]
VAST
68.0
91.0(+0.7)
90.4(+0.7)
149.0*
(9.1(-0.4)
60.2
80.2
Audio-Text
T-to-A Retrieval
Captioning
benchmark
[lothov[[12]
C1othov2[12]
AudioCaps(13)
Clothov
[1othov2[12]
AudioCaps(
SOTA
17.5[7]
21.5[18]
42.2[18]
42.3[7]
48.8[18]
78.7[18]
VAST
25.1(+7.6)
26.9(+5.4)
52.0(+9.8)
50.7(+8.4)
51.9(+3.1)
78.2
MM Video-Text
T-toV Retrieval
benchmark
MSRVTT[32]
YouCook2[51]
VALOR-32K[7]
VATEX[31]
DiDeMo[52]
ANET[53]
SOTA Vis SOTA MM
58.8[54]
33.7[55]
43.4[56]
71J[577
70.4[54]
66.8[54]
54.417]
31.3(20]
73.2[7]
76.9[7]
57.6[7]
63.4[7]
VAST
63.9(+5.1)
50.4(+16.7)
80.0(+6.8)
83.0(+6.1)
72.0(+1.6)
70.5(+3.7)
MM Video-Text
Captioning
QA
benchmark
35
YoaCrol2(51)
VAI VALOR-32K[T]
VATEX[31]
TVQ[5]]
MSRVTT(44]
MUSIC[19]
ANET[
SOTA_ Vis
75.9[19]
131.2[19]
27.3|611
91.5*[19]
66.1
19.5[19]
17.9[54]
SOTA MM
74.0
190.0[58]
61.5[7]
97.
66.0/62]
49.2[7]
78.9(7]
48 48.6[7]
78.0(42.1)
198.8(-8.8)
62.0(40.5)
99.5%(43.7)
74.1(-8.0)
50.1(-0.6)
80.7(-18)
50.4(+1.8)
5 Experiments
5.1
Implementation Details
VAST is trained using the PyTorch framework on 64 Tesla V100 cards. The vision, audio, and text encoders are initialized from EVAClip-ViTI ViT-G [39], BEAT',, and BERT-B, respectively, resulting in total parameter size of 1.3B. The training is conducted on combination corpus consisting of VAST- 27M, VALOR-IM, WavCaps, CC14M, and I 110M randomly sampled pairs from LAION-400M, for total 200K training steps. At each training step, one corpus is sampled for training. In addition, the raw captions of CC14M and LAION are replaced with captions generated by the trained vision captioner. The initial learning rate is set to le-4, and a linear decay schedule is used. The batch size is set to 1024. During the pertaining stage, one video frame and two 10s long audio clips randomly sampled for each video clip. For ablation studies, CiT--ViT-B [40] is used as the vision encoder, with its parameters frozen increase efficiency. Further details such the mixture ratio of the pretraining dataset and the downstream finetuning configurations can be found in the Appendix.
Regarding the adaptation to downstream tasks, for retrieval tasks, all candidates are ranked using VCC, and then the Top-50 candidates are reranked using VCM. For captioning tasks, beam search with a beam size of 3 is employed. For QA tasks, they are formulated as open-ended generative problems, where questions serve as prefixes, and answers are predicted without any constraints. For SOTA comparison and ablation studies in the main paper. Recall @ 1, CIDEr, and Acc are used as evaluation metries for retrieval, captioning, and QA tasks, respectively.
5.2 Comparison to State-of-the-Art Models
We present a comparison of the VAST foundation model with state-of-the-art models across var- ious vision-text, audio-text, and multi-modal video-text benchmarks. The corresponding results are summarized in Table Although the primary focus of VAST lies in enhancing omni-modality understanding and generation capabilities, it also demonstrates remarkable performance in image-text,
7

---

Table 4: Comparisons of V-T quality in VAST-27M and open-sourced vision-text corpora.
MSVD
MSRVTT
Model
PT Data
RET
CAP
QA
RET
CAP
QA
(a)
23.7
101.2
35.4
58.9
43.6
(b) WebVid2.5M
43.1
139.2
53.0
42.4
63.7
45.5
(c) CC14M
46.1
139.2
53.9
45.2
66.4
46.3
(d)
LATON- 150M
42.4
139.3
53.2
44.9
66.9
(e)
VALOR-IM
42.3
136.7
52.6
42.8
65.3
45.5
(f) VAST-27M (subtitle)
37.3
124.5
40.3
40.3
61.9
45.1
(g) VAST-27M (omc)
46.8
143.2
54.3
47.5
67.4
46.7
(h) VAST-27M (vision caption)
47.7
149.6
55.3
49.1
68.9
46.8
Table 5: Comparisons of A-T quality between VAST-27M and open-sourced audio-text corpora.
Clothov2
AudioCaps
Model PT Data
RET
CAP
RET
CAP
(a)
17.2
42.1
41.1
70.4
(b) VALOR-IM
21.0
47.3
47.4
74.8
(c) WavCaps
22.0
47.2
43.2
73.0
(d) VAST-27M (audio caption)
23.1
48.9
47.4
76.9
vision-only video-text, and audio-text benchmarks. Specifically, VAST surpasses BEİT-3 [47] and BLIP-2 [46] on the text-to- toxt-to-imag retrieval benchmark of Flickr30K under finetuning and zero-shot settings, respectively, establishing new state-of-thh-arr results TGIF-QA and MSVD-QA bench- marks. Furthermore, VAST exhibits exceptional performance in audio-text benchmarks, achieving five new state-of-the-art results with significant improvements, due to the abundant generated captions in the VAST-27M corpus.
In the context of multi-modal video-text benchmarks, they can be classified into three categories: subtitle-orie oriente (YouCook2,, TVC) benchmarks that require comprehensive understanding of subti- tles for reasoning, audio-oriented (VALOR-32K) benchmarks that necessitates careful attention to audio cues, and even-oriented benchmarks (others) where both audio and subtitles provide supportive roles to vision. Our results demonstrate that VAST excels across all types of benchmarks, outper- forming previous both vision-only and multi-modal state-of-the- art models. Notably, we surpass the GIT2 [19] foundation model, specialized captioning tasks, by 2.1, 67.6, 5.0, and 8.0 CIDEr points on MSRVTT, YouCook2, VATEX, and TVC captioning benchmarks, respectively, while utilizing only 22.5% its parameters and 3.4% its training data size, Furthermore, we achieve significant margins over the vision-audio-text foundation model VALOR [7] benchmarks. More detailed comparison results can be found in Appendix.
5.3 Comparison to Open-Source Cross-Modality Training Corpus
In this subsection, we quantitatively compare the quality of the proposed VAST-27M to current open-source video, audio, and audiovisual pretraining corpora. achieve this by training models on these corpora and fine-tuning them different types of downstream tasks, including video retrieval (RET), captioning (CAP), and question answering (QA).
V-T Quality. We train multiple V-T models on different corpora and fine-tune them on the MSVD and MSRVTT datasets. is important both pretraining and fine-tuning, and all models are trained for 50K steps. As shown in Table 4, the model trained with vision captions in VAST-27M (h) achieves the best results on all six benchmarks, outperforming the baselines and models trained with other corpora by a significant margin. In contrast, model (f), which treats subtitles as captions and can be viewed as raw HD VILA_ 100M dataset, performs the worst due to the weak vision-text relations. Model trained on VAST-27M (g) outperforms other corpora but is still inferior to model (h), as the multi-modal captions introduce noise when only vision contents fed into the model.
that only the visual contents in videos
utilized for

---

Table 6: Comparisons of OMV-OMC quality between VAST-27M and VALOR- 1M.
MSRVTT
YouCook2
VALOR-32K
Model
PT Data
RET
CAP
QA
RET
CAP
RET
CAP
(a)
37.6
60.3
44.6
32.6
120.9
37.8
40.8
(b)
VALOR-IM
45.4
67.8
46.4
29.3
120.1
70.8
53,9
VAST-27M (ome w/o LLM)
44.4
71.4
47.7
36.9
159.0
49.9
48.1
VAST-27M (omc)
53.0
71.7
47.9
44.0
187.5
58.8
48.5
Table 7: Downstream performances of models trained on VAST-27M with different modalities used during training and finetuning.
Model
MSRVTT
YouCook22
VALOR-32K
RET
CAP
QA
RET
CAP
RET
CAP
(8)
V
35.4
58.7
43.7
6.2
77.0
29.1
36.5
37.3(-1.9)
61.1(+24)
44.6(+0.9)
7.14099)
74.1(229)
38.44-9.)
40.2(+3.6)
(6)
V4S
35.8(-0.4)
59.1(+0.4)
43.6(-0.1)
31.2(+25.0)
(19.1(+12.1)
29.66+0.5)
38, (++5))
(d)
V+A+S
37.66-2.2)
60.3(+1.6)
44.6(+0.9)
32.6(+26/4)
120.9++13.9)
37.86+8.7)
10.8(+1.2)
V
V
47.5
67.4
46.7
14.0
93.6
56.5
45.0
(0)
V+A+8
49.2(-1.7)
70,6(4322
47.3(+0.6)
30.4(+1644
130 1436.5)
58.50-2.0)
44.94-011
(g)
V+A
V+A
53.8(-6.3)
70.9(+3.5)
47.4k-0.7)
30.3(+16.3)
130.7(437.))
62.6(+6.1)
46.3(+1.)
(0)
V+S
V+8
49.3(-2.0)
68.9(+155
47.40-0.7)
42.6(228.6)
186.5
48.20-8.)
43.60-44)
53.0K-5.5)
7L7(+443)
47.9(=1.2)
44.0(+30.0)
187.5(+93.9)
58.86+2.3)
48.5(+3.5)
0
VIAR & V4A-S
VIAIS
58.3(-7.8)
71.7(14.3)
48.2(+1.5)
45.9(131.9)
188.0(194.))
62.2(+5.7)
48.4(13.4)
A-T Quality. We train multiple A-T models on different corpora and fine-tune them on audio-text benchmarks, including Clothov2 and AudioCaps. We observe that VALOR-IM and WavCaps are prone to overfitting, so we adjust the training iterations to obtain the best results for models trained on each corpus instead of training for the same number of iterations. As shown in Table 5, model (d) surpasses the baseline and models trained on both VALOR-IM and WavCaps on all four benchmarks.
OMV-OMC Quality. We further investigate the correspondence quality between OMV and OMC in the VAST-27M dataset and compare it primarily the VALOR-IM dataset. All models undergo 50K training steps with Lom as the objective, utilizing all modalities in both pretraining and finetuning. The results are presented in Table 6. From the table, we observe that omni-modalle pretraining on VAST-27M significantly improves performance across all 7 benchmarks compared to the baseline. Additionally, VAST-27M outperforms VALOR-IM
benchmarks the MSRVTT and YouCook2 datasets, but lags behind on the VALOR-32K benchmarks due to the consistent video distribution and caption style between VALOR-IM and VALOR-32K. As an additional experiment, we train model (c) on VAST-27M but replace the omni-modalig captions generated by LLM with simple concatenation of vision, audio captions, and subtitles. Model (d) outperforms model (c) on all benchmarks, showcasing the necessity and effectiveness of leveraging the powerful capabilities of LLM to integrate single-modality captions into omni-modality ones.
5.4 Ablation Study
We conduct comprehensive ablation studies on pretraining and finetuning models using different modalities for video representations to demonstrate the strength and necessity of omni-modality foundation model pretraining. Models are evaluated the MSRVTT (open-domain), YouCook2 (subritle-impoottant), and VALOR-32K (audio-important) benchmarks. The results are presented in Table when no pretraining applied, models trained with audio and subtitles incorporated (d) can improve the vision-only baseline on all benchmarks. When omni-modality pretraining is applied, the improvement becomes more evidently on most benchmarks, which can be reflected by the comparison between the green values in model (f) and model (d), demonstrating the effectiveness of proposed method. In addition, model (i) outperforms model (f) on all benchmarks, indicating that, similar to the V-T capability, the OMV-OMC capability also benefits from large-scale pretraining. It worth noting that model (h) and (f) exhibit inferior results on VALOR-322 -333 benchmarks compared to model (g) and even the vision baseline (e). This degradation can be attributed to the inconsistency between pretraining and finetuning, where all clips have subtitles VAST-27M while most videos in AALOR-32K lack subtitles. When the modality grouping strategy is applied, model (j) demonstrates

---

good generalization across all types of tasks, as the correlations between text and every modality group have been explored during pretraining.
Conclusion, Broader Impact and Limitation
In this paper, introduce the VAST-27M corpus, large-scale omni-modalee video caption dataset, aimed at advancing research in multi multi-modality pretraining. Each video clip in the dataset accom- panied by automatically generated vision and audio captions, well that integrates information from vision, audio, and subtitles using a pre-existing Large Language Model (LLM). We also train unified foundation model called VAST, capable of understanding and connecting the various modalities in videos and captions. Through extensive evaluations, VAST demonstrates its effectiveness in wide range of vision-text, audio-text, and multi-modal video-text tasks, including retrieval, captioning, and question answering, surpassing existing state-of-the-art methods public cross-modality benchmarks.
omni-modality caption
The advancements in multi-modal understanding and generation of video content can have a sig- nificant impact various domains such entertainment, education, security, transportation, and healthcare, among others. The development omni modilmty foundation models, like VAST-27M and VAST, can contribute the progress and practical applications in these areas. However, it is important to acknowledge the limitations of our method. For instance, there is a need for more diverse and larger-scale omni-modality corpora beyond VAST-27M. Furthermore, although VAST already supports a wide range of downstream tasks, the integration of LLM necessary to further enhance its generalization capabilities. Additionally, due the reason that the data collection process of VAST-27M have used LLM, and the training process of video and audio captioners have utilized some open-sourced mross-modality corpus, VAST-27M dataset and VAST model may suffer the same bias of those datasets and models.
10

---

Acknowledgments
This work was supported by the National Key Research and Development Program of China (No. 20222D0118801,, National Natural Science Foundation of China (U21B2043, 62102416, 62206279).
References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Noclakantan, P. Shyam, G. Sastry,
A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[2] J. Devlin, M.-W. Chang. K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805. 2018.
[3] M. Bain, A. Nagrani, G. Varol, and A. Zisserman, "Frozen in time: A joint video and image encoder for end-to and-to-end retrieval," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2021. pp. 1728 1738.
[4] A. Micch, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivie, "Howtol00m Learning text-video embedding by watching hundred million narrated video clips," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 2630-2640.
[5] H. Xue, T. Hang. Y. Zeng, Y. Sun, B. Liu, H. Yang. J. Fu, and B. Guo, "Advancing high-resolution video-language representation with large-scale video transcriptions." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5036-5045.
[6] R. Zellers, X. Lu, J. Hessel, Y. Yu, J. S. Park, J. Cao, A. Farhadi, and Y. Choi, "Merlot: Multimodal neural script knowledge models," Advances in Neural Information Processing Systems, vol. 34, pp. 23 6344332336 651, 2021.
[7] Chen, X. He, I.. Guo, X. Zhu, W. Wang, J. Tang, and J. Liu, "Valor: lision-uddiollanuaae omni- perception pretraining model and dataset," arXiv preprint arXiv:2304.00 08345, 2023.
[8] W.-L. Chiang, Z. Li, Lin, Y. Sheng, Z. Wu, Zhang, Zheng, Zhuang, Zhuang, Gonzalez, I. Stoica, and E. Xing, "Vicuna: An open source chatbot impressing gpt-4 with 90% chatgpt quality," March 2023. [Online]. J. Available: http:://mmss..or/boog//0000
03- 30 vicuna/
[9] Changpinyo, Sharma, N. Ding. and R. Soricut, "Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts," in Pruceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021, pp. 3558-3568.
[10] P. Sharma, N. Ding, S. Goodman, and R. Soricut, "Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning." in Proceedings of the 56th Annual Meeting of the Association for Computational I Linguistics (Volume I: Long Papers), 2018, pp. 2556-2565.
[11] F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 776-780.
[12] K. Drossos, S. Lipping, and T. Virtanen, "Clotho: An audio captioning dataset," in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 336-740.
[13] C. D. Kim, B. Kim, H. Lee, and G. Kim, "Audiocaps: Generating captions for audios in the wild," in Proceedingy of the 2019 Conference the North American Chapter the Asxociation for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Pupers), 2019, pp. 119-132.
[14] I. Martin Morato and A. Mesaros, "Diversity and bias in audio captioning datasets." 2021.
[15] M. Wu, H. Dinkel, and K. Yu, "Audio caption: Listen and tell," in ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 830-834,
[16] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. audio pretraining with feature fusion and jeyword-to-caplee augmentation." in ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
and S. Dubnov, "Large-scale contrastive language-
[17] F. Font, G. Roma, and X. Serra, "Freesound technical demo." in Proceedings of the 21 ACM international conference on Multimedia, 2013, pp. 411-412.

---

[18] X. Mei, C. Meng, H. Liu. Q. Kong. T. Ko. C. Zhao, M. D. Plumbley. Y. Zou, and W. Wang, "Wavcaps: A chatgpt-assisted labelle audio captioning dataset for ludio-angguage multimodal research." arXiv preprint (xXi::3333.17999 2023.
[19] J. Wang. Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang. "Git: A generative image-to-text transformer for vision and language," arXiv preprint arXiv:2205. 14100, 2022.
[20] L. Li, J. Lei, Z. Gan, L. Yu, Y.-.. Chen, R. Pillai, Y. Cheng, L. Zhou, X. E. Wang, W. Y. Wang et al., "Value: A multi-task benchmark for videoaand-lanuue understanding evaluation," arXiv preprint arXiv:2106.0460
2021.
[21] V. Gabeur, C. Sun, K. Alahari, and C. Schmid, "Multi-modal transformer for video retrieval," in European Conference on Computer Vision. Springer, 2020, pp. 214-229.
[22] Chen, X. Zhu, D. Hao, W. Liu, J. Liu, Z. Zhao, L. Guo, and J. Liu, "Mm21 pre-training for vidco understanding challenge: Video captioning with pretraining techniques," in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 4853-4857
[23] H. Luo, L. Ji, B. Shi, H. Huang, N. Duan, T. Li, J. Li, T. Bharti, and M. Zhou, "Univl: A unified video and language pre-training model for multimodal understanding and generation," " arXiv preprint arXiv:2002.06353, 2020.
[24] P. H. Sco, A. Nagrani, and C. Schmid, "Look before you speak: Visually contextualized utterances," "
in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16 16877-16888 887.
[25] A. Rouditchenko, A. Boggust, D. Harwath, B. Chen, D. Joshi, S. Thomas, K. Audhkhasi, H. Kuehne, R. Panda, R. Feris et al., "Avlnet: Learning audio-visual language representations from instructional videos," arXiv preprint arXiv:2006.09 199, 2020.
[26] R. Zellers, J. Lu, X. Lu, Yu, Y. Zhao, M. Salchi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi, "Merlot reserve: Neural script knowledge through vision and language and sound," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16375-16338 387.
[27] Z. Yang, Y. Fang, C. Zhu, R. Pryzant, D. Chen, Y. Shi, Y. Xu, Y. Qian, M. Gao, Y.-.. Chen et al., "i-coe: An integrative and composable multimodal learning framework," " arXiv preprint 0rX1v22005.0/818, 2022.
[28] J. Li, D. Li, C. Xiong, and S. "Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation," arXiv preprint arXiv:2201. 12086, 2022.
[29] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, "Laion-400m: Open dataset of clip- filtered 400 million image-text pairs," arXiv preprint arXiv:2111.02111 14, 2021.
[30] T.-Y. Lin, M. Maire, S. Belongie, Hays, P. Perona, D. Ramanan. P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in European conference on computer vision. Springer, 2014, pp. 740-755.
[31] X. Wang, J. Wu, J. Chen, L. Li, Y.-F Wang. and W. Y. Wang, "Vatex: A large- high-quality multilingual dataset for video-and lan-uag research." in Proceedings of the IEEE/CVF International Conference Computer Vision, 2019, pp. 4581-4591.
[32] J. Xu, T. Mei. T. Yao, and Y. Rui, "Msr-vtt: A large video description dataset for bridging video and language," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.
[33] D. Chen and W. B. Dolan, "Collecting highly parallel data for paraphrase evaluation," in Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 2011, 1, pp. 190-200.
[34] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix. B. Rozière, N. Goyal, Hambro, F. Azhar et al., "Llama: Open and efficient foundation language models," arXiv preprint arXiv:2302.139) 2023.
[35] A. Dosovitskiy, L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghhnn, M. Minderer, G. Heigold, S. Gelly et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint 1rXi::2010.11929, 2020.
12

---

[36] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins. Z. Chen, and F. Wei, "Beats: Audio pre-training with acoustic tokenizers"" arXiv preprint arXiv:2212.09058, 2022.
[37] Y. Wu, M. Schuster, Z. Chen, Q. Le, M. Norouzi, W. W.Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey al., "Google's neural machine translation system: Bridging the gap between human and machine translation," arXiv preprint arXiv: (669008144 144, 2016.
[38] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, "Align before fuse: Vision and language representation learning with momentum distillation," Advances in neural information processing systems, vol. 34, pp. 9694-9705, 2021.
[39] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao, "Eva-clip: Improved training techniques for clip scale," arXiv preprint arXiv:2303. 2023.
[40] A. Radford, J. W. Kim. C. Hallacy. A. Ramesh, G. Goh, S. Agarwal, G. Sastry. A. Askell, P. Mishkin. J. Clark et al., "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021. pp. 8748-8763.
[41] S. J. Rennie, E. Marcheret, Y. Mrouch, Ross, and Goel, "Self-criticn sequence training for image captioning." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7008-7024.
[42] A. Plummer, L. Wang. C. M. Cervantes, J. C. Caicedo, Hockenmaier, and Lazebnik, "Flickr30k en- tities: Collecting region-to-phrase correspondences for richer image-to-sentence models," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2641-2649.
[43] Y. Li, Y. Song, L. Cao, J. Tetreault, L. Goldberg, A. Jaimes, and J. Luo, "Tgif: A new dataset and benchmark on animated gif description." in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4641-4650.
[44] D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang, "Video question answering via gradually refined attention over appearance and motion," in Proceedings of the 25th ACM international conference on Multimedia, 2017, pp. 1645-1653.
[45] Y. Goyal, T. Khot, D. Summers-Stay. D. Batra, and D. Parikh, "Making the in vqa matter: Elevating the role of image understanding in visual question answering." " in Proceedings the IEEE conference on computer vision and pattern recognition, 2017, pp. 6904-6913.
[46] J. Li, D. Li, S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," arXiv preprint arXiv:2301.122 125997, 2023.
[47] W. Wang. H. Bao, L. Dong, Bjorck, Z. Peng. Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, Som et al., "Image as foreign language: Beit pretraining for all vision and vision-language tasks." arXiv preprint ar0iv:2208. 10442, 2022.
[48] Wang, A. Yang, R. Men, Lin, S. Bai, Z. Li. Ma. C. Zhou, J. Zhou, and H. Yang. "Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-seyy International Conference on Machine Learning. PMLR, 2022, pp. 23 23318-23340.
learning framework," in
[49] W. Kuo, A. Piergiovanni, Kim, X. Luo, B. Caine, W. Li, A. Ogale, Zhou, A. Dai, Z. Chen et al. "Mammut: A simple architecture joint learning for multimodal tasks," arXiv preprint 2303. 2023.
[50] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer et al., "Pali: A jointly-scaled multilingual language-image model," arXiv preprint arXiv:2209.06799 2022.
[51] I.. Zhou, C. Xu, and J. J. Corso, "Towards automatic learning of procedures from web instructional videos." in Shirry-Second AAAI Conference on Artificial Intelligence, 2018.
[52] L. Anne Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell, "Localizing moments in video with natural language," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 5803-5812.
[53] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles, "Dense-captioning events in videos." in Proceedings the IEEE international conference on computer vision, 2017. pp. 706-715.
[54] K. Li, Y. Wang. Y. Li. Y. Wang, Y. He. L. Wang. and Y. Qiao, "Unmasked teacher: Towards training- efficient video foundation models." arXiv preprint 2rXiv:3303.16058 2023.
13

---

[55] D. Ko, J. Choi, H. K. Choi. K.-W. On, B. Roh, and H. J. Kim, "Meltr: Meta loss transformer for learning fine-tune video foundation models," arXiv preprint arXiv: 13009, 2023.
[56] H. Luo, I. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li, "Clipáclip: An empirical study of clip for end to end video clip retrieval and captioning." Neurocomputting, vol. 508, pp. 293-304, 2022.
[57] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang et al., "Intern- video: General video foundation models via generative and discriminative learning"" arXiv preprint arXiv22212003191 2022.
[58] J. Lei, L. Yu, T. L. Berg, and M. Bansal, "Tvr: A large-scale dataset for video-subtille moment retrieval," in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XXI 16. Springer, 2020, pp. 447-463.
[59] G. Li, Y. Wei, Y. Tian, C. Xu, J.-R. Wen, and D. Hu, "Learring to answer questions in dynamic audio-visual scenarios," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 19 108-19 118.
[60] Z. Yu, D. Xu, J. Yu. T. Yu, Z. Zhao, Y. Zhuang, and D. Tao, "Activitynet-qa: A dataset for understanding complex web videos via question answering." in Proceedings of the AAAI Conference Artificial Intelligence, vol. 33, no. 01, 2019, pp. 9127-9134.
[61] K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y. Lu, and L. Wang, "Swinbert: End-to-end transformers with sparse attention for video captioning." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. pp. 17 17949-17 17 958.
[62] M. Tang, Z. Wang, Z. Zeng, F. Rao, and D. Li, "Clip4caption++: Multi-clip for video caption." arXiv preprint arXiv:21 10..0504, 2021.
[63] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan. F. Viola, T. Green, T. Back, P. Natsev et al., "The kinetics human action video dataset." arXiv preprint arXiv: 066500 2017.
[64] S. Chen, Y. Thaa, Q. Jin, and Q. Wu, "Fine-grained video-text retrieval with hierarchical graph reasoning," " in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10638-10641
[65] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, "Movie description," International Journal of Computer Vision, vol. 123, no. 1, pp. 94 120, 2017.
[66] Y. Jang, Y. Song. Y. Yu, Y. Kim, and G. Kim, "Tgif-qa: Toward spatio-temporal reasoning in visual question answering." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2758 2766.
[67] A. Oncescu, A. Koepke, J. F. Henriques, Z. Akata, and S. Albanic, "Audio retrieval with natural language queries," arXiv preprint arXiv:2105.02 029..0222.2222 2021.
[68] A. Karpathy and L. Fei-Fei, "Deep visual-semant alignments for generating image descriptions," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3128 -137.
[69] X. Cheng, H. Lin, X. Wu, F. Yang, and D. Shen, "Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss," arXiv preprint (rXiv:2109.0444 2021.
[70] J. Lei, T. L. Berg, and M. Bansal, "Revcaling single frame bias for video-and- -nd-uuga learning"" arXiv preprint Xix::22006.0 034288 2022.
[71] J. Wang, D. Chen, 7. Wu, C. Luo, I.. Zhou, Y. Zhao, Y. Xie, C. Liu, Y.G.. Jiang, and L.. Yuan, "Omnivl: One foundation model for image-language and language tasks." arXiv preprint XXi::2209.07526. 2022.
[72] Q. Ye, G. Xu, M. Yan, H. Xu, Q Qian, J. Zhang, and F. Huang, "Hitea: Hierarchical temporal-aware ladeo-language pre-training." arXiv preprint arXiv:2212. 14..4466,, 2022.
[73] F. Cheng, X. Wang, J. Lei, D. Crandall, M. Bansal, and G. Bertasius, "Vindlu: A recipe for effective video-and- lad-uugg pretraining." arXiv preprint 2rx 2022.
[74] L. Li, Z. Gan, K. Lin, C.-C. Lin, Z. Liu, C. Liu, and L. Wang. "Lavender: Unifying video- understanding masked language modeling," arXiv preprini arXiv:2206.07160, 2022.
[75] A. J. Wang, Y. Ge, R. Yan. Y. Ge, X. Lin, G. Cai, J. Wu, Y. Shan, X. Qie, and M. Z. Shou, "All in one: Exploring unified sideo-lanuaaee pre-training." arXiv preprint arXiv:2203.07300 2022.
14

---

[76] Y. Ma. G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji. "X-elip: End-to-end multi-grained contrastive learning for video-text retrieval," in Proceedings of the 30th ACM International Conference Multimedia, 2022, pp. 638-647.
[77] H. Xu, Q. Ye, M. Yan, Y. J. Ye, Y. Xu, C. Bi, Q. Qian, W. Wang el., "mplug-2: 2: A modularized multi-modal foundation model across text, image and video," " arXiv preprint arXiv:2300. 004,00 2023.
[78] H. Xue, Y. Sun, B. Liu, J. Fu, R. Song. H. Li, and J. Luo, "Clip-vip: Adapting pre ve-trained image-text model to video- representation alignment." arXiv preprint arXiv:2209. 06630, 2022.
[79] V. Gabeur, A. Nagrani, C. Sun, K. Alahari, and C. Schmid, "Masking modalities for cross-modal video retrieval," in Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, 2022, pp. 1766-1775
[80] Y.B. Lin, J. Lei, M. Bansal, and G. Bertasius, "Eclipse: Efficient long-range vidco retrieval using sight and sound," arXiv preprint arXhv:2244.08874 2022.
[81] M. Patrick, P. Y. Huang. Y. Asano, F. Metze. A. Hauptmann. J. Henriques, and A. Vedaldi, "Suuppot-seet bottlenecks for video-text text representation learning," arXiv preprint arXiv:2010.02824, 2020.
[82] Q. Wang, Y. Zhang, Y. Zheng, P. Pan, and X.SS. Hua, "Disentanglod representation learning for text-video retrieval," arXiv preprint arXiv:2203.077 011111 2022.
[83] H. Xu, G. Ghosh, P.-Y. Huang, P. Arora, M. Aminzadeh, C. Feichtenhofer, F. Metze, and L. Zeule- moyer, "Vlm: agnk-tgn video-language model pre-training for video understanding," arXiv preprint arXiv:2105.098 09996, 2021.
[84] D. Li, J. Li, H. Li, J. C. Nichles, and S. C. Hoi, "Align and prompt: Video-and-languagg pre-training with entity prompts." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4953-4963.
[85] T.J.. Fu, I.. Li, Z. Gan, K. Lin, W. Y. Wang, I.. Wang, and 7. Liu, "Violet: End-to-end video-language transformers with masked visual-token modeling," arXiv preprint arXiv:21 12681, 2021.
[86] Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang. B. Li, C. Li al., "Florence:
A new foundation model for computer vision," arXiv preprint arXiv:2111.11433 2021.
[87] J. Lei, L. Li, L.. Zhou, Z. Gan, T. L. Berg, M. Bansal, and Liu, "Less is more: Cliphert for video-and- language learning via sparse sampling," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7331-7341.
[88] T.J. Fu. L. Li. Z. Gan, K. Lin, W. Y. Wang. L. Wang, and Z. Liu, "An empirical study of end-to-end video- ladou-language transformers with masked visual modeling," arXiv preprint Xrv::2209.00 540, 2022.
[89] J. Huang, Li, J. Feng, X. Sun, and R. Ji, "Clover: Towards unified video-language alignment and fusion model." arXiv preprint wiXiv:2007.07885 2022.
[90] A. Yang, A. Micch, J. Sivie, I. Laptev, and C. Schmid, "Just ask: Learning answer questions from millions of narrated videos." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1686-1697.
[91] "Zero-shot video question answering via frozen bidirectional language models." arXiv preprint ar arXiv:2006.08155 2022.
[92] J.BB Alayrac J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Milli- can, M. Reynolds al., "Flamingo: a visual language model for few-shot learning"" arXiv preprint arXiv:2204.1 14198, 2022.
[93] S. Yan, T. Zhu, Z. Wang. Y. Cao, M. Zhang. S. Ghosh, Y. Wu, and J. Yu, "Video-text modeling with zero-shot transfer from contrastive captioners," " arXiv preprint arXiv:2212.04979, 2022.
[94] A. Nagrani, P. H. Seo, B. Seybold, A. Hauth. S. Manen, C. Sun, and C. Schmid, "Learning audio-video modalities from image captions." " arXiv preprint 27Xiv:2204.0067, 2022.
[95] P. Wang, S. Wang, J. Lin, S. Bai, X. Zhou, J. Zhou, X. Wang, and C. Zhou, "One-peace: Exploring one general representation model toward unlimited modalities," arXiv preprint arXiv:2305. 2023.
[96] X. Xu, H. Dinkel, M. Wu, Z. Xie, and K. Yu, "Investigating local and global information for automated audio captioning with transfer learning," in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 905-909.
15

---

[97] Z. Wang. J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov. and Y. Cao, "Simvlm: Simple visual language model pretraining with weak supervision." arXiv preprint 10X04 2021.
[98] C. Jia, Y. Yang, Y. Xia, Y.T. Chen, Z. Parekh, H. Pham, Le, Y.HH. Sung, Z. Li, and T. Duerig, "Scaling up visual and vision-language representation learning with noisy text supervision," in International Conference Machine Learning. PMLR, 2021. pp. 1904-4916
[99] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, "Coca: Contrastive captioners image-text foundation models," arXiv preprint arXiv:2205.01917 2022.
[100] M. Ding, B. Xiao, N. Codella. P. Luo, J. Wang, and L. Yuan, "Davit: Dual attention vision transformers," arXiv preprint 2rX6v:2004003645, 2022.
16

---

Appendix
A
More about VAST Foundation Model
A.1 Pretraining Settings
Specific pretraining configurations of VAST including training corpora, training steps for each corpus (i.e., dataset mix ratio), and training objectives on each corpus are presented in Table 8. To enhance data quality, we use trained vision captioner to generate new captions for CC12M and LAION datasets and replace original captions with them. It is noted that VAST have been trained for relatively small steps (205K steps), but have already shown excellent performances on various types of downstream tasks, and we believe that training by more steps can further increase the model capabilities.
Table 8: Model configurations and pretraining settings of VAST. It is noted that 400M Web data used in CLIP [40] and LAION-400M [29] used in EVAClip [39] are also counted for training samples statics. LAION- 102M and LAION- 10M are both random sampled subsets from LAION-400M. Regarding training objectives, 'ret' represents for the combination of VCC and VCM, while 'cap' denotes VCG, and differnet modality groups are separeted by '%'.
Model
Param
Sample
Training Corpus
Batch Size
Steps
Epoch
Objectives
VAST-27M
1024
60000
2.3
cap vast&
VALOR-IM
1024
25000
25
vap
VAST
1.3B
442M
WavCaps
1024
15000
38
ret%al + capstat
CC4M
2048
30000
12
rettivt caplevt
CC12M
2048
20000
r
LAMON-1I0M
2048
55000
cap
A.2 Downstream Datasets Descriptions
We evaluate VAST on multiple popular domnstream datasets, including MSRVTT, VATEX, YouCook2, VALOR-32K, MSVD. LSMDC, DiDeMo, ActivityNet Caption, TGIE, MUSIC-AVQA, TVC, Clotho, AudioCaps, MSCOCO, Flickr30K and VQAv2. Specific train/val/test splits of those benchmarks can be found in Table 9 and specific descriptions of them are as follows.
MSRVTT [32] contains 10K video clips and 200K captions. The videos cover wide range of topics and scenes, including human activities, sports, natural landscapes, and more. We evaluate text-to-video retrieval, video captioning and video QA on this dataset. Following methods presented in Table 11, we use the IK-A split" for retrieval evaluation. For captioning and QA, we use the standard split.
VATEX [31] contains 41,250 video clips sourced from Kinetics-600 dataset [63] and 825,000 sentence-level descriptions. We evaluate text-to-video retrieval and video captioning this dataset. For captioning, we use the official split. For retrieval. we follow the HGR [64] split protocol.
YouCook2 [51] consists of 14K video clips from 2K instructional cooking videos from YouTube. Each video includes multiple actions performed by the chef, along with corresponding textual descriptions and temporal annotations. We evaluate text-to-video retrieval and video captioning on this dataset with official splits.
VALOR-32K [7] is an audiovisual video- video-language benchmark that contains 32K 10 seconds long audible video clips sourced from AudioSet [11]. Each video clip is annotated with an audiovisual caption which simultaneously describes both visual and audio contents in videos. We evaluate text-to-video retrieval and video captioning on this dataset with official splits.
MSVD [33] contains 1,970 videos, each of which is paired with around 40 captions. We evaluate video QA this dataset and use the split proposed by Xu et al. [44].
LSMDC [65] consists of 118K clips form 202 movies, each of which paird with one caption. We evaluate text-to-video retrieval on this dataset with official split.
17

---

Table 9: Downstream dataset splits.
#Videos/#Images
#Captions/#Qaa Baatoonss/OAApuaaaiea
Task Type
Modal Type
Benchmrark
Train
Val
Test
Train
Val
Test
V-T(Vis)
MSCOCO
113287
5000
5000
566747
25010
25010
Flickr30K
29000
1014
1000
145000
5070
5000
ClothaVI
2893
1045
14465
5225
A-T
V2
3839
1045
19195
5225
AudioCaps
49291
428
816
49291
2140
4080
Retrieval
MSRVTT
9000
1000
180000
1000
YouCook2
10337
3492
10337
3492
VALOR-32K
25000
3500
3500
25000
3500
3500
V-T(M)
VATEX
25991
1500
1500
259910
1500
1500
DiDeMo
1065
1003
1065
ANET
10009
4917
10009
4917
LSMDC
101046
7408
1000
101046
7408
1000
MSCOCO
113287
5000
5000
566747
25010
25010
V-T(Vis)
MSVD
1200
100
670
48774
4290
27763
Clotho
2893
1045
14465
5225
A-T
C2
3839
1045
19195
5225
Caption
AudioCaps
49838
495
975
49438
2475
4875
MSRVTT
6513
497
2990
130260
9940
59800
YouCook2
10337
3492
10337
3492
V-T(MM)
VALOR-32K
25000
3500
3500
25000
3500
3500
VATEX
25991
3000
6000
259910
30000
60000
TVC
86603
10841
174350
43580
MSVD-QA
1200
250
$20
30933
6,415
13157
V-T(Vis)
TGIF- Fram-erameQA
32345
7132
39389
13691
VQAr2
82783
40504
37K/81K
4437570
2143540
1.1M/4.5M
MSRVTT-QA
6513
497
2990
158581
12278
72821
V-T(M))
MUSIC-AVQA
9277
3815
6399
32087
4595
9185
ANET-QA
3200
1800
800
32000
18000
8000
DiDeMo [52] contains 10K long-form videos from Flickr and for each video, four short sentences are annotated temporal order. We follow methods Table 11 to concatenate those short sentences and evaluate "paragraph-to- tovvvooovv' retrieval on this benchmark. The official split is used.
ActivityNet Caption [53] contains 20K long-form videos (180s as average length) from YouTube and 100K captions. We evaluate text-to-video retrieval and video QA on this dataset. For retrieval we use official split and for video QA, split proposed by Yu al. [60] used.
TGIF [66] contains three video QA benchmarks including ActfoAction, TGIF-transition and TGIF- Frame, and the first two are multiple-choice QA while the last is open-ended QA. We evaluate VAST on TGIF-frame benchmark with official split.
MUSIC-AVQA [59] is audiovisual video QA benchmark containing more than 45K Q-A pairs covering 33 different question templates spanning over different modalities and question types. The offcial split is used.
TVC [58] a multi-channel video captioning dataset containing 108K video moments and 262K paired captions. Video subtitles can be used as additional input. We evaluate video captioning this benchmark with official split.
Clotho [12] contains I 15-30 second audio clips and has two versions. The original (vl) has 4981 audios, while an expanded version (v2) includes 6974 audios, enlarging solely the training set. We evaluate text-to- sot-to-audio retrieval and audio captioning on those benchmarks with official split.
AudioCaps [13] contains 51K 10-second clips, with caption in the training set and five in the validation and test sets. We evaluate rext-to-audio retrieval and audio captioning on it. For captioning, use the official split, and for retrieval we follow the Sophia al. [67] split protocol.
MSCOCO [30] contains 123K images each of which paired with annotated captions, We evaluate text-to-image retrieval and image captioning on this dataset with Karpathy split [68].
18

---

Table 10: Downstream task finetuning settings. Lr, Bs, Epo, Obj and Res denote learning rate, batch size, epoch, training objectives and resolution, respectively. Vf(Tr), Vf(Te), Ac(Tr,, Ac(Te) denotes sampled video frames (Vf) or audio clips (Ac) in training (Tr) and testing (Te), respectively. The marks Obj are the same as those in Table 8. Most hyperparameters in the table are not precisely tuned.
Task
Modality
Benchmark
Lr
Bs
Epo
Ohj
Vi(T)
Vi(Te)
Ac(Tr)
Ac(Te)
Res
MSCOCO
lc-5
256
5
ret%vvt
384
V-T(Vis)
Flickr
le-5
256
5
ret%ivt
384
CIothoV1/V2
2c-5
64
10
retSbat
3
3
A-T
AudioCaps
2o-5
64
10
ret%bat
RET
MSRVTT
2e-5
64
3.6
retsivast
8
16
224
YouCook2
3e-5
30
retsv
224
VALOR-32K
20-5
10
ret%vva
224
V-T(MM)
VATEX
2e-5
64
2.5
vats
16
224
DiDeMo
2e-5
64
40
vet
32
2
2
224
ANET
2e-5
64
20
vet
32
2
2
224
LSMDC
2e-5
64
5
ret%vvl
32
224
MSCOCO
le-5
64
5
capSevt
480
V-T(Vis)
MSCOCOISCST)
2.5e-6
64
2.5
capševt
480
ClothoV1/V2
2e-5
64
10
capStat capSaat
A-T
AudioCaps
2e-5
64
10
CAP
MSRVTT
2e-5
128
10
cap%
8
8
224
YouCook2
3e-5
30
capt
224
VALOR-32K
le-5
64
10
vap
12
224
V-T(MM)
VATEX
2c-5
64
10
capfv
20
224
VATEX(SCST)
7e-6
64
5
vast
8
20
224
TVC
3e-5
64
40
capfevst
8
8
224
MSVD-QA
le-5
64
10
qulovt
8
14
224
V-T(Vis)
FGIF-FrameQA
2e-5
64
10
qulevt
4
224
VQA+2
20-5
qu%ovt
384
QA
MSRVTT-QA
20-5
64
4.5
qa%vast
224
V-T(MM)
MUSIC-AVQA
2e-5
64
20
qutEvat
224
ANET-QA
20-5
64
10
qatevat
16
224
Flickr30K [42] contains 31K images each which paired with 5 annotated captions, We evaluate text-to-image retrieval this dataset with Karpathy split [68].
VQAv2 [45] was used as the basis of the 2017 VQA Challenge2, it contains 1.1M questions with 11.1M answers relating to MSCOCO images. The official split is used.
A.3
Finetuning Settings
Specific finetuning hyperparameters of VAST for different benchmarks are presented in Table 10.
A.4 Detailed Comparisons to State-of-the-AAe Methods
Text-to-Video Retrieval. We compare VAST SOTA methods on six multi-modal ext-to-videe retrieval benchmarks. As shown in Table 4, VAST improves previous SOTA methods by 5.1, 1.6, 3.7, 6.1 points on MSRVTT, DiDeMo, ActivityNet, VATEX benchmarks, respectively. Besides above mentioned vision-oriented benchmarks, VAST outperforms VALOR-L [7] by 6.8 points on the audio- oriented benchmark VALOR-32K, and surpass MELTR [55] by 16.7 points on the mnbtitl--ricnnted benchmark YouCook2, which demonstrate the strong generalization capabilities of VAST towards different types of downstream datasets. In addition, the zero-shot retrieval performance comparison is shown in Table 12, VAST achieves 49.3 and 55.5 zero-shot R @ performance that surpasses previous SOTA by 7.3 and 6.9 points, respectively.
Video QA. We evaluate VAST on five open-ended video QA benchmarks. As shown in Table 13, VAST have achieved new SOTA performances on all benchmarks, and outperform recent proposed large-scale foundation models such as GIT [19], MaMMUT [49], Flamingo [92] and CoCa [93]. In addition, on the audiovisual video QA benchmark MUSIC-AVQA, VAST surpasses VALOR by 1.8 points, demonstrating its better capabilities to answer both visual and audio questions.
19

---

Table 11: Performance comparison on Text-to-Vided Retrieval benchmarks. Recall @ 1,5,10 are used evaluation metrics. For fair comparisons, performances before employing post-processing such dual-softmax [69] are reported and compared. All benchmarks are multi-modal benchmarks (containing audio and subtitle tracks). Methods utilizing audio subtitle modalities besides vision for video representation are marked with gray background color.
MSRVTT
DiDeMo R@5
ActivityNet
Method
Sample
@@D
R@5
R@
@@1
R@10
R@
R@5
R@
10
Singularity [70]
17M
41.5
68.7
77.0
53.9
79.4
86.9
47.1
75.5
85.5
OmniVL [71]
17M
47.8
74.2
83.8
52.4
79.5
85,4
HiTeA [72]
17M
46.8
71.2
81.9
56.5
81.7
89.7
49.7
77.1
86.7
VINDLU-L.IT [73]
25M
48.8
72.4
82.2
59.8
86.6
91.5
55.9
82.3
90.9
LAVENDER [74]
30M
40.7
66.9
77.6
53.4
78.6
85.3
All-in-one [75]
138M
37.9
68.1
77.1
32.7
61.4
73.5
CLIP4Clip [56]
400M
44.5
71.4
81.6
43.4
70.2
80.6
40.5
72.4
X-CLIP [76]
400M
49.3
75.8
84.8
47.8
79.3
46,2
75,5
mPLU/G-2 [77]
417M
53.1
77.6
84.7
56.4
79.1
85.2
UMT-L [54]
425M
58.8
81.0
87.1
70.4
90.1
93.5
66.8
89.1
94.9
CLIP-VIP [78]
500M
54.2
77.2
84,8
50.5
78.4
87.1
53.4
81.4
90.0
MMT [21]
136M
26.6
57.1
69.6
28.7
61.4
AVLNet [25]
136M
22.5
50.5
64.1
Gabeur et al. [79]
136M
28.7
59.5
70.3
29.0
61.7
ECLIPSE [80]
400M
44.2
45.3
75.7
86.2
VALOR-L [7]
433.5M
54.4
79.8
87.6
57.6
83.3
88.8
63.4
87.8
94.1
VAST
442M
63.9
84.3
89.6
72.0
89.0
91.4
70.5
90.9
95.5
VATEX
VALOR-32K
YouCook2 R@5
Method
R@1
RGP
R 10
Metbol
R@l
R@B
R@
Method
R
R 10
Support-act [81]
44.9
82.1
89.7
Frozen [3]
32.9
60.4
71.2
UnVL [23]
28.9
57.6
70.0
CLIP4Clip [56]
55.9
89.2
95.0
CLIP4Clip [56]
43.4
69.9
79.7
MELTR [55]
13.7
63.1
74.8
DCR 1821
65.7
92.6
96.7
AVLNes [25]
VLM [83]
27.1
56.9
69.4
VALOR-L
76,9
96.7
98.6
VALOR-L
73.2
91.6
95.4
VALUE [20]
31.3
53.0
VAST
83.0
98.2
99.2
VAST
80.0
93.7
96.6
VAST
50.4
74,3
80.8
Table 12: Performance comparison on zero-shot Text-to-Vides Retrieval benchmarks. Methods utilizing audio or subtitle modalities besides vision for video representation are marked with gray background color.
MSRVTT R@5
DiDeMo R@5
Method
Sample
@
R@
R@I
R@ 10
Frozen [3]
5M
18.7
39.5
51.6
21.1
46.0
56.2
ALPRO [84]
5M
24.1
44.7
55.4
23.8
47.3
57.9
Singularity [70]
5M
28.4
50.2
59.5
36.9
61.6
69.3
HiTeA [72]
17M
34.4
60.0
69.9
43.2
69.3
79.0
Omni [71]
18M
42.0
63.0
73.0
40.6
64.6
74.3
VIOLET [85]
183M
25.9
49.5
59.7
23.5
49.8
59.8
UMT-L [54]
425M
40.7
63.4
71.8
48.6
72.9
79.0
Florence [86]
900M
37.6
63.8
72.6
VAST
443M
49.3
68.3
73.9
55.5
74.3
79.6
Video Captioning. In Table 14, we compare VAST to state-of-the-art methods on five multi-modal video captioning benchmarks. According to the results, VAST have achieved new ofate-of-the-arn CIDEr score on all five benchmarks with evident margins. Compared to previous vision-language modal SOTA method GIT [19] which takes 5.1B DaViT [100] as vision encoder and conduct pretraining on 12.9B private image-text text corpus, VAST surpass it with only 22.5% parameters and 3.4% training data, demonstrating the high efficiency of our method. Compared to previous multi- modal video-language SOTA method VALOR [19], VAST can additionally process subtitle-oriented benchmarks such YouCook2 and TVC. and achieves better results due to that jointly models the relations between text and omni-modalities in videos.
20

---

Table 13: Performance comparison Video QA benchmarks. Acc is used evaluation metric. MSVD-QA and TGIF-QA -A are vision-only benchmarks while others are multi-modal benchmarks. Methods utilizing audio subtitle modalities besides vision for video representation are marked with gray background color.
Method
Sample
MSRVTT-QA
MSVD-QA
TGIF-QA
ActivityNet-AA
MUSIC-AVQA
ClipBERT [87]
5.4M
37.4
60.3
ALPRO [84]
5M
42.1
45.9
VIOLETY2 [88]
44.5
54.7
72.8
Clover [89]
43.9
51.9
71.4
OmniVL
17M
44.1
51.0
HiTeA [72]
17M
45.9
55.3
73.2
46.4
SINGULARITY [70]
I7M
43.5
43.1
VINDLU-- [73]
17M
43.8
44.6
LAVENDER [74]
30M
73.5
JustAsk [90]
69M
41.5
46.3
MERT .OT [6]
180M
43.1
69.5
41.4
All-in-one [75]
228.5M
46.8
48.3
66.3
FrozenBiLM [91]
410M
47.0
54.8
68.6
43.2
mPLUG-2 [77]
417M
48.0
58.1
75.4
UMT-L [54]
425M
47.1
Vider [57]
646M
47.1
55.5
72.2
GIT [19]
1.7B
43.2
56.8
72.8
MaMMUT [49]
2B
49.5
60.2
Flamingo (80B) [92]
2.3B
47.
VideoCoCC (2.1B) [93]
4.8B
46.0
GIT2 (5.1B)
12.9B
45.6
58.2
74.9
VALOR-L
433.5M
78.7
78.9
VAST(13BB
442M
50.1
60.2
79.1
50.4
80.7
Table 14: Performance comparison on Video Captioning benchmarks. All benchmarks are multi- modal benchmarks. BLEU@4 (B @4) and CIDEr (C) metrics are used as evaluation metrics. On VATEX benchmark, follow most state-of-the-art methods [19; 7: 62] employing SCST finetun- ing [41] after cross-entropy training, and corresponding results are marked with Methods utilizing audio subtitle modalities besides vision for video representation are marked with gray background color.
Method
Sample
MSRVTT
VATEX
YouCook2
TVC
VALOR-32K
B B@4
C
B@4
C
B@4
B@4
C
B@4
C
SwinBERT [61]
41.9
53.8
38.7
73.0
9.0
109.0
14,5
55.4
5.4
27.3
VIOLETv2 [88]
5M
58.0
HÎTeA [72]
5M
62.5
LAVENDER [74]
30M
60.1
MaMMUT [49]
2B
73.6
GIT[19)
1.7B
53.8
73.9
41.6*
91.5*
10.3
129,8
16.2
63.0
GIT2(5.1B) [19]
12.9B
54.8
75.9
42.7
94.5*
9.4
131.2
16.9
66.1
SMPFF [22]
48.4
58.5
39.7
70.5
7.5
37.1
VALUE [20]
136M
58.1
12.4
130.3
11.6
50.5
UniVL [23]
136M
41.8
50.0
17.4
181.0
MELTR [55]
136M
44.2
52.8
17.9
190.0
CIIP44Cppiio+++ [62]
400M
40.6*
85.7*
15.0
66.0
VALOR-L
433.5M
54.4
74.0
45.6*
95.8*
9.6
61.5
VAST(L.3B
442M
56.7
78.0
45.0*
99,5*
18.2
74.1
9.9
21

---

Table 15: Performance comparison on Text-to-Audio Retrieval benchmarks. Recall @ 1,5,10 are used as evaluation metrics.
C R@5
ClothoV2
AudioCaps R@5
Method
R@1
R@ 10
R@1
R@5
R@10
R@1
@@ 10
Oncescu ct al. [67]
9.6
40.1
25.1
73.2
Nagrani al. [94]
12.6
45.4
35.5
84.5
LAION [16]
16.1
38.3
51.1
36.1
71.8
83.9
CNN14- 4-RERTTTTTT [18]
21.5
47.9
61.9
35.1
70.0
82.1
HTSAT- BERT [18]
19.7
45.7
59.4
42.2
76.5
87.1
VALOR-B [7]
17.5
42.7
55.3
40.1
73.9
83.1
ONE-PIECE [95]
22.4
49.0
62.7
42.5
77.5
88.4
VAST
25.1
51.5
64.0
26.9
53.2
66.1
52.0
76.8
82.9
Table 16: Performance comparison on Audio Captioning benchmarks. BLEU@4 (B@4) and CIDEr (C) metries are used evaluation metrics.
ClothoV1
Clotho
AudioCaps
Method
B@4
M
B@4
M
R
C
B@4
M
Xu et al. [96] CNNI4-BART [18]
15.9
16.9
36.8
37.7
23.1
22.9
46,7
66.0
18.0
18.5
40.0
48.8
27.2
24.7
49.9
75.6
HTSAT-BART (18)
16.8
18.4
38.3
46.2
28.3
25.0
50.7
78.7
VALOR-B [7]
16.2
17.4
38.2
42.3
27.0
23.1
49.4
74.1
VAST
18.5
18.9
39,9
50.7
19.0
19.3
40.8
51.9
29.5
24.7
50.9
78.1
Text-to-Audio Retrieval and Audio Captioning. As shown in Table 15, VAST have largely improved previous SOTA methods on three text-to-audio retrieval benchmarks, by 7.6, 5.4 and 9.8 R@I points, respectively. and for audio captioning task, VAST achieves new SOTA performances on Clotho benchmark (both V1 and V2), and comparable performance on AudioCaps benchmark to WayCaps [18]. It is noted that WavCaps explored four model architectures with different audio encoder and text encoders targeting different benchmarks, while VAST takes unified architecture without targeted optimizations for specific downstream benchmarks.
Image- Benchmarks. We evaluate VAST text-to-image retrieval, image captioning and image QA benchmarks. The results are presented in Table 17, from which we can find that even though VAST is designed as a omni-modality video-language understanding and generation model, also shows strong capabilities on image-(ext benchmarks, demonstrating its generalization capabilities towards tasks of various modality types. Specifically, VAST achieves new SOTA performance R
Table 17: Performance comparison on Image-Text downstream tasks. CIDEr (C) and SPICE (S) metrics reported for captioning. On MSCOCO caption benchmark, we follow SOTA methods [19; 7; 48) employing SCST finetuning [41], and corresponding results are marked with **
MSCOCO-Ret
Flickr30K Ret
MSCOCO-Cap
VQAv2
Method
Sample
R@1
R@s
R@10
R@I
R@s
R@10
dev
std
ALBEF [38]
14M
60.7
84.3
90.5
85.6
97.5
98.9
76.04
OFA [48]
18M
154.9
26.6*
82.0
82.0
BETT-3 [47]
21M
67.2
87.7
92.8
90.3
98.7
99.5
147.6
25,4
84.19
84.03
BI.IP [28]
129M
65.1
86.3
91.8
87.6
97.7
99.0
136.7
78.25
78.32
BLIP-2 [46]
129M
68.3
87.7
92.6
145.8
82.19
82.30
mPLUG-2
65.7
92.6
88.1
97.6
99
137.7
23.7
81.11
VALOR-L
433.5M
61.4
84.4
90.9
152.5
25.7*
78.46
78.62
Florence [86]
900M
63.2
85.7
87.9
98.1
80.16
80.36
PaLl [50]
1.6B
149.1
84,3
84.3
GIT [19]
1.7B
151.1
26.3*
78.6
78.8
SimVLM (97]
1.8B
143.3
25.4
80.03
80.34
ALIGN [98]
1.8B
84,9
98.6
Flamingo (80B)
2.3B
138.1
82.0
82.1
CoCa(2.1B) [99]
4,8B
143.6
24.7
82.3
82.3
GIT2(5.1B) [19]
12.9B
152,7*
26.4
81.7
81.9
VAST
442M
68.0
87.7
92.8
91.0
98.5
99.5
149.0*
27.0*
80.23
80.19
22

---

score of Flicker30K and R@ @@ 10 scores of MSCOCO dataset, which outperforms image-text pretrained foundation models such as B [46] and BEiT-3 [47]. On COCO caption benchmark, VAST achieves 27.0 SPICE score which outperforms all previous methods such as OFA [48] and GIT2 [19]. On image QA benchmark, VAST achieves better performance than GIT [19], which is also a generative methods and predicts answers in a fully open way without any constraints.
23

---

B More about VAST-27M Dataset
B.1 Word cloud distribution
hown yourgeed news
a
man
gave
playing
Speak camerai
FEBGE BAGEOAAA dera pet persana
S Standing
Sone eeethnng Pone
woman
front
sound
played
Bivling
heal
rttting
WO
siving
stoie
doo
Showing
ng
deo
saying! Foep
ePT
eccorded
ngg
un
oa
someone
Ponnno ther
peopleca
ca
men
nekirulllde eooddw De
voice
"Shor
Vision Caption
Audio Caption
ithatstime don th
Saree grght iight na
Wa want
Video one
woman
We 11
W
sound
m
ment ons FOOO
really
see guys
think
ec
Subra
ta
Spakinng
goe
ong
gett
got
also
little
Playings
ommone
Subtitle
Omni-modality Caption
Figure 3:
Word cloud map (Top-200) for vision, audio, omni-modalln captions and raw subtitles of
VAST-27M.
B.2
Prompts for Omni-Modality Caption Generation
Single Modality Captions
1. person surting the ocean in tront of varees
man surting middle ocean and
ocean wavee and man speaking are heard, talkingg heard.
's second place on opens up with litle the ridee. through the euplostiv clossout and look into better"
A surfer surfboard rides in weve, then jumps and starts surfing
ModilModality Caption Generation with Different Instructional Prompts
"Hanaa: About one video, hore viston eagtions: 0.3 audio eaptions: and one speooh subtide: You nood to undersiand and
A surfor rides wave, jumping up and surfing in the background, compate the spot. with a close-up of the surter tiraating the ride and looking shead.
audio captions maybe redundintt sentence. encods with speech subntts, Do not simply concalerate them together: weights sidzeleudeokpaschh are equaled.
them befors
Monil-Modillty
"Human: About one video, vatoo captions: 0,3 underaland and encode
sertence. Vision and audio capiions maytee indiundnnt cummarte there bofore wideolandookpeechh Absiatant: The sentence to
Omni Aonii-Mddlltty Captioner (Vicuns- 13b)
the sound the oceen waves speaking surting the midile of The oooin with the sound of
sound of eas serves and people talking --
"Humar: About video, here valion captions: 0.3 audio captions: En and one speech subride: 8. You need to audlo captions rndundant, summarizs theme before enoods with speech subnits, Do not simply concatarate them logether. woights aidiolurdiolppp are equaled.
Omni-Moodii (Vicuna-13)
betore navigating through an eeplosiiv clossose looking for beiter spot.
Figure 4: Ablation study for instructional prompt used for omni-modality video caption generation in VAST-27M.
24

---

B.3 More Examples
boy riling a shancboard at skile park. scimooarcing
skarescord
shate part will
skotnboording
Skandoaarter oxplining
movis and coplaitiing
skalcboord
skandooarders are skaling
your honds and goe your rotn thero
Thes
Labove through
your logs, golb just grab your nose a Bile 28 and help get
cartacn charactrs pet ready go
pharedeess
a group of cerloon charastors hawe their sultase openina
incluting rlogy
prety proparto
churector shecking they
carsonn charader speaking. antrrated niking
makking
munoyatil
snother dhanacte ng amportanin
the houne.
you alivery
surglanaes while read. a man criving doeen the toad.
Amron in surg
driving oor with Higs spoid beckground on reedt with cesire just oliv off aurset
sceaking about
speed engine sound backgiound
reatlly wcai vide rewendape So mate bosel
a Woman le peuriig licad into harge plaos bowt putting
introducing and
showing
inmoduding ingredents.
intia cling
favorte
a person cookingg
und sdblo the macket, and
Lble nowadaps. Bo contenned Mil. concereed
oi boot
Figure 5: More samples in VAST-27M.
25


Reviews:
[{"review_id": "oEHjhIv4jC", "reviewer": "Reviewer_1sdZ", "summary": "This paper proposes a large-scale omni-modality video caption dataset based on an existing video-text dataset, namely VAST-27M. Specifically, the authors utilized the VL models to generate captions and LLM to generate omni-captions for videos. Then a vision-audio-subtitle-text model is trained for a range of video tasks and establishes SOTA in several benchmarks.", "questions": "Please see the weakness part.", "limitations": "The authors have addressed the scale limitation of the current dataset. It would be great if they could share more insights into the potential limitations of using their dataset and model, such as biases it could contain.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and easy to read. \nThe idea of combining text, video, and audio in one caption is interesting and novel. I believe this will contribute to the multimodal learning community. \nThe performances over a wide range of video-language-audio tasks are strong.\n", "weaknesses": "1. Despite the dataset contribution, the technical contribution of the pre-training model is somewhat weak and lacks novelty.\n2. I am confused by some details in the approach. (1) In Lines 185-186, why do you need to feed the caption tokens again instead of using the textual embeddings from the text encoder above? (2) In OM-VCG, do you use the masking strategy like BERT or generative training like GPT? Based on my understanding, you are adopting the former one, for which I think the name of “video caption generation loss” is improper, masking language modeling (MLM) may be better.\n3. Although the experimental results are very strong, I am unsure where the performance gain comes from. Several points that might be attributed to this are not discussed. (1) Combination of different datasets. The model combines several different datasets, including VAST-27M while the dataset scale is not compared with SOTA methods. (2) Strong backbones. As indicated in Line 213, EVAClip-ViT-G is adopted as a visual backbone which is different from SOTA models, which makes the comparison in Table 3 unfair. (3) More modalities. Although Table 7 provides an extensive comparison of using different modalities, there are no results that could be directly compared in Table 3 for a fair comparison. \n4. Some details are missing. (1) How do you select the 27M video clips from 100M clips in HD-VILA-100M? What is the selection standard? (2) Do you use audio in the HD-VILA-100M dataset for audio captioning and as input for the audio encoder? It is not clearly explained in the paper. (3) How do you sample frames in videos and how do you deal with video embedding from frame features in ViT (i.e., how to get f_v)? By Meanpooling?\n5. Lack of discussion on related works. As an extension of HD-VILA, CLIP-ViP used an off-the-shelf image captioner to generate captions for the HD-VILA-100M dataset. The authors should also compare that one and your advantage over CLIP-ViP. It seems that only audio information is not used in the CLIP-ViP model.\n6. It would be helpful to show some examples in the dataset for illustration.\n7. Some typos: Row 1 in Table 5. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**weakness 1**  \nAbout the technical contributions of VAST model have been explained in our global rebuttal **[Common Question 1]**.  \n\n**weakness 2**  \n(1) Using origin captions in OM-VCM, OM-VCG and OM-VCC can keep consistence with BERT whose parameters are inherit for text encoder initialization. In addition, we have also tried use text embeddings of OM-VCC as input for text encoder used in OM-VCM and OM-VCG, but found performance slightly reduced.   \n\n(2) We assume that common generative modeling method can have two forms: generative modeling like GPT style (LM, input current word, output next word), and causal masked language modeling (CMLM, mask current token and predict it, but with only previous tokens as conditions). As stated in Section 4.2 (line196), VAST takes the latter one (CMLM) for the reason that in the early development stage we found that BERT+CMLM performs slightly better at most downstream benchmarks than GPT+LM. We name it OM-VCG from the target perspective that is to generate captions, and it can also be renamed as OM-CMLM.\n\n**weakness 3**  \n(1) For the space limitation reason, we put more detailed comparison results including the dataset scale comparison among different SoTA methods in Appendix.   \n+ From the results in Table 4 and Table 5 (Appendix), we can find that VAST (442M data) outperforms CLIP-VIP(500M),  Florence(900M) on text-to-video retrieval benchmarks with huge margins.\n+ From the results in Table 6 (Appendix), we can find that VAST (442M) outperforms VideoCoCa(4.8B), GIT2(12.9B), Flamingo(2.3B) on video QA benchmarks with huge margins.  \n+ From the results in Table 7 (Appendix), we can find that VAST (442M) outperforms GIT2(12.9B), GIT(1.7B), MaMMUT(2B) on video caption benchmarks with huge margins.  \n\n(2)  To get the best performance and demonstrate our methods function at the most advanced vision models, we use a strong vision backbone (EVACLIP-Giant) in the main paper, which is not consistent with other methods. To make fair comparison, we additionally trained a VAST-B model which use CLIP-VIT-B/16 as backbone and compare it to  methods with the same backbone used. The results  are shown in the following table which demonstrated that VAST can outperform both only-vision and multi-modal video-language pretraining models on three representative benchmarks.\n\n| Model | Vision Backbone | MSRVTT-Ret(R@1/R@5/R@10) | MSRVTT-Cap(CIDEr) | MSRVTT-QA(Acc) |\n|---|---|---|---|---|\n| GIT2-B | CLIP-VIT-B/16 | - | 57.8 | 41.0 |\n| mPLUG2-B | CLIP-VIT-B/16| 48.3/75.0/83.2  | 72.4 | 46.3 |\n| UMT-B | CLIP-VIT-B/16| 51.0/76.5/84.2 | - | 44.9 |\n| VALOR-B | CLIP-VIT-B/16| 43.0/72.2/82.1 | 66.6 | 46.7 |\n| **VAST-B (Ours)**  | CLIP-VIT-B/16| **56.9/79.8/87.1** | **74.2** | **48.2** |\n\n(3) In the Table 4,5,6,7 of Appendix, a groups of methods utilizing multiple modalities beyond vision and text (audio, subtitle) are marked by gray background colors and we mainly compare VAST with those groups of methods for fair comparison. From the results we can find that VAST outperforms all not only single-modality but also multi-modality methods.\n\n**weakness 4**  \n(1) The specific selection standard has been illiustrated in the global rebuttal **[Common Question 2]**  \n\n(2) Audios in HD-VILA-100M are not used in audio captioner training cause that the subtitle is ASR transcriptions rather than objective captions. Instead we use the combination of VALOR-1M and WavCaps datasets, as illustrated in Section 3.1 (line136). In addition, audios in VAST27M (sourced from HD-VILA-100M) is used as input for VAST model pretraining.  \n\n(3) As illustrated in section 5.1 (line219), during  pretraining we sample only one frame for efficiency and during finetuning, we sample different frames (8~32 frames) for different benchmarks as shown in Table 3 in appendix. With regards to f_v, we do not conduct mean pooling but simply concatenate patch features of all frames together to reserve most information for multimodal encoding or decoding. The computation cost is affordable due to cross-attention mechanism (linear complexity for condition features) used in multimodal fusion instead of merge attention (quadratic complexity). We are sorry about not clarifying f_v, and will modify it in next version of paper, thanks for pointing it.  \n\n**weakness 5**  \n+ CLIP-VIP  proposes to re-generate captions for HD_VILA_100M dataset through open-source caption model OFA, and use both raw subtitle and generated captions for video-language pretraining. Compared with CLIP-VIP, VAST has explicit differences and strengths which can be summarized as follows.\n+ Firstly, OFA  which is the captioner CLIP-VIP used is a image-text model which has not seen video caption data during training, while our video captioner is trained on mixture image-text and video-text corpus that can generate more motion-related video captions. In addition, their captions reflects visual contents only while captions in VAST-27M contains both vision, audio, subtitle descriptions and omni-modality caption, which is apparently more appropriate for multi-modality pretraining, given its importance as discussed in Common Question 1. \n\n+ Secondly, CLIP-VIP has tried use together generated visual captions and raw subtitles and models both \"V-T\" and \"V-S\" relationships while VAST models VS-T correlations. This is because we think at most practical scenario we would like to put subtitle on the video input to help video content understanding, instead of predicting them which can be independently addressed by mature ASR algorithms. In addition, CLIP-VIP does cross-modality tasks (caption, QA, retrieval, etc.) only based on vision signal (1 signal), while VAST can utilize vision, audio and subtitle (3 signals).\n\n**weakness 6**  \nConsidering the limited space in main paper and many contents need to be clarifed, we put more examples in the Appendix (Page 9 B.3).\n\n**weakness 7**  \nThanks for pointing it, we will change \"PT Data\" to \"Model\"  in the next version. "}, {"review_id": "fgSuVMIPGF", "reviewer": "Reviewer_v49r", "summary": "The paper proposed to build a new multi-modal foundation model that includes vision, audio, subtitle and text modalities. The designed architecture of the model is very similar to previous works, such as BLIP. The paper also made another contribution, a new multi-modal video dataset VAST-27M. VAST-27M reuse videos from another dataset HD_VILA_100M. The paper uses off-the-shelf video captioning models and audio captioning models to generate text descriptions for the input video. Large language models are then applied to summarize the text descriptions together with the original subtitles of the video using customized prompts. The paper presents experimental results on 22 downstream datasets to demonstrate the effectiveness of the proposed approach.", "questions": "See above.", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "High-quality pre-training datasets that are openly accessible to the research community is very important for us to experiment with new research ideas and push the research boundaries. The new VAST-27M dataset could be very valuable for researchers working on related topics. The experimental results also demonstrate the advantages of the trained model.", "weaknesses": "The following are more detailed comments and suggestions about the paper.\n\n1, What is the difference between subtitles and text? The paper seems to treat them as different modalities. Also does the audio modality already contain all the information from subtitles?\n\n2, Based on the implementation  details in Section 5.1, the final model is initialized with other pretrained encoders, and pretrained with lots of other datasets as well. It is not clear how much the improvement is from the proposed VAST-27M dataset. It will be better if the paper can show the difference for with or without using the VAST-27M dataset. This can help us better understand the quality of VAST-27M.\n\n3, How to select a subset of 27M videos from HD_VILA_100M? Please elaborate the details why the paper selected a subset from HD_VILA_100M.\n\n4, In Line 60, why does the cross-attention layer only involve the text modality? Do we have cross-attention between video and audio?\n\n5, The paper seems to use BLIP for vision captioning. Why not use BLIP-2, which can generate better captions?\n\n6, The overall design of the learning objectives is very similar to BLIP, which also uses three losses, i.e., Image-Text Contrastive Loss (ITC), Image-Text Matching Loss (ITM), and Language Modeling Loss (LM). The novelty of the proposed learning framework seems to be very limited.\n\n7, All the captions from VAST-27M are generated by models. Does the paper look into the bias of the used models and their generated captions? Will they limit the quality of the dataset? \n\n8, A lot of details in the paper are similar to reference [7]. It is unclear what is the real contribution of the current submission. Many sections are very similar to [7]. I recommend clarifying the difference with [7], e.g., what are the contributions from the current submission, what has been done in [7].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**weakness 1**  \n+ Subtitle is usually achieved by ASR or OCR techniques and can be a supplementary modality for video understanding, while captions are usually objective descriptions about objects existed or events happened in images or videos. Even though they are both represented as text, they have different data distribution and information density. In this work, the word \"T\"/\"text\" in VAST refers to \"caption\" which is a language output or retrieval query, while \"S\"/\"subtitle\" is an input modality. We use text(T) instead of caption(C) in \"VAST\" simply for getting a abbreviation that sounds better.   \n+ In this work, audio modality already contains all information from subtitles (ASR texts), but ablation study in Table 7 shows that explicitly taking subtitle as input achieves better results than taking only audio signal as input, and we attribute it to that current audio encoder doesn't possess strong speech reconigiton ablities and we think it's a valuable future research direction to perceive both environment sounds and human speech information from audio signal only.   \n\n**weakness 2**  \nThanks for your suggestions. We conduct the following experiments to demonstrate the effectiness of VAST-27M. VAST-B is trained on same datasets with VAST but vision backbone replaced with CLIP-VIT-B/16 for fast training. Both two models are trained for same iteration and batch size. The results in following table shows that VAST-27M contributes a lot to VAST's high performance, without which the performances drops drastically on seven benchmarks. \n| Model | MSR-ret | MSR-cap | MSR-qa | YouCook2-ret | YouCook2-cap | VALOR32K-ret | VALOR32K-cap |\n|---|---|---|---|---|---|---|---|\n| **VAST-B**  | **56.9/79.8/87.1** | **74.2** | **48.2** | **36.3/57.7/67.3** | **186.7** | **70.0/90.2/94.7** | **54.6** |\n| VAST-B (w/o VAST-27M) |  50.8/76.1/85.5 | 70.7 | 47.0 | 32.6/57.5/67.8 | 130.3 | 62.8/86.5/92.5 | 53.0 |\n\n**weakness 3**  \nThe specific selection standard has been illiustrated in the global rebuttal [Common Question 2]  \n**weakness 4**  \nIn VAST, text encoder have taken the place of fusion interface for multimodal  understanding and generation via cross attention. Text takes the role of query while single one or some combinations of vision, audio and subtitle  take the role of key and value. We take this design mainly because most tasks we target at (captioning, QA, retrieval, etc.) can be uniformly modeled with language interface. In current version of VAST video or audio has not taken the role of fusion interface and  we think is important to update VAST in future to handle more tasks like language-guide video/audio prediction/generation/edition and then it will be necessary to add cross attention layers in video /audio encoder and also take them as output interfaces.  Thanks for your suggestions.  \n**weakness 5**  \n In fact, VAST uses self-trained vision captioner as illustrated in Section 3.1 (line128). We do not use open-source BLIP or BLIP-2 mainly because they are finetuned using image-text data (MSCOCO) only while the capabilities of describing dynamic events or actions are ignored. So we resort to trained a more general captioner that focus on both static or dynamic vision scenarios by finetuning on mixture of image/video captioning datasets, in order to generate more high-quality captions for VAST-27M.  \n**weakness 6**  \nAbout technique novelty has been elaborated in the global rebuttal [Common Question 1].  BLIP [6] is a image-language pretraining method aiming at bootstrapping model with trained captioner and filter. The differences between VAST and BLIP can be summarized as follows.\n\n+ About data generation. The vision captioner  for VAST-27M is inspired by BLIP, but we make several modifications to fit multi-modality pretraining tasks including 1) training vision captioner with the mixture of image and video caption datasets to increase captioner’s generality. 2) promoting from vision-language field to audio-language filed and trained a  general audio captioner. 3) utilizing LLM to integrate multi-perspective captions.   \n\n+ About  model architecture. Different from BLIP that contains a image encoder and text encoder, VAST contains additionally an audio encoder which supports audio input. In addition, vision encoder of VAST can support both image and video input, and text encoder can support both caption and subtitle input.\n\n+ About  training objectives. Image-text contrastive learning(ITC), matching(ITM), and image-conditional text generation are three most common training loss taken by current image-language pretraining models including BLIP.  However, only relationship of image and text can be constructed through these losses, VAST have successfully modified them to omni-modality losses (OM-VCM, OM-VCC and OM-VCG as illustrated in Section 4.2), and use modality grouping loss illustrated in Section 4.3 to handle different modality combinations, which can not be realized with BLIP’s architecture and training objectives.  \n  \n**weakness 7**    \nThe dataset bias has been illiustrated in the global rebuttal [Common Question 3]     \n**weakness 8**  \nVALOR proposed a one-million level video dataset that connects text with both video and audio, and  a vision-audio-text model. VAST has explicit differences and strengths compared with VALOR, which can be summarized as follows:  \n\n+ Firstly, VALOR-1M contains limited semantic concepts and cannot be friendly scaled up due to it's manual annotaion . By contrast, VAST27M is automatically generated that  can be scaled up easily, and contains much more semantic concepts, which can be quantitatively reflected by experiments shown in Table 4,5,6 in the paper: models pretrained on VAST-27M outperform models trained on VALOR-1M on V-T, A-T and AV-T tasks.\n\n+ Secondly, as illustrated in Common Question 1, VALOR stands for a series of works that models both vision and audio while VAST for the first time unifies all those modalities in one framework.  \n"}, {"review_id": "c4xcXupQaG", "reviewer": "Reviewer_hHHX", "summary": "This paper introduces a large-scale dataset called VAST-27M, which contains audios, videos, subtitles, and generated captions. The authors employ an LLM to combine all forms of textual information to create omni-modality captions. They train an omni-modality video-text foundational model on this dataset. Extensive experiments conducted on various downstream tasks demonstrate the effectiveness of both their dataset and model.", "questions": "I'd like to inquire about the authors' plans for sharing the data and code, including the captioning model, VAST model, and the dataset.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- They provide a valuable dataset, and the experiments demonstrate the superiority of this dataset. \n- The paper is well-written with clear explanations. \n- The experiments are thorough, including the validation of the methodology, the effectiveness of the dataset, and comprehensive comparisons with the state-of-the-art (SOTA) approaches. \n- The model achieves state-of-the-art (SOTA) performance on 22 downstream tasks.", "weaknesses": "- The authors utilized the Vicuna model instead of ChatGPT, which might have an impact on the quality of the data. Could you provide a comparison of the quality of data generated by ChatGPT and Vicuna?\n- Why wasn't there consideration for multimodal LLMs like LLaVa or MiniGPT-4, and instead, but using a separate trained captioning model?\n- The methodology lacks technical contributions and doesn't differ significantly from previous works in terms of model architecture and training. When dealing with multiple forms of text simultaneously, the current models simply aggregate multiple losses, which may not fully leverage the available information.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**weakness 1**  \n+ Our work needs to generate  27M omni-modality captions, and locally deployed open-resource large language modal (such as Vicuna) can fulfill parallel generation which is efficient. However this is not feasible for ChatGPT due to its API is relatively inefficient and requies additional costs, so we choose to use the former for caption generation.  \n+ With regards to caption quality comparison between Vicuna and ChatGPT, we visualize two examples in the following table. The same video caption,  audio caption,  subtitle and instruction are given to both models and we sample omni-modality captions twice. From the examples shown in the following table, we can find that both models can generate relatively high-quality captions and can comprehensively reflect infomation of all modalities. Captions generated by ChatGPT tends to be longer and cover most clues given in the prompts, and can also imagine scences like \"wind rustled through the trees\" while 'trees' does not appear in the prompts. By contrast, Vicuna tends to generate short but accurate captions, while some details like \"splashing\" and \"mailbox\" may be missed.\n|Modality|Caption|\n|---|---|\n| Audio| Wind, birds, women speaking, splashing, and human sounds are heard. |\n| Video| Woman picked up a phone and walked around the garden. |\n| Subtitle | I can't take delivery now. I'm on vacation, why don't you put it next to the mailbox at the door of my house. |\n| Omni(by Vicuna-13b)  | A woman walks with a phone in a garden while birds chirp, wind blows, and a human speaks, with the sound of splashing in the background, and the delivery cannot be taken now as the person is on vacation, so they suggest leaving it near the door of their house. |\n|  | A woman walks with a phone in a garden while wind, birds, and human sounds are heard, and a person is unable to receive a delivery due to being on vacation. |\n|  Omni(by Chat-GPT) | The woman, while on vacation, picked up her phone and walked around the garden, surrounded by the sounds of wind, birds, splashing, and various human sounds. In a speech subtitle, she mentioned being unable to take delivery and suggested leaving it next to the mailbox at the door of her house. |\n|  | The woman, who was on vacation and unable to take delivery, walked around the garden, speaking into her phone, while wind rustled through the trees, birds chirped in the background, and various splashing and human sounds filled the air. She suggested leaving the package next to the mailbox at the door of her house. |  \n|**Modality**|**Caption**|\n| Audio | A turkey is heard, followed by coughing, wind, bird chirping, and animal sounds. |\n| Video | Teenage boys walking in a zoo togehter on a narrow pathway surrounded by trees. |\n| Subtitle| Where do we go now, shall we see elephants first? But I heard that the gorilla exhibition is going to start soon, why don\\'t we go see that first. whatever, we have plenty of time today. |\n|Omni(by Vicuna-13b) | Two teenage boys walking in a zoo surrounded by trees and trees, with a turkey and various animal sounds in the background, discussing their next destination. |\n|  | In a zoo, teenage boys walk together on a narrow path surrounded by trees, while a turkey is heard, followed by coughing, wind, bird chirping, and animal sounds, and they discuss where to go next, considering seeing elephants or the gorilla exhibition. |\n| Omni(by Chat-GPT) | As the teenage boys walked together on a narrow pathway surrounded by trees at the zoo, a turkey's call echoed in the background, followed by the sound of coughing, gentle wind, bird chirping, and a symphony of animal sounds. One of the boys remarked, \"Where do we go now? Shall we see the elephants first? But I heard that the gorilla exhibition is going to start soon. Why don't we go see that first? Whatever we decide, we have plenty of time today.\" |\n|  | As the teenage boys strolled through the zoo surrounded by trees, the air was filled with the sound of a turkey, accompanied by the occasional cough, gentle wind rustling the leaves, the cheerful chirping of birds, and a variety of other animal sounds. Amidst their conversation, one of the boys pondered their next move, suggesting they start by visiting the elephants. However, another boy excitedly mentioned that the gorilla exhibition was about to begin and proposed going there first. Regardless of their decision, they reassured each other that they had plenty of time to explore the zoo today. |\n\n**weakness 2**    \n+ Firstly, this work(VAST) has been done concurrently with LLava and MiniGPT-4. In addition we think that even though MLLMs can generate more detailed descriptions, but their caption accuracy may be not promised (such as the common hallucinations which is a general problems in those models), and inaccurate captions may have bad effects. \n+ Secondly, the inference speed of those models are limited compared to trained captioners due to their extremely large hyperparameters.\n+ Thirdly, there seems not be available audioLLMs for generating audio captions.\n+ Fourthly, LLava and MiniGPT-4 use image-text data for pretraining which may not be appropriate for captioning videos that concentrate on both static appearances and dynamic motion informations.  \n\nConsidering above reasons, we choose to train captioners on mixed datasets (both image and video data) to promise its caption quality.   \n**weakness 3**  \nThe technical contributions of VAST have been illustrated in the global rebuttal [Common Question 1].   \n**question**  \nWe promise that all of VAST-27M data (vision captions, audio captions and omni-modality captions), model checkpoints, training and testing codes of VAST will be public released to facilitate research of multimodal learning communities."}, {"review_id": "N2qgQ9kLfA", "reviewer": "Reviewer_zFyv", "summary": "This paper proposes an automatic pipeline to create captions for videos, where multiple modalities are involved in the generation process of the captions. Authors initially train separate audio and video-based captioners. Then, those audiovisual-driven captions, alongside subtitles are restructured into a single caption by prompting a pertained LLM that is responsible for aggregating multiple text/caption sources. Next, such a dataset is used to train a multimodal model capable of addressing a variety of text-to-video/audio retrieval, generation, and question-answering tasks.", "questions": "1. On OM-VCM: $f_{omv}$ which is fed to the text encoder plays the role of condition, however, a part of  $f_{omv}$ is $f_{s}$ which itself is the output of text encoder. I am not quite sure how this makes sense in terms of the space of inputs to the text encoder. The authors should clarify and elaborate.\n2. Line 206: what does “we uniformly model the relations of V-T, A-T, VA-T, VS-T, and VAS-T ” mean?\n3. Eq 2: why this cannot be captured via contrastive loss? Is it only due to maintaining temporal resolution? If so, authors should try sequences of length 1 which is [cls] token to confirm that.\n4. Line 219: using a single video frame is quite strange. Considering the length of video clips (e.g 5-30 seconds for VAST) does it make sense to pick a single frame, especially if we are aiming to create captions for videos? Did authors try more than 1 frame during pertaining to study the effect of such choice? \n5. Line 223-224: What is the contribution of VCM. In other words, what if you equally weigh the losses or reverse them, how do results vary?\n", "limitations": "This work relies on multiple pretrained models and datasets. The bias-fairness concerns hence scale considerably due to such a design. On top of it is the choice of leveraging LLMs whose hidden biases are not quite yet clear for the research community. This is not to discourage using such models, but I believe authors can further improve the limitation and broader impact section of the paper.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. This work confirms that today’s unimodal caption generation models (datasets) are very strong baselines (resources) that diminish the need for a multimodal data collection process which could be costly and non-scaleable, thanks to summarization and aggregation capabilities of the latest LLMs.\n2. The downstream experiments are comprehensive and show promising results. \n", "weaknesses": "1. Paper is not easy to follow, especially due to the poor choice of naming for components (i.e features, losses etc) and abbreviations. \n2. Paper is highly incremental. It is heavily built on top of numerous existing pertained models and offers almost no technical novelty even in the aggregation step. \n3. Authors propose multiple components to the loss function (Eq 1-5). Still, there is almost no discussion on how these different components interact with one another, and impact the quality of learned representations. The experimental section is only focused on downstream tasks and the generated dataset.\n4. The dataset generation step is shaped by the power of the pertained models which authors have adopted. They are also combining multiple datasets during pertaining. Hence, outperforming SOTA is not only expected but one wonders whether such comparisons are even fair. At the end of the day, VAST stands on the shoulders of numerous pertained models developed by others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**weakness 1**  \nConsidering that the motivation of our work is to build a more general comprehensive omni-modality foundation model,  it unavoidably involves multiple modalities (text, audio, vision, subtitle), multiple features (single/fusion modality features, global/local features), multiple losses (contrastive, matching, captioning). We have indeed paid efforts to make those concepts clear but considering the complexity of algorithms themselves, perhaps the representations seems a little bit complicated and takes some time for understanding, we are sorry about this and will further improve representations in next version.  \n\n**weakness 2**\n+ Firstly, the technical novalty has been illustrated in the global rebuttal [Common Question 1].\n+ Secondly, we think that the statement about \"VAST is highly incremental because the single-modality encoders are initialized by other models\" is not appropriate.  The target of cross-modality pretraining research is to build strong connections between modalities and serve as a miultimodal generlist for downstream tasks like captioning and retrieval, rather than training single modality representations from scratch to serve for single-modality tasks. Likewisely, in CV filed most segmentation and detection methods are developed on the basis of vision backbone (ResNet or ViT), and it's not appropriate to say that all those methods are incremental.\n+ Thirdly, training those single-modality encoders from scratch will results in slower convergence and need more computation resource which is not economic and not affordable for all researchers. By contrast, utilizing pretrained models as single-modality encoders can serve as a good start point and make model focus more on cross-modality correlations learning, and this paradigm has been widely adopted by most currently main-stream cross-modality pretraining works (such as BLIP, BLIP-2, i-Code, VALOR, etc.).  \n \n**weakness 3**  \nThanks for pointing it, we conducted additional ablation experiments  with regards to the contribution of different losses. All models are trained on VAST-27M with \"V+A+S\" modalities used in both pretraining and finetuning. Due to time limitation we only train half of iterations as those experiments in Table 7 of main paper. The results are shown in the following table from which we can find that all losses (Eq1-Eq4) are important for VAST's superior performance.  The effectiveness of Eq5 has been demonstrated in the Table 7 of main paper.\n| Pretraining Loss | MSR-ret (R@1)| MSR-cap (CIDEr) | MSR-qa (Acc)|\n|---|---|---|---|\n| - | 37.6|  60.3| 44.6\n| OM-VCC  | 43.9 | 61.3 | 45.0 |\n| OM-VCC + OM-VCM | 46.5 | 63.2 | 45.6 |\n| **OM-VCC + OM-VCM + OM-VCG (default settings)** | **47.8** | **68.1** | **46.8** |\n\n**weakness 4**      \n\nDue to that web-crawled datasets have limitations in both caption accuracy and diversity, generating training data can supply more high-quality cross-modality samples for training by combining and transfering model’s learned representations into training corpus. This routine has been widely adopted in today’s models such as BLIP, BLIP-2, CLIP-VIP, and MLLMs such as LLaVa and miniGPT4. Combining multiple datasets is a also widely-used way to let model see different data distribution and present overfitting to single corpus.\n\n**question 1**  \n\nConsidering subtitles (ASR transcriptions in our work) takes also the format of text like captions, we use one single text encoder for both caption and subtitle encoding (share weights for two modalities). An alternative way is to use two separate encoders. From following experiments results we can find using shared one encoder has better performance on five benchmarks and also less total parameters.\n\n| Text and Subtitle encoders| MSR-ret(R@1) | MSR-cap (C) | MSR-qa(Acc) | YouCook2-ret(R@1) | YouCook2-cap(C) |\n|---|---|---|---|---|---|\n| **Share(default settings)** | **47.8** | **68.1** | **46.8** | **41.2**| **179.0** |\n| Seperate | 45.6| 67.8 | 46.3 | 37.7 | 175.0 |\n\n**question 2**  \n+ Firstly, the V,A,T,S represents for vision, audio, text and subtitle, following the abbreviations of Table 1. Thanks for point it and we will explain this in next version. \n+ Secondly, Eq1-Eq4 singly model the relationship between ‘VAS’ and ‘T’, however, as section 4.3 states, in practical scenario  there are many circumstances that only some of modalities but not all  modalities are available. Considering this, we additionally build the relations between V-T, A-T, VA-T and VS-T to keep training-testing consistency and let model performs well at different modality inputs (Eq 5).  \n\n**question 3**    \nContrastive loss have already been used in Eq1 (OM-VCC) to pull the global representations of matched samples together. Eq2 is the match loss which infers if a pair of video and caption matched to each other or not. Compared to VCC, VCM encourages more fine-grained interactions between caption tokens, vision or audio patches. Both losses are widely-adopted in current cross-modality researches (usually named ITC and ITM). Above experiments in Weakness 3  also demonstrates the necessity of OM-VCM.\n\n**question 4**    \nWe conduct following experiments with differnet frames utilized in pretraining. Using more frames can imporve performances a liitle bit. Considering the balance of performance and training cost, we choose to sample only 1 frame during pretraining.\n| Sampled frames | MSR-ret(R@1) | Msr-cap(CIDEr)  | Msr-qa(Acc) |\n|---|---|---|---|\n| 1 (default setting) | 47.8 | 68.1 | 46.8 |\n| 2 | 48.0 | 68.4 | 46.9 |\n| 4 | 48.0 | 68.6 | 47.1 |\n\n**question 5**    \nIndeed, in the training process loss ratio of VCM and VCC have already been equal (1:1). And at test process we first compute similarities according to VCC which is fast and then use VCM to rerank top-50 candidates computed by VCC which can further improve the retrieval accuracy, this process is commonly used in latest vision-language pretraining methods such as ALBEF, BLIP, BLIP-2."}]

Author rebuttal:
We sincerely thanks all reviewers for your recognition of our work VAST (dataset and foundation model), and very insightful reviews. Your reviews and suggestions are very important for us to improve VAST model and paper. We will add necessary detail descriptions and model comparisons in the next version of paper. With regards to rebuttal, we will firstly make an overview reply for some common questions proposed by at least two reviewers, and then make reply for others individually.

**[Common Question 1] About VAST model’s technique novelty and contribution**

VAST is the first onmi-modality pretraining model which aims at  building cross-modality connections between vision, audio, subtitle and text, and supporting both single vision-language, multi-modal video-language  and audio-language tasks. We think VAST’s innovations can be summarized as follows:

+ Under the scenarios that current multi-modal video-language models are somewhat specialist either additionally supporting audio modality beyond vision and language (i-Code [1], VALOR [2], etc) which can tackle audio-related tasks, or supporting subtitle modality (UniVL [3], MELTR [4],etc) that enhance the understanding of human speech, recipes or news, VAST is the **first** generalist foundation model that can support all vision (both image and video), audio, subtitle and text as input. 

+ VAST have **firstly** soundly proven two things through extensive large-scale pretraining-finetuning ablation experiments. Firstly, utilizing all modalities (vision, audio, subtitle) can get consistent improvement over a very broad range of different types of modality-oriented downstream tasks (Table 7 line-j vs line-e). Secondly, the multi-modality enhancement functions more apparently with the help of large-scale pretraining (Table 7 line-j&line-e vs line-d&line-a), which demonstrated the meaning and necessities for omni-modality pretraining research.

+ VAST has achieved absolutely dominated SoTA performances on 22 cross-modality benchmarks (image-language, video-language and audio-language benchmarks) which is kept till now. We think this to some extent shows that compared to further promote VLP architecture or training objectives, bringing in more modalities is a much bigger cake, but how to enjoy this cake is relatively less researched. We hope that VAST can attracts more attention over the process of "steping further from conventional vision-language pretraining towards omni-modality pretraining", and we promise that all datasets, codes and model checkpoints will be publicly released to help communities for reproduction and further research.


**[Common Question 2]  About data selection rule of VAST-27M from HD_VILA_100M.**

+ Videos in HD_VILA_100M [5] dataset covers a broad distribution of categories including music, gaming, sports, film, animals, education, entertainment, etc. After original processing in [5], each pair in the HD-VILA-100M consists of a video clip about 13.4 seconds on average and a sentence with 32.5 words on average. We choose to use a subset of HD_VILA_100M instead of using all of them mainly due to following three reasons: 1) the comprehensive consideration about the time and money cost of downloading, storage and computation. 2) those 100M clips originate from 3.3M long videos, which means that many clips are from the same video and share similar scenes. 3) many of them cannot be downloaded from YouTube.

+ In fact, we select the videos from HD_VILA_100M mainly according to following 4 rules: 1) we filtered out clips whose length are too short (less than 5 seconds) or too long (more than 30 seconds). 2) we filtered clips whose one or some modalities of vision, audio and subtitle are missing. 3) similar numbers of clips are sampled from every video in those 3.3M ones. 4) for each long video, the sampled clips are as disperse as possible to each other to ensure diversity and reduce redundancy. To the end, we got 27M video clips with an average duration of 10.0 seconds. The average lengths of vision, audio, and omni-modality captions in VAST-27M are 12.5, 7.2, and 32.4, respectively.


**[Common Question 3]  About potential data bias of VAST-27M.**

+ Due to the reason that the data collection process of VAST-27M have used a large language model (Vicuna-13b), and the training process of video and audio captioners have utilized some open-sourced cross-modality corpus such as CC4M, CC12M, LAION, VALOR-1M and WavCaps, so VAST-27M and VAST model may suffer the same bias of those dataset and bias of Vicuna-13b.

[1] Yang et al. i-code: An integrative and composable multimodal learning framework.   
[2] Chen et al. Valor: Vision-audio-language omni-perception pretraining model and dataset.  
[3] Luo et al. Univl: A unified video and language pre-training model for multimodal understanding and generation.  
[4] Ko et al. Meltr: Meta loss transformer for learning to fine-tune video foundation models. 
[5] Xue et al.Advancing high-resolution video-language representation with large-scale video transcriptions.

Meta-review: The contributions of this paper are two-fold: 1) proposing a new dataset creation pipeline and a large-scale multimodal video dataset, VAST-27M as a result, and 2) building an omni-modality foundation model that achieves new SOTA performances on 22 benchmarks.

The paper received positive scores from three reviewers (6, 5, 5) and a negative score from the other (3). The three positive reviewers recognize their technical contributions, especially on the dataset creation side. In my understanding, this is the first dataset consisting of all three modalities (vision, audio, and subtitle) and additional captions. Having this additional caption should be properly appreciated as it is critical to have another text for each example to use subtitles as a part of the multimodal inputs.

Although it is concerned that the method is highly incremental and relies too much on existing models both for building a new model and for the dataset creation, it is a common practice to use these existing unimodal pretrained models as a starting point to build a larger multimodal model. Also, using highly-performing captioning models to build a new benchmark if needed for omni-modality training can still be novel. 

Overall, the AC thinks that the merits outweigh the flaws of the paper, and would like to recommend acceptance of the paper.
Decision: Accept (poster)