InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Authors: Yi Wang
Yinan He
Yizhuo Li
Kunchang Li
Jiashuo Yu
Xin Ma
Xinhao Li
Guo Chen
Xinyuan Chen
Yaohui Wang
Ping Luo
Ziwei Liu
Yali Wang
Limin Wang
Yu Qiao
Venue: ICLR 2024 spotlight (2024)
Keywords: video-language dataset
video understanding
video generation
multimodal understanding
action recognition
video retrieval
Primary area: datasets and benchmarks
TL;DR: This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation.

Abstract: This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. InternVid contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.

Published as a conference paper at ICLR 2024
INTERNVID: A LARGE-SCALE VIDEO-TEXT DATASET FOR MULTIMODAL UNDERSTANDING AND GENERA-
TION
Yi Wang"1, Yinan He*1, Yizhuo Li Kunchang Li Jiashuo Yu', Xin Ma
Xinhao Li Guo Chen Xinyuan Chen', Yaohui Wang', Ping Luo Ziwei Liu
Yali Wang 6.1 Limin Wang 2,1 Yu Qiao
'OpenGVLab Shanghai Laboratory "Nanjing University Monash University
"The University of Hong Kong "Nanyang Technological University
Shenzzen Institutes of Advanced Technology. Chinese Academy of Sciences
https://github
ABSTRACT
This paper introduces Intern large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for mul- timodal understanding and generation. Intern Vid contains over million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is develop scalable approach to autonomously build a high-quality video-text dataset with large lan- guage models (LLM), thereby showcasing its efficacy learning video- sideo-language representation at scale. Specifically, we utilize multi-scale approach to generate video-related descriptions. Furthermore, we introduce VICLIP, a video-text rep- resentation learning model based VIT-L. Learned on Intern Vid via contrastive learning, this model demonstrates leading zero-shot action recognition and com- petitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video gen- eration research. These proposed resources provide tool for researchers and practitioners interested in multimodal video understanding and generation. |
wagon store as she walks through the store.
woren dressed in traditional costures standing in roon.
[Myytika] [aпподиссенттз] 1s
drive car out.
([Music) (Applause]]
brushing thein
older wonen sitting at table eating bowl food with chapsticks.
bathroom.
4 Shop 4220 E74
Pio122 scay
2ttie debf, T-l Rt. N PER
F12. AAREE Hundeettans, june wost
granny is grodd aeppert pircy
buy
wrything
Figure 1: Examples (we give three frames of each video clip), the corresponding generated captions, and ASR transcripts in Intern Vid. the captions, we highlight nouns in blue and verbs in green. Non-English transcripts are translated English using LLM (Brown et 2020).
* Equal contribution.
Corresponding authors.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Learning transferable video-text representations is both challenging and essential for video under- standing in various real-world applications, such as autonomous driving, intelligent surveillance, human-computer interaction, to name few. While contrastive learning using web-scale data has been successful in image-text representation, it remains underexplored in the video-text domain.
A key reason for this limited exploration is the lack of a high quality video-language dataset for pretraining at scale. Current research relies on datasets like HowTo100M (Miech et al. (019). HD-VILA (Xue al., 2022), and YT-Temporal (Zellers et al. [021] (022), whose texts are generated using automatic speech recognition (A (ASR). Despite their large scale, these datasets often have low semantic correlations between the videos and corresponding textual descriptions (Miech et al. 2019)
Xue et al. 2022, Zellers et al., 2021 2022). Empirical studies demonstrate that improving this correlation (e.g. aligning videos with subtitles to improve their matching) significantly benefits downstream tasks such as video retrieval and video question answering (Bain et al. (021). Recent works have utilized WebVidIOM (Bain et al. 202T), a dataset with higher-quality alt-texts, to address the low video-text correlation issue. However, its limited scale and dynamics hinder its use in current data and model scaling studies. Specifically, only 10M video-text pairs are provided, and the depicted scenes contain relatively few actions activities.
We propose a large-scale video-centric dataset Intern Vid to address the challenge of scaling up video- language modeling while maintaining high video-text correspondence. Visual examples given in Figure [I Note the ASR transcripts barely depict visual elements in videos while the generated captions do. The dataset contains highly-correlated video-text pairs and includes over 7 million videos, totaling 760, 000 hours and resulting in 234 million video clips, with various subsets for different needs. These videos cover 16 scenarios and around thousand motion descriptions. To improve video-text matching, we generate captions using multiscale approach. In the coarse scale, we caption the middle frame of each video and use the description as the video caption. the fine scale, we produce and summarize frame-by-fine
captions with language model.
Leveraging Intern we scale a video-language transformer (ViT-L) in contrastive learning from a data perspective, and its experiments prove Intern enables learning scalable video-text models. We introduce video masking to the model to accelerate the whole learning without compromising its effectiveness. The video and text encoders same scale. With Intern Vid, we learn a video-text model for several epochs, achieving impressive zero-shot performance. Compared with previous Video CLIP variants, our proposed ViCLIP shows notable performance improvement, especially in zero-shot settings.
initialized from the CLIP pretrained model with the
In addition to large-scale video-language contrastive pretraining, we discover its effectiveness in producing interleaved video-text data for learning a video-centric dialogue system like Flamingo (Alayrac et al., 2022) Awaadalla et al., 2023), and advancing video generation. Since the text-annotated clips are extracted from videos, we naturally collect clips and their corresponding text based on the sampling locations. This results approximately 7 million interleaved data pieces, suitable for instruction tuning as multi-turn video-centric dialogue. For video generation, we filter the core set and obtain 18 million video clips. Alongside Web Vid- 10M, Intern Vid can significantly improve a based video generation model to new heights.
In summary, our contributions are threefold.
We introduce a new web-scale video-language dataset Intern Vid. This dataset, aimed at advancing video-related multimodal understanding and generation at scale, is created using a multi-scall scale video captioning approach powered by LLM, ensuring high-quality video-text data with minimal human intervention. includes computational features (video-text correlation and visual aesthetics) across the whole dataset and gives way to diverse subsets to cater to varying training needs.
We learn new video- langu-ln model, ViCLIP, which is trained on Intern Vid using VIT-L. It incorporates both contrastive learning and mask modeling, allowing for efficient learning of transferrable video-language representation. This model achieves state-of-the-art zero-shot action recognition in Kinetics, scoring 75.7, 73.5, and 66.4 on K400, K600, and K700 with the average top1 and top5 accuracies, respectively. It gets competitive performance on video retrieval, setting a new baseline for video-text understanding.
Intern Vid fosters the development of multimodal dialogue systems and fext-to-video generation. ViCLIP learned on Intern Vid could serve as a backbone of video-centric dialogue systems (Zhu et al.
2023a Li et al. 2023c Liu et al. 2023). conducting tasks as action recognition, temporal
2

---

Published as a conference paper at ICLR 2024
Dataset
Caption
Domain
#Vidoos
#Clips
Leactip
Lencc
Durth)
Ros
MSR-VTT (Xu al.
2016
Manual
open
7.2K
10K
15.0
9.3
411
240P
DideMo
Anne Hendroks et al.
2017
Manual
Flickr
10.5K
27K
6.9
8.0
87
LSNDC
Kootrbcc et a.
201
Manual
movie
200
118K
4.8
7.0
158
1080P
YouCookZ hohou
2018
Manual
cooking
2K
14K
19.6
8.8
176
How2
Sanatera et m1.
2018
Manual
instruct
13.2K
80K
90.0
20.0
2K
ANet Caption
Knshia 2t 0.
201
Manual
action
20K
100K
36.0
13.5
849
VideoCC3M
Naurani
2022
Transfer
open
6.3M
10.3M
10
17.5K
WebVid
Baam et al.
2021
Alt-text
open
10.7M
10.7M
18.0
12.0
52K
360P
WTS70M
Strond 0E 2020
Metadata
action
70M
70M
10
194K
10WT
Miech
2019
ASR
instruct
1.2M
136M
3.6
4.0
134,5K
240P
HD-VIL.A -O0MA-100M
[uue of al.
2012
ASR
open
3.3M
103M
13.4
32.5
371,5K
720P
YT-Temporal- 180M
(Zelers 2f al.
2021
ASR
open
6M
180M
(ours)
Generated
open
7.1M
234M
11.7
17.6
760.3K
720P
Table 1: Statistics of Intern Vid and its comparison with existing video-language datasets.
understanding, reasoning, and creativity within an open-ended environment. Furthermore, we provide subset, Intern Vid-Ae, aiding in generating high-resolution watermark- videos. Utilizing Vid-As, both visual and quantitative outcomes of a text-to-video baseline can be noticeably enhanced (FVD: 705.3 - 616.5).
2 RELATED WORK
Multimodal Datasets.
Vision-text data are necessary to enable crossmodal learning. To learn vison- language representation effectively, these datasets should be large at scale and high at vision-text correlations. To this end, researches usually leverage existing web images with alt-text (Thomee et al.)
2016 Sharma al., 2018 Changpinyo et al.) 202 ||
Hu et al., 2022 Desai et al.] 2021 Schuhmann
ct al.
2022) and videos with ASR transcriptions (Miech et al. 2019 Zellers et al. 2021 2022 Xue
al.
2022 Bain al. 202 Srinivasan et al. 2021) for scalable learning.
For video-centric multimodal datasets, How To (Miech et al.) 2019) collected instructional
YouTube videos and exploited the corresponding ASR subtitles for learning joint representations.
Nagrani et al. (2022) proposed VideoCC3M by transferring image-text datasets to video ones. Zellers
atal. (202 (2022) and Xue al. | (2022) proposed YT-Temporal and HD-VILA for Audio-Visual-
joint learning and high-resolution video crossmodal learning, respectively. On the other hand, Bain
al.
[202I) found video-text alignment matters more than their quantities, so they produced Web Vid
(Bain et al. 2021 where 10M videos with the corresponding alt-texts. This is frequently employed
in recent Video-language pretraining approaches (Li et al., 2023d Cheng et al. 2023).
Video Understanding. Pretraining large-scale video-text models and fine-tuning them for down- stream tasks has become the norm in the video-language field (Miech et al.
2020
[Li Wang 2020
Xu et al.) |021 Li et al. 2023d, 2022al Xu et al., 2021 Hu et al. 2022, Dou et al.
2022; Shen et al.
2021 Yao et al. 2021 Sun et al. 2019 Zhu & Yang 2020 Wang et al. 2022b
Chen et al. 2022
Zellers et al. 2021 2022 Zeng al. 2222b al Chen et al. 2023a bj He al.
2023 Chen et al.
2023ch. Early techniques (Sun et al. 20199/Zu & Yang. 2020) 2020)
used pretrained visual and language
encoders obtain offline video and text features, but recent methods (Li & Wang 2020 Miech et al.)
2020 Hu et al. 2022; Dou et al. 2022; Tong et al. 2022 Wang et al. 2023) highlight the advantages of end-to-end training. Common practices include two three pretraining tasks, such as masked
language modeling (Li et al. [2022)). video-text matching (Wang et al.)
2022a). video-text contrastive
learning Xu et al. 2021] Wang et al. 20226), masked video modeling (Tong et al. 2022 Wang et al.)
2023 2022b), and video-text masked modeling (Fu et al. (02T).
In the multimodal video context, VIOLET (Fu et al., 202T) combined masked language and video modeling, while All-in-one (Wang et al., (022a) proposed a unified pretraining with a shared backbone. LAVENDER (Li et al., 2022b) unified tasks through masked language modeling. Despite their success in multimodal benchmarks, these methods' reliance on limited video-text data hampers performance in video-only tasks like action recognition. Conversely. Intern Video (Wang et al. 2022b) and UMT (Li et al., 2023d) combined masked modeling with crossmodal contrastive learning. leading to competitive performance in both video-only and video-language tasks. MERLOT Reserve (Zellers) al. 2022) exploited 20 million video-text-audio pairs for training joint video representations using contrastive matching, setting new standards in video recognition and visual commonsense reasoning.
To address modality entanglement crossmodal learning. mPLUG-2 (Xu et al. (023) introduced shared module across image, video, and text to encourage modality collaboration while reserving modality -specfffc modules for their differences.
3

---

Published as a conference paper at ICLR 2024
Image Captioning (DLIP2, Tag2Text)
a person cooking food on top
the food is prepared and ready be
person cooking some food in a frying
table topped
person putting food in basket
with lots of food
Language Model
person cooks various dishes an stove top and Fryinng pan, prepares the food, sets table with lots food, and sone of it in basket.
Figure 2: The proposed multiscale video caption pipeline. The captions in coarse and fine scales marked in green and blue, respectively.
3 INTERNVID: A VIDEO-CENTRE MULTIMODAL DATASET
A high-quality tixe dataset at scale is premise to conduct acaee-scale video-language learning and associated tasks. We build this dataset considering three factors: substantial temporal dynamics, rich and diverse semantics, and strong video-text correlations. To improve the temporal dynamics in the dataset, we gather videos retrieved using action/activity--nn varied semantics, not only crawl trending videos across various categories but also deliberately increase the proportion of data consciously collected from various countries and languages. To strengthen video-text correlations, employ image captioning and language models to generate video descriptions from specifcc annotations. Next, we elaborate the dataset construction process and discuss its characteristics.
query words. For rich and
3.1 DATA CURATION
We collect videos from YouTube considering the diversity and richness of its data, and its support for academic usage. We obtain million public YouTube videos with an average duration of 6.4 minutes, covering 16 topics. To avoid the overlap with the existing datasets, we acquire videos by creating a database of Yobeube video IDs and excluding any videos already present in publicly available datasets (released prior April 2023). The data curation strategies are two-fold. On one hand, We select popular channels and the corresponding hot or high-rated videos from the categories e.g. news, gaming, etc., resulting in million videos. On the other hand, we create list of motion descriptions. With it, we obtain 5.1 million videos by choosing the top retrieved ones.
Defining Actions for Queries. We define around 6K action phrases from American Time Use Survey (ATUS), public video datasets. and text corpus. Then they refined both manually and automatically. We employ actions from ATUS from 2017 to 2022 (Heilbron et al. (015)). merging them and removing the duplicates. For the used video datasets, Zisserman 2017). SomethingSomething (Goyal et al.) 2017] Mahdisoltani et al.) 2018). UCF10T (Soomro et al. 2012), and on. This provides with 1003 action labels. Moreover, we access several visual grounding corpus (Song et al.) 2021 I Yang et al. 2022 Li al. 2017). A language model (Brown al. 2020) employed extract actions and their corresponding targets (if exist) to form phrases from the corpus, leading to 5001 actions with manual checking. Totally, we collect 6104 action queries for searching videos.
leverage Kinetics (Carreira &
Collection Strategies. For quality control, we establish crawling rules. We collect videos that are between 10 seconds and 30 minutes in duration and have resolutions ranging from 360P to 720P. Videos with resolutions above 720P are processed to 720P. To enrich the dataset descriptions, we gather videos with their audio, subtitles, titles, summaries, and other metadata.
Trimming, We segment videos into clips using scene variance. We directly employ the corresponding filter in DeSceneDetect with a threshold as 27. During this procedure, we also filter out clips in still extreme dynamics (e.g. browse of photo gallery). After the filtering, we get 234M clips.
3.2 MULTISCALE VIDEO CAPTIONING
To generate video captions that are scalable, rich, and diverse, employ multiscale method with two distinct captioning strategies, as depicted in Figure 2 On the finer scale, we simplify the video captioning process by concentrating the common objects, actions, and scene descriptions within the video clip. We deliberately overlook intricate details such as subtle facial expressions & movements, and other nuanced elements. On the coarser scale, we adopt the single-frame bias assumption from
4

---

Published as a conference paper at ICLR 2024
Video Countries
Video Durations
Video Categories
People Bllngr
1.599
855
1756
B Smznn 10min
29%
Bowws Style
18%
cm
E1
10aii-1 15mh
5%
Garning
26%
9%
14%
Selencee Technology
10%
20min
B Other slanguages
Btther 11 Cutegeries
Figure 3: Video statistics in Intern It encompasses a diverse set of categories, gathered from multiple countries and averaging a duration of five minutes.
Clip Durations
Clip Caption Length
UMT-SIM & Aesthetic
0.5
14%
Oveende
03
10 20wsrds
8:00-20weete
10.00-22..0
Z6N% BB:00e
Figure 4: Clip statistics in Intern Vid. Intern Vid contains diverse distribution of clip durations and caption lengths. It also offers aesthetic scores and multimodal similarity scores for each clip.
eL al. 2022) and exclusively caption the central frame of the video. Given focus brief clips (around 10 seconds) filtered via scene segmentation, most videos predominantly display consistent objects without substantial appearance alterations. This cireumvents the identity-prest preserving issue when dealing with videos from image perspectives. We employ the lightweight image captioning model Tag2Text (Huang et al. 2023b) for the finer scale, which describes videos low fps in trame-by-frame manner. These individual image captions are then synthesized into a comprehensive video description using a pretrained language model T5 T5-summary (Raffel et al. 2020 Chiang et al.)
2023).
At the coarser scale, we use BLIP2 (i ei al. 2023b) to caption the middle frame of the clip.
3.3
STATISTICS AND FEATURES
We present the statistics of Intern Vid with other video-language datasets in Table We collected videos from 16 mainstream categories, as in Figure Unlike prior studies (Miech et al. 2019) Xue al. 2022 Zellers et al. 202T), we emphasize diversity by selecting videos from countries with different languages instead of only English. In terms of duration, every video lasts 351.9 11.9 on average. Almost half (49%) of the videos are five minutes or less, while quarter (26%) fall between five and ten minutes. Among the curated videos, 85% were high-resolution (720P), while the remaining 15% had lower resolutions ranging from 360P to 720P. Though the lower-resolution vidcos may not support high-quality video generation, they can still be useful in video-text representation learning.
Intern Vid exhibits diverse clip durations and caption lengths at the clip level. The aesthetic scores and clip-caption similarities distributed uniformly, shown in Figure The majority of clips 0-10 seconds in length, accounting for 85% of all clips (Figure left). Approximately half of the clips have captions with 10-20 words. The statistics of the captions and transcripts is given in App. C
We measured the aesthetic scores and clip-caption similarity of all clips using the models in (Schuh mann et al., 2022) and (Li et al. 2023d), respectively, given in Figure right. Based on these scores, we can build different versions of Intern Vid for various purposes. We uniformly sampled four frames of each clip, calculated their aesthetic scores, and took the maximum score as the video aesthetic score. For clip-caption similarity, dings and text ones, again using a uniform sampling of four frames for each clip. Most clips score around 4-6 in aesthetics, accounting for approximately 75% of the data. For UMT-SIM [ over 80% of the clips scored between 0.3-0.4.
computed the cosine similarity between video embed-
3.4 INTERLEAVED TIXEO-TEXT DATA GENERATION
Utilizing video-caption data, we can develop an interleaved video-text dataset for generative mul- timodal pretraining, Previous researches (Alayrac et al. 2022;] Awadalla et al. 2023 Huang et al. 2023a Zhu et al. 2023b) confirm that pretraining on the interleaved image-text sequences results in significant multimodal in-context abilities. Our work makes the initial step in creating a large-scale
'UMT-SIM refers to the use of Unmasked Teacher (UMT) [Li et 2023d) to compute the similarity score hetween given video clip and the accompanying text.
5

---

Published as a conference paper at ICLR 2024
Timestamp
Timestampp
(8)
(b)
Timeslampp
Timeslamp
2
Figure 5: Interleaved video-text data generation in Intern Vid with three formats.
Time
Vision Transformer
Align
Video
Image
Text Transformer
people standing outside log cabin.
Text
Figure 6: Framework ViCLIP.
interleaved video-text dataset Intern Vid-ICL, containing 7.1M interleaved video-text data pairs. We
give three methods for organizing clips and their captions: (1): Arrange clips and their descriptions sequentially based on their temporal order within the same video, as illustrated in Figure (a). (2): Enhance diversity in interleaved video-text items by assigning ASR text to used clip in addition to its caption, as demonstrated Figure (b). (3): Extend method (1) by concatenating two interleaved multimodal items, creating video-centric dialogue simulating user queries involving multiple videos (Figure (c)). One visual example of these arrangements is provided in Table S
4 VICLIP: LEARNING VIDEO-TEXT REPRESENTATION AT SCALE
Built upon CLIP (Radford et al. 2021 we make a simple video-text pretraining baseline VICLIP. It consists of a video encoder (ViT) Dosovitskiy et al. 2021) and a text encoder, as given in Figure Both modules are initialized from the corresponding CLIP components. We update the native attention in the video encoder to spatiotemporal attention while maintaining other designs. For efficient learning, we apply masking to videos pre-training. The optimization target the contrastive loss between input video and text embeddings.
Video & Text Encoders with Masking Learning. Our video encoder uses a ViT with spatiotemporal attention. We apply random patch masking following MAE-based methods (Tong al. 2022 Wang
et al. 2023) to the input videos. It signfiicanllyy alleviates the computational burden. The used text encoder also transformer followed by (Radford et al.) 2021 Schuhmann al. 2022).
Unmasked Video- Text Pretraining. We feed all visual tokens into the video transformer instead of just the masked ones towards the end of the pretraining process. This helps bridge the gap between pretraining and downstream applications where the full video is used as input. We perform unmasked training for 0.5 epochs with learning rate of 4e-6.
Training Objectives. Our framework optimizes video-text alignment. It minimizes InfoNCE. loss (Oord et al. 2018) using global video and text features, as
exp(sim(f.) . ffF)/T Σ log Σ exp(sim(J) fF//T)
N
exp(sim(fF ,f,,, f/T/
Lc Cg C TT-V
log Σ exp(sim( (f)))
(1)
where /V and fT denote the learned video and text embeddings, respectively. sim(-) computes the cosine similarity between features. the learnable temperature.
EXPERIMENTS
We learn VICLIP subsets of Intern Vid and evaluated its performance video-related benchmarks using full-fine finetuned and zero-shot settings. We sample subsets Intern 10M, Intern Vid-50M, and Intern Vid-200M randomly. Sampling details given in App. E.T

---

Published as a conference paper at ICLR 2024
Method
Training Data
K4000
KO00
K700
top-1 (?)
AVG (1)
top-1 (f)
AVG (1)
top-1 (1)
AVG (1)
CLIP Radioni et al. 2021
CLIP400M RRdford al. 2021
58.42
70.14
$5.11
67.16
46.12
58.38
CLIP Radllor
2021
DataCong- 1B (Gailre
56.14
67.67
54.15
65,83
45.36
57.01
EVA- CLIF-L
Sun c
2023
Mergod-2IB el A025
65.00
64,90
59.10
EVA-CLIP
LAION- 28 Schulmann
2022
69.80
69.30
63.40
VILIP
+WebVid10ST Banm et al 2021
59.88
71.03
58.66
69.84
50.23
61.86
VICLIP
+Inler 10M
56,68
68.17
54.67
66.28
46.53
58.73
VICLIP
+InlemVidd-00)
57.18
68.93
$5.36
67.07
47.00
59.36
VICLIP
+InterVii 200M
59,80
71.09
57.80
69.34
49.30
61.25
VICLIP
+latemVVdd110M -I---II
63.00
74.15
60.68
72.07
52,50
64.59
VICLIP
+lotemVVid110M 10M-FLT
64.80
75.70
62.20
73,53
54,30
66.38
Table 2: sero-shot action recognition results Kinetics 400/600/700.
Method
Training Data
top-1
K400
top-5 (t)
top-1 (t)
SthSthV2
top-5 (1)
CLIP (Radford et
2021
CLIP400M (Radford al.
2021
86.7
97.2
70.1
92.5
CLJIP (Radfford (KaPfoRa
et al. 2021
DataComp- TH (Gndre et
202
85.6
96.8
68.9
91.8
ViCLIP
+Web Vid (Bai et al.
2021
85.0
96.8
68.7
91,9
VICLIP
+Inter Vid-IoN--TTP JOM-FLI
86.8
97.3
71.2
93.2
VICLIP
+Inter Vid- Vid--oM---I---- 10M-FLT+K710
88.0
97.8
71.8
93.6
VICLIP
+Inter Vid-200M
87.9
97.9
73.6
94,9
VICLIP
+Inter Viddd-000--K7777
88.7
98.2
74.2
95.0
Table 3: Fine-tuned action recognition results on Kinetics 400 and SomethingSomethingy2.
5.1
TRANSFERABLE VIDEO REPRESENTATION PERFORMANCE
Action Recognition. In addition to OpenAl's CLIP-L (CLIP400M (Radford et al. 2021)) and LAION
DPataComp111 (Gadre al., 2023)), we also include EVA-CLIP-L/14 and E EVA-CLIP-E/14 (Sun)
ct al. 2023) for comparison. More experimental settings are given in App. E.2
Zero-Shot. Table shows that when trained on Intern Vid-10M-FLT, VICLIP outperforms all other methods, including EVA-CLIP This result validates Intern Vid's effectiveness in learning video- text embeddings. Note that ViCLIP with Intern 10d-IOM-FLT sets new records on zero-shot action recognition in Kinetics 400/600/700, demonstrating significant performance boost compared to ViCLIP with VebVid or other models. Moreover, VICLIP trained on Intern Vid- 10M-IO0-FLT exceeds its performance on Intern Vid-200M. Normally, we would expect the model trained on Intern 2i0M to perform better than those on -10M-DVV or -LLT, given that the latter two subsets derive from the former. Unless this discrepancy results from improper lcarning, false negative samples could severely impede video-text contrastive learning if we don't 't purposefully reduce the number of clips taken from the same video. Specifically, hypothesize that clips from the same video share similar representations and captions. Contrastive learning, however, assumes these clips to be different. This situation also undermines the significance of using a large batch size in current training since it increases the probability of encountering more false negatives.
conjecture that
Fine-tuned. In Table note when comparing ViCLIP trained on Intern Vid with image CLIP models or ViCLIP trained with Web Vid, there is clear increase in accuracy. Unlike the zero-shot results, when ViCLIP is pretrained with a larger number (200M) of video-text data pairs, achieves higher accuracy in fine-tuned recognition tasks (87.9% 9%% in K400 and 73.6% in SthSth compared to when pretrained (86.8% K400 and 71.2% in SthSth V2) with fewer data (10M). This suggests that Intern Vid provides greater benefits for fine-tuned action-relater tasks. The decrease in performance of ViCLIP with WebVid highlights the importance of addressing the distribution gap between WebVid and the action videos used for evaluation, emphasizing the need to collect videos with evident temporal dynamics.
Video-Text Retrieval. We evaluate the video retrieval performance of baselines and VICL.IP using different pretraining datasets on five popular benchmarks (Heilbron al., 2015 Xu et al. 2016 Rohrbach et al., 20155 Anne Hendricks et al., [2117] Chen & Dolan 2011). ), as shown in Table and We uniformly sample eight frames from the input videos. For the CLIP models from OpenAI (Radford et al. 2) and LAION (Schuhmann et al., (022). we utilize their officially released ViT-L models and extract video embeddings by averaging the computed frame-win wise image embeddings. Our ViCLIP directly predicts video embeddings. For evaluating retrieval performance, report R@/ scores for both texi-lo-video (12v) and video-to-text (v21) tasks in 4 [
4 and demonstrate that video-language pretraining is crucial for enhancing fine-tuned and zero-shot retrieval performance. This point is substantiated by the comparison between CLIP and ViCLIP using Intern Vid- 50M. Table 4 exhibits Aexhibiss boost of nearly 4- 10 points across different benchmarks in the zero-shot setting. Mcanwhile, Table [ shows an increase of approximately 10 points across all R @ scores in the fine-tuned setting.
Both Table
7

---

Published as a conference paper at ICLR 2024
Method
Data
MSR-VTT
LSMDC
DiDeMo
MSVD
ANot
T2V
VZT
T2V
V2I
12V
VII
T2V
V2T
T2V
VIT
CLIP Radrord et al. 2021
CLIP400M [Raford et
2021
29.0
25.8
13.9
13.2
11.5
19.1
37,9
60.0
8,3
12.2
CLIP Redlord
2021
DataComp- 13
13,9
11,9
40.5
9.1
13.2
CLIPP4C.0pp
1i0 et m 2722
HOWT
Miech el m
2019
32.0
15.1
38.5
VICLIP
+WebVidIUM
et
16.5
13.4
45.3
12.4
19.0
VICLJP
IntemViid100.
36.4
37.1
17.1
15.0
16.4
25.9
45.2
69.8
13.5
23,4
VICLIP
+iniemVid 50M
39.7
40.7
18.0
16.7
16.7
26.4
46.5
72.2
13.6
23.2
VICLIP
Viteenid-2200
39.3
39.5
18.3
16.6
17.1
25.5
47.3
70.0
13.7
21.6
VICLIP
41.5
41.6
18.5
17.4
17.7
26.2
48.6
71.9
14.8
23.4
VICLIP
IIM-IUM
20.1
27.9
49.1
15.1
24.0
Table 4: Results of zero-shot video retrieval on MSR-VVRRR VTR, LSMDC, DiDeMo. MSVD, and ANet.
MSR-VTT
LSMDC
DiDeMo
MSVD
ANet
Method
T2V
V2I
12V
V2T
T2V
V2I
T2V
VZT
12V
VZT
CLIPP Radford et 2021
CLIPP40M (Radford al..
38.7
22.5
32.2
33.9
69.9
26.1
CLIIP Radford al 20ZT
DataComp-Ie 18
Gädre et
37.2
37.5
18.7
18.5
33.5
34.2
66.3
70.2
24.5
25.8
CLIP4Clip Lub Lub
2022
+How To100M
Mizcn 0
2012
45.6
45.9
24.3
23.8
43.0
43.6
45.2
48,4
40.3
41.6
VICLIP
4 WichVid (OM
Baih ét
50.8
49.3
27.3
28.4
48.1
48.5
76.7
81.2
44.5
43.2
VICLIP
Vid
51.8
49.7
28.5
29.4
49.5
50.6
77.2
80.0
49.7
48.4
VICLIP
52.2
30,9
49.4
48.7
80,0
VICLIP
SantemVid-20MM.
53.7
53.4
29.3
31.3
51.1
50.8
79.9
78.4
52.8
51.1
VICLIP
Incerm Vid-1OM-DIV
55.0
53.3
32.0
30.0
51.7
52.1
75.8
77.8
50.4
48.9
VICLIP
+Inter VI01MM-LL
52.5
518
33,0
32,5
49.4
50.2
77.2
79.0
49.8
48.1
Table 5: Results of fine-tuned video retrieval on MSR- VTT, LSMDC, DiDeMo, MSVD, and ANet.
Zero-Shot.
Intern InternVid- Vid- 10M outperforms WebVid when employing the same method, ViCLIP. with an average increase of 6.3% in R@ across nearly all benchmarks. This improvement can be further amplified by diversifying the training clips used, as Intern Vid-1OM-DIC and -FLT surpass Web Vid on ViCLIP with gains in R@ of 14.0% and 17.1%, respectively. These results underline, once again, the effectiveness of the correspondence between our generated video captions and their corresponding videos. Comparing CLIP4Clip using HowTo100M with ViCLIP using Web- VidIOM or Intern Vid-10M shows that the correlation between video and text influences performance more significantly than their quantity. Moreover, the zero-shot performance demonstrates that the video-text representation learned using Intern Vid is transferable. This claim is supported by its superior performance across multiple video retrieval benchmarks.
Fine-Tuned. Table 5 exhibits noticeable improvement when transitioning from Intern Vid- 10M to WebVid OM while using VICLIP for both 12v and v2t retrieval across almost all datasets. On average, there is 3.7% increase in 12v R @ 1 across all benchmarks, with particularly significant rise observed in ActivityNet (an increase of over 11.9%). However, VICLIP using Web Vid yields better v2t R @ scores than when using Intern 10M (81.2 vs. 80.0). We believe this does not alter the overall trend that Intern Vid- 10M generally provides more advantage to ViCLIP than WebVid10M does.
The benefits of used video data become even more apparent when comparing Intern 1id-100MDIV or Intern Wid-10M-FLT with WebVid10M. Their overall increases are 5.8% and 5.1 5.1%, respectively. Despite these improvements, issues related to data diversity persist.
Data Scaling and Issues. Figure 7 and 8 illustrate how ViCLIP's performance changes in zero- shot and fine-tuning settings when varying the scale of Intern Vid. In both scenarios, increasing the data scale results in significant increases in performance. As shown in Figure 7, ViCLIP's discriminative ability linearly increases with the increasing volume of training videos used (10M -> 200M). Meanwhile, Figure 8 shows that the retrieval performance increase becomes marginal when scaling the training data beyond 50M. It's vital to note our model is trained using only contrastive loss without employing popular designs such as matching head and its corresponding loss. Consequently, this retrieval result doesn't 't allow for any definitive conclusions about whether there exists turning point after which scaling up the training videos becomes less beneficial currently. More explorations are necessary in these retrieval experiments. However, these findings generally suggest that enhancing the scale of pretraining data can improve the transferability of the learned representation.
5.2 OTHER APPLICATIONS
Toxt-to- Video Generation. Our Intern Vid dataset improves existing text-to-video (12v) generation models by providing high-quality video-text pairs. We extend spatiotemporal modeling on the latent space of text-to-image diffusion model (Rombach al. 2022) a t2v baseline. We train the baseline with two settings: one using Web VedVid,OM, and the other using Intern Vid-Aes- Aid-Aes-18M in addition to Web Vid OM. Intern Mid-Aes-18MM is subset of Intern consisting of clips with aesthetic
8

---

Published as a conference paper at ICLR 2024
60
$4.0
Fin-Ttred
20.0
58
535
273
56-
530
210 18
K40P Kt0p
52.5
26.5
K700
52.0
26.0
515
25.
51.0
3080
2008
10M
S0M
200M
Figure 7: Zero-shot action recognition (top-1 ac- Po-Fugure 8: Video retrieval average performance (text-to-video R@ across five datasets.
curacy) on Kinetics-400/ 16000
UCF- I01
MSR-VTT CLIPSIM (t)
Method
Training Data
(t)
(L)
FVD
VideoCrafter
WehVidIOM
Bain al.
2021
18.26
66.95
910.87
0.2875
VideenFusion
WehVidIOM
Bain
2021
17.49
75.77
639.90
0.2795
12v baseline
WehVidIOM
Buin al.
2021
13.97
98.25
705.25
0.2657
12v baseline
WodVid10MFLilta Vidis
21.04
7.07
60.25
a8. no
616.51
88.74
0.2951
10.0244
Table 6: Zero-shot text-to-video generation performance.
score of at least 4. Quantitative (Table and qualitative (Figure 12 evaluations demonstrate the effectiveness of Intern in video generation tasks. Evaluation protocols are given in App. E.3
In Table we observe that (2v baseline trained on WebVid10M performs poorly in terms of IS, FID, and CLIIPSI when compared to other approaches. However, with the addition of Intern InternVid-Aed. Vid-Aes, t2v baseline demonstrates significant improvements in these metrics and outperforms other methods by considerable margin. In Figure 12 we observe that the 12v baseline using both WebVid10M and Vid-Aesi-- significantly outperforms others in visual quality and temporal coherence. These results demonstrate the potential of Intern Vid for nigh-quality video generation.
Video-Centric Dialogue System. Inspired by recent vision-centric dialogue systems (Li et al. 2023c MMhammad Maaz & Khan 2023 Li al., 2023a),, we integrate our pretrained VICLIP (with Intern Vid) into VideoChat (Li et al. 2023c) to show how our data and model can empower multimodal dialogue methods with effective video modeling capability. In implementation, we inherit just replacing its visual encoder with our ViCLIP (trained on Intern Vid). We evaluate VideoChat- ViCLIP in spatial understanding (Figure 13) action recognition VideoChat-ViCLIP (Figure 14), temporal understanding (Figure 15), video reasoning (Figure 16 and video creative
nearly all designs of VideoChat-Emin
(Figure tasks. Our qualitative evaluations demonstrate decent video-to-tex capabilities.
Correctness of Information
Detail
Contextual
Temporal
Evaluation Aspect
Consistency
Avg
Orientation
Understanding
Understanding
VideoChat (Eva-g)
2.23
2.50
2.53
1.94
2.24
2.29
Video-ChatGPP
2.52
2.62
1.98
2.37
2.38
VideoChat- ViCLIP
2.86
2.52
3.08
2.36
2.40
2.64
Table 7: Performance benchmarking text generation models.
In terms of quantitative comparison, as shown in Table 5.2 VideoChat-ViCLIP notably outperforms
the vanilla VideoChat (using Eva-g as the vision encoder) and others across all evaluation aspects of the video conversation evaluation in the work of Muhammad Maaz & Khan (203).. Specifically, the model shows remarkable improvements in the correctness of information (from 2.23 to 2.86), contextual understanding (from 2.53 to 3.08), and temporal understanding (from 1.94 to 2.36). The average score also increases from 2.29 to 2.64, showing an overall performance gain.
CONCLUSION
Our dataset, Intern is designed for multimodal research (both understanding and generation) focused on videos. It consists of over 200 million video clips sourced from million high-resolution YouTube videos. We use existing models with a multiscale approach to generate clip-level descrip- tions. Our studies confirm the efficacy of captions, and the large volume of video-text data enables crossmodal learning and text-to- video generation at scale. By training with our data, we develop video-text representation baseline ViCLIP using ViT-L and analyze briefly how the data scale affects learned crossmodal embeddings. In addition perception tasks, we show that Intern Vid improves text-to-video generation and supports multimodal dialogue systems. With its data, annotations, metadata, and computed scores, we believe Intern Vid can fuel variety of studies and applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is partially supported by the National Key R&D Program of China (No. 20222D0160101,, National Natural Science Foundation of China (No. 62076119, No. 61921006). the Science and Technology Commission of Shanghai Municipality under Grant No. 23QD1400800 400080 and No. 23YF1461900 and Ministry Education, Singapore, under its MOE AcRF Tier (MOE- T2EP20221- 0012) and NTU NAP.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, abs/2204.14198, 2022.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. ICCV, pp. 5803-5812, 2017.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, et al. Openflamingo, 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In ICCV, pp. 1728-1738, 2021.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pp. 22563-22575 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877-1901, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text text pre-training to recognize long-tail visual concepts. In CVPR, pp. 3558-3568 2021.
David L Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies- Volume I, pp. 190-200. Association for Computational Linguistics, 2011.
Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin- Dong Zheng, Bingkun Huang, et al. Internvideo-egoAte pack of champion solutions ego4d challenges. arXiv preprint arXiv: 22/ 2 1.09299, 2022.
Guo Chen, Yin-Dong Zheng, Jiahao Wang. Jilan Xu, Yifei Huang, Junting Pan, Wang, Yali Wang, Yu Qiao, Tong Lu, al. Videollm: Modeling video sequence with large language models. arXiv preprint XrXiv:2305. 2tti22355..2222 13292, 2023a.
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: Vision Vision-audio- ludio-language omni-percephl pretraining model and dataset. arXiv preprint
0.X0:22304.083455 2023b.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio suuii-subtitle-text onni-modality foundation model and dataset. arXiv preprint arXiv:2305. 18500, 2023c.
10

---

Published as a conference paper at ICLR 2024
Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. Vindlu: A recipe for effective video-and-lang and-language pretraining. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 10739-10750, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez. lon Stoica, and Eric P. Xing. Vicuma: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https
7/1 //1mays..org http:/...r///looo /ttpollgg22222222 2023-03-2222 00330-3--000000
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. NeurIPS, 35:16 35:16344-16359 2022.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.1143/ 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-language transformers. In CVPR, 2022.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint wrXiv:2/11.1266 2021.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint DrXiv:2304.14/00 2023.
songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pp. 102 118. Springer, 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pp. 5842-5850, 2017.
Xingjian He, Sihan Chen, Fan Ma, Zhicheng Huang, Xiaojie Jin, Zikang Liu, Dongmei Fu, Yi Yang, Jing Liu, and Jiashi Feng. Vlab: Enhancing video language pre-training by feature adapting and blending, arXiv preprint arXiv:2305. 13167, 2023.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE conference on computer vision and pallern recognition (CVPR), pp. 961-970. IEEE, 2015.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR, pp. 17980-17989, 2022.
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv. Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023a.
Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vasion-language model via image tagging. arXiv preprint trXiv:2303.05657. 2023b.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, pp. 706-715, 2017.
11

---

Published as a conference paper at ICLR 2024
Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for video-and arr--annd-lnnguuge learning. arXiv preprint arXiv:2206.03228, 2022.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu, Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint Siti:::306.0522233 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang. and Yu Qiao. Uni- formerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:2211.09552, 2022a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 2023c.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang. Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efffcient video foundation models. arXiv preprint 16055 2023d.
Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Laven- der: Unifying video-language understanding as masked language modeling. arXiv preprint 00620, 2022b..
Tianhao Li and Limin Wang. Learning spatiotemporal features via video and text pair discrimination. CoRR, abs/2001 1.0691, 2020. WRLLhttps:// http://axxx....gg/eae/ /ttpgg//bb///001. 2001. 05691
Zhenyang Li, Ran Tao, Efstratios Gavves, Cees G. M. Snock, and Amoll W. M. Smeulders. Tracking by natural language specification. CVPR, 2017.
Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang. Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Internchat: Solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint arXiv:2305.0566 2023.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang. Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In CVPR. pp. 10209-10218, 2023.
Farzaneh Mahdisoltani, Guillaume Berger, Waseem Gharbieh, David Fleet, and Roland Memisevic. On the effectiveness of task granularity for transfer learning. arXiv preprint orXh::180.009235, 2018.
Antoine Miech, Dimitri Zhukov, Man--Batiste Alayrae, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto Learning text-video embedding by watching hundred million narrated video clips. In ICCV, pp. 2630-2640, 2019.
Antoine Miech, Jean- -aan-taptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisser- man. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
Salman Khan Muhammad Maaz, Hanoona Rasheed and Fahad Khan. Vidco-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv 2306.05424, 2023.
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. In ECCV, pp. 407-426. Springer, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /000:0007400 2018.
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21 21(1):5485-5511 2020.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3202-3212, 2015.
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. IJCV, 123:94-120, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. How2: large-scale dataset for multimodal language understanding. arXiv preprint arXiv:1811.00347, 2018.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pp. 2566-2565, 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit arXiv:2107.06383, 2021.
tasks? arXiv preprint
Sijie Song, Xudong Lin, Jiaying Liu, Zongming Guo, and Shih-Fu Chang. Co-groume Co-grounding networks with semantic attention for referring expression comprehension in videos. In CVPR, June 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 222222422 2012.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2443-2449, 2021.
Jonathan C Stroud, Zhichao Lu, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid, and David A Ross. Learning video representations from textual web supervision. arXiv preprint arXiv:2007.14937, 2020.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: joint model for video and language representation learning. ICCV, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303.15389, 2023.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfce100m: The new data in multimedia research. Communications of the ACM, 59(2):64-75 -3, 2016.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data- efficient learners for self sulf-ssee video pre- traiingin In NewrlPS, 2022, doi: 10.48550/arXiv. 2203. 12602. htt.pttss: wtaeiooooorggg// / 48550/8rXX0. 2203. 2203.12602
13

---

Published as a conference paper at ICLR 2024
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge. Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video- aado-language pre-training. arXiv preprint arXiv:2203.07300 2022a.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang. Jilan Xu, Yi Liu, Zun Wang. Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvidoo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 2022b.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212. 1565, 2022.
Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. mplug-2: modularized multi-modal foundation model across text, image and video. arXiv preprint arXiv:2302.00402. 2023.
Hu Xu, Gargi Ghosh, Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pp. 5288-5296, 2016.
Hongwei Xue, Tiankai Hang. Yanhong Zeng. Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, pp. 5036, 2022.
Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio- temporal video grounding with transformers. In CVPR, pp. 16442-16453, 2022.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2 / 2021.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jac Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. NeurIPS, 34:23634-2365 2021.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In CVPR, pp. 16375-16387, 2022
Ziyun Zeng, Yixiao Ge, Zhan Tong, Xihui Liu, Shu-Tao Xia, and Ying Shan. Tvtsv2: Learning out-of-the-box spatiotemporal visual representations at scale. arXiv preprint arXiv:2305. 14173, 2023a.
Ziyun Zeng, Yuying Ge, Xihui Liu, Bin Chen, Ping Luo, Shu-Tao Xia, and Yixiao Ge. Learning transferable spatiotemporal representations from natural script knowledge. In CVPR, pp. 23079- 23089, 2023b.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision- languaee understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023a.
Linchao Zhu and Yi Yang, Actbert: Learning global-local video-text representations. CVPR, 2020.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint (xX.i:::334400333) 2023b.
14

---

Published as a conference paper at ICLR 2024
paper plss korea des
textwastn scene
one
photo
chinese
movie woman
microphone wind
many
dress
someone
couch
phone
mourtee
minecraft glasse
city yellow picture character
japanese
white
boxt
green gun
Walking
women
tree background
group building
large
door different
adside several
computer
groune looking night
hareenhair road
neal sithouse colfer
floor three girl
young box
thumbnal
people
satens food
player
view
stand
video
south
game
word@nother
animated
front
word
holding
middle
book tv
driving
unluen drama
blue
two
kitchen
pink
papp
logo
new
store
mask
face
street
office
piece
dark
making
sitting
cartoon
aie
standing
man
men
wood
anime
next
youtube
desk
screenshot
ue
field
laptop
eating
near working boy
& wall person
open dog reclook
wearing light plant
shirtshown asian
tues
orange inside dedde
using image
plant
jacket
room
purple
chair
sey
Korean
hand
de
machine
television
around
talking
ta
black
pomtinng
suit
map
stageshow languagenad
table
sign
small
camera
liam showingold
cutting speaking shot
water
Figure 9: The word cloud (Top-200) of the generated captions in the Intern Vid dataset reveals that
the captions predominantly highlight the rich actions of the objects.
S
short
build
fulleower play
lyric
hind
VI
video
game
e
nead life
@sseep first
challenge
best
band
ttime
live
way find
sang water
making
home
giieebock
ennglish
make
set
tipe
year,
ouk
using
free
kid
money
day
test
Membunnees
part
Officia
usedly
claan
will
food
shorts
131
change tutorial
musicshow
Soaby
n
new
one
beginner guide
easy
top
P
English.
Chinese.
Hots for aaor
TO No p
H to
anleitungstrgg baby evir
einfach
dance
wie
rezept
of
cken eing selber
best
TH
A/E
2
au
alle
trick gan
ow tell
test win spiel
anf
es. Enger
haare
bal
& en
se du's & den
uno
play
video
eev tag
world sie
Beshinneww
tutorial
livero
nicht
im
aus
20
7
O
OO
II
fl
Q
00
RIN
180
0
al m eein game
lernen nur
spielen
essen motin
erki ai
máchen zur man let letml [ beima
deutsch
short
richtig challenge
der
vom ohne
nach
tipp
ver
Sorddlla melne,
die
da
wane
bei
wir
ber
Korean.
German.
Figure 10: The word clouds of the ASR transcripts of four different languages (English, Chinese, Korean, and German). We collect videos from various countries or regions with 11 different languages. Here we list four of them to show how these transcripts are distributed in words.
A DATA AVAILABILITY STATEMENT
We committed maintaining transparency and compliance in our data collection and sharing methods. In accordance with these principles, please note the following:
Publicly Available Data: The data utilized in studies is publicly available. We do not use any exclusive or private data sources.
Our data sharing policy builds upon the precedent set by prior works like Kinetics, HD-VILA. and others. Instead of providing the original raw data, we only supply the YouTube video IDs necessary for downloading the respective content.
Data Sharing Policy:
Usage Rights: The data released by us is intended exclusively for research purposes. Any potential commercial usage is not sanctioned under this agreement.
Compliance with YouTube Policies: Our data collection and release practices are strictly in accord with YouTube's data privacy policies. We ensure that user data privacy rights are violated during the process.
Data Licence: We employ the protocol of CC BY 4.0.
15

---

Published as a conference paper at ICLR 2024
B LIMITATIONS & SOCIETAL IMPACT
All video data used our research are downloaded from YouTube using Safe for Work (SFW) queries and channels. To ensure appropriate content, we employ a simple NSFW filter: a binary classifier designed to recognize and exclude non-ethical videos. For privacy considerations and respect of data sharing practices, works. This approach aligns with YouTube's 's data protocols and ensures no violation of privacy or data usage rules. Despite these precautions, diversity and representativeness. Although YouTube an extensive source encompassing a wide range of video categories, certain specific types of footage may be excluded scarcely collected, including: public area surveillance, sports competitions, movies, documentaries, etc. The exclusion of such categories is often due copyright restrictions or other limits imposed by the platform. Therefore, while our dataset provides a broad view of everyday video content, its coverage does not extend to every possible category or type of video. These limitations should be taken into account when considering the generalizability of our results across all types of video data.
share only the YouTube ID of the videos, similar to previous academic
work has some limitations, primarily related to data
C MORE STATISTICS IN INTERNVID
Actionness. Intern Vid contains way more verbs than the WebVidIOM. We used NLTK toolkit to analyze the number of verbs in captions, focusing on tagging all unique verbs. We found a total of 109,485 verbs in the Web Vid10M, while Intern Vid contained 212,155 ones. While the counts may not be that accurate due to our simple counting, we believe they provide a rough indication of the actionness of the two datasets.
Video Caption and Transcript Distribution. To analyze the word distribution of our generated captions and multilingual (ASR) transcripts, we compute their distributions. The resulting word distribution of the captions is presented in Figure which includes objects (tv, car, door, plant, etc.), attributes (green, young, large, long, etc.), locations (middle, behind, south, next, etc.), scenes (room, stage, kitchen, office, etc.), actions/events (walking, eating, cutting, holding, etc.), and more.
We also include four word distributions of different languages in Figure [0] reflecting trends in different countries and offering potential data customization along with the provided metadata.
Aesthetic Scores and Clip-Caption Similarity. We uniformly sampled four frames of each clip, calculated their aesthetic scores, and took the maximum score as the video aesthetic score. For clip-caption similarity, we computed the cosine similarity between video embeddings and text ones, again using a uniform sampling of four frames for each clip.
Potential Biases. We focus on age, gender, and race distributions, as these are commonly recognized areas where bias can occur. Our methodology consisted of counting keywords related to these categories in the generated video captions. It's important to note that these synthetic captions may not fully reflect the truth of the corresponding videos, thereby creating a gap between our analysis and the actual reality.Here are the results of our analysis:
Age distribution: We counted nouns related to children, grown-ups, and the elderly. We found that 30.71% of the video captions contained such descriptions. Within this subset, the majority were about adults (84.59%), followed by children (15.31%) and barely any mentions of senior citizens (0.08%).
Gender distribution: We counted nouns specifically related to males and females. According to our findings, 33.7% of video captions contained some form of gender-related text. Among these, 64.27% pertained to men and 35.73% pertained to women.
Race distribution: Only around 2.51% of video captions contained descriptions related to race. This could be due to the limitations of our captioning pipeline, which might not be capable of capturing such attributes accurately. Further exploration using a dedicated race recognition model is needed for more accurate statisties.
16

---

Published as a conference paper at ICLR 2024
D INTERNVID-ICL: INTERLEAVED VIDEO-TEXT FOR IN-CONTEXT VIDEO LEARNING
[..., "the inside of a home has rug and light on.", " We could leave the Christmas lights up til January J", "woman with blond hair playing guitar" "s Have I known you
seconds or years? d",
"close-up of a bathroom sink with soap bubbles and other items", "a bathroom is seen with a sink and two lights", woman swiming inside of fishbowl with ladder and man",
Can go wher you go?
"devils roll the dice, angels roll their eyes", "* And, take me out, and take me home d" in room with pink carpet", " You' re my, my d", "a woman in yellow is dancing with a man in a red room", " My, My lover à"
"the man is standing
"a woman is sitting on a chair, playing guitar and woman holding balloon", "* a d", "two men smiling while holding wine glasses and drinking beer", "A We could let our friends crash in the 1iving d" ...]
Table 8: Interleaved video-text data format (b) in InternVid. The caption and ASR transcript of each clip is shown in black and gray, respectively. We can achieve interleaved video-text data format (a) by abandoning ASR transcripts. To obtain data format (c), we concatenate multiple videos with interleaved video-text data (a).
Visual Examples. As given in the paper, we provide examples video+text interleaved entries for in-cntext learning Flamingo. Table T an example about format (a): arrange clips and their descriptions sequentially based on their temporal order within the same video. Note the videos are randomly dropped with probability (0.3) for constructing richer text context compared with the original vidco-text pair combinations in sequential.
E IMPLEMENTATION DETAILS
E DIVERSITY SAMPLING IN CONSTRUCTING INTERNVID SUBSETS
For DIV (diversity sampling), aim sample video clips from all long videos available to maximize data diversity. This was done by counting the frequencies of long videos in the segmented clip pool and sampling clips with probabilities inverse to these frequencies. Here is pseudocode example of this process:
from collections import Counter
import json
import random
import numpy as np (open (ropen("Aae ("/path/to/to satoos_smmmm)))
data = json.
video id = set ( [[x (viieeo"" wswlit ("/") [-1] [:11] for x in data])
video id counter Counter [x[ "video"] split ("/") [:11] for in data
sampling
sampling.weights weights = [1.0 / video id id counter [x [viideo" [.video"].spiie ("/")
[-1][:11]] for x in data]
np wwn.oa.do..mmm.... seed (42)
himpling_weightt. np. array (sampling_weighte
17

---

Published as a conference paper at ICLR 2024
config
MSRVTT DiDeMo ANet LSMDC MSVD
optimizer
AdamW
optimizer momentum
B1, 82 00.9, 0.999 0.02
weight decay
learning rate schedule
cosine decay
learning rate
2c-5
4e-5
2c-5
2c-5
4e-5
batch size
256
warmup epochs
total epochs
7
8
5
10
20
input frame max text length drop path flip augmentation
12
32
96
64
64
150
0.3
0.2
0.3
0.3
0.2
yes
augmentation
MultiScaleCrop [0.5, 1]
Table Video-text retrieval fine-tuning settings.
hempling__wiightt = sampling_weightt / sampling samplinggweighhs... sum sempledd_inder np. random. choice (len (data), 10647458, replace=False, pa
is data [data for i sampled index] 14 json dump (data, open ("
"w"))
FLT (filtering). we applied a series filtering strategies to video data alongside DIV sampling. These included: a) Removing video clips shorter than Is (approximately 23.15% of the total) or longer than 120s (around 0.84% of the total). b) Computing CLIPScore for each video clip using randomly sampled frame from the clip with OpenAl's LLIP-VVT-L/14, then selecting clips within the top 30% of CLIPScores. c) Sampling 10M out the remaining clips using DIV sampling.
E.2 VICLIP
Implementation. VICLIP is learned with NVIDIA A100 GPUs days with 50M video-text pairs. introduce DeepSpeed and FlashAttention (Dao 2022 for training and inference.
Action Recognition. the settings CLIP and EVA-CLIP, we report the mean of top-1 and top-5 accuracy for Kinetics-400/ / /600 700. In Section 5.1 model.
the zero-shot action recognition, sample 8 frames each vidco. Following
we show ViCLIP learnt on WebVid or Intern Vid is an effective zero-shot action recognition
In the full fine-tuned setting, conduct experiments with receipts. In Table for the experiments where the training data excluded K710, followed the common practice of finetuning the pretrained VICLIP with the training data from the evaluation dataset. On the other hand, for the experiments where the training data included K710, we adopted training trick inspired by (Li al. 2022a). We first finetuned the pretrained VICIIP with K710 (Li et al. 2022a), and then proceeded with the common supervised finetuning setting. By incorporating the supervised fineturing with K710, ViCIIP demonstrated better performance the fine-tuned tasks compared to experiments that did not include K710.
Video Retrieval. In the full-finetuning setting, we the pretrained ViCLIP with only video-text contrastive loss but also vidco-text matching loss on the training data the evaluated benchmarks, During both training and testing, we sample 12 frames. Detailed hyper-parametert are given in Table 9] In the zero-shot setting, we sample only 8 frames for evaluations.
E.3 VIDEO GENERATION BASELINE
We used the spatiotemporal modeling approach from Wu et al. 2022) and built our text-to-video generation baseline the work of (Rombach al. 2022. Our approach consists of U-Net with transformer that models latents, using interleaved spatiotemporal attention (ST-Attn), cross attention for visual-texx feed-fore forward network (FFN), and temporal attention (T-Att)) as illustrated in Figure [I To adapt the 2D convolutional layers in (Rombach et al, 2022) to 3D, we extended 3 x 3 kernels into 1 x 3 3 ones, We also extended the original spatial attentions spatiotemporal ones. We initialized our baseline using all text-to-image diffusion model parameters, while the newly added temporal attention layers used default parameters.
18

---

Published as a conference paper at ICLR 2024
Reconstruction Loss
Diffusion
noise
"A cat walking on the piano.
Conv Block
Transformer
Block
Training
WFPN
Atrn
Attm
Frozen
Figure 11: Framework of our text-to-video generation baseline.
Retrieval
Action Recognition Zero-Shot
Captioning Method
Zero-Shot
Fine-Tuned
MSR-VTI
MSR-VTT
K400
K600
K700
T2V
V2T
T2V
V2T
top-1
AVG
top-1
AVG
top-1
AVG
VideoChat
33.9
32.3
46.6
47.1
54.68
67.74
51.70
64.91 68.27
43.67
56.51
Ours
38.6
38.5
49.0
49.2
58.52
71.11
55.37
47.09
59.98
Table 10: Video retrieval and action recognition results of ViCLIP-B trained Intern Vid-FLLT-OM with the captions generated by VideoChat and our captioning approach.
ST-Attn implementation, we used frame embeddings from U-Net encoder instead of video embeddings as in (w et al. 2022). We concatenated the embeddings of the previous and current frame for values and keys in attention, while using the current frame embedding alone as queries. The rest of the implementation remained the same the original.
Tixdeo Evaluation. evaluate our text-to-video model, we conducted zero-shot experiments on UCF-101 and MSRVTT datasets, following the method from (Blattmann et al.) (2023). UCF-101, we used the class names as text prompts and generated 20 samples per class (total of 2,020 videos). For MSRVTT, we randomly selected one caption per video from the official test set (total of 2,990 videos). To ensure fair comparison, used the official implementation of VideoCrafter and VideoFusion (Luo al. 2023 to generate the same number of videos frames per video.
same text prompts. During video sampling and evaluation, we generated 16
We assess the overall quality of the synthesized results on UCF-101 using framewise-FID. FVD, and Inception Score (IS), and evaluate the text-video semantic similarity MSRVTT using clip similarity (CLIPSIM). For Tramewise-FID and IS, use the pretrained Inception13d model and followed TATS method (Ge et al.) 2022). compute CLIPSIM. we calculate the clip text-image similarity for each frame with respect to the given text prompts and computed the average score. We use the ViT-B-32 clip model as the backbone, consistent with previous work (Blattmann) al. 2023
the pretrained Inceptionv3 network weights
image encoder. For FVD.
F MORE RESULTS
EFFECTIVENESS OF OUR MULTISCALE CAPTIONING APPROACII
To further validate the effectiveness of our proposed captioning method, we establish a video caption baseline using the video multimodal model VideoChat (Li et al. 2023c) for comparison. We input the video clip into the model with the prompt malease descri be the content in the given video. and apply Intern Vid- 10M-FLT, resulting in million new captions generated hy VideoChat. Subsequently, we train two versions of ViCLIP-B Base using Intern Vid- 10M-FLT, each version trained with one of the two types of captions.
Table 10 demonstrates that ViCLIP-B trained using our captions outperforms the version trained using captions from VideoChat in both video retrieval (MSR-VTV) VTT) and action recognition (K400/600/700), These results are particularly noteworthy considering that the only difference in training lies the captions generated by the
19

---

Published as a conference paper at ICLR 2024
K400
K600
K700
Method
Data
top-1
AVG
top-1
AVG
top-1
AVG
ViCLIP ViCLIP
Vid--MM
51.70
64.69
49.20
62.34
40.90
53.70
38.40
51.58
36.40
49.19
29.10
40.68
Table 11: Zero-shot action recognition results of ViCLIP using different captions on Kinetics
400/600/700.
MSR-VTT LSMDC DiDeMo
MSVD
ANet
Method
Data
T2V V2T T2V V2T T2V V2T T2V V2T
T2V
V2T
ViCLIP Vid-2M
31.8 33.7
14.3
12.7
13.6 21.5 39.6
62.5
9.9
16.8
ViCLIP +Intern Vid-2-Bidd-M-LLIP
21.7
21.9
5.2
5.1
7.1
12.7
24.6 42.1
6.4 10.2
Table 12: Results of zero-shot video retrieval from ViCLIPusing different captions on MSR-VT VTT,
LSMDC, DiDeMo, MSVD, and ActivityNet.
two different approaches. Therefore, these findings further confirm the superior performance of our proposed captioning method compared to the baseline VideoChat.
We also ablate the necessary of including fine-level captions (by tag2text) more than only using coarse ones (by BLIP2). Specifically, an ablation performed on two subsets of the dataset Vid-2M and Intern Vid- 2M- BLIP), each having 2 million video-text pairs. Intern Vid-2M utilized fused captions, combining both coarse- and fine-level ones. In contrast, Intern I Vid-2M-BI.IP only used the coarse-level captions produced by BLIP2 on the central frames. For the mentioned using fine-level captions, concatenating the framewise captions from tag2text as the video captions not promising opinion as these captions are quite long and full of reptitions, unsuitable for contrastive learning. Thus, we not include this setting in experiments.
Zero-shot experiments are conducted on these models. Due to computational constraints, the ViCLIP-B is trained with a batch size of 4096 using 8 A 100 GPUs, with a mask ratio set to 0.9. All remaining training parameters were consistent with those in the main paper. Contrasting the results from Table [1 and [2] it's evident that the of combined coarse and fine-level captions video-text contrastive learning rendered superior zero-shot performance than utilizing the coarse level ones alone. It shows the effectiveness our given video captioning pipeline.
How the LM impacts motion-related words when summarizing framewise captions into video ones. From a statistical perspective, generating video captions from frame-level captions using a language model has negligible effect on the number of motion-related words captured for video-based understanding. To illustrate this, counted the unique verbs (using nitk package) in the captions from a 10m subset of Intern Vid under two settings: 1) In the first setting, the captions are video captions generated by the language model. 2) In the second setting, the captions are Trame-wisc ones from BLIP2 and tag2text. We found that the number of unique verbs in the video captions is 109859, whereas for the frame-wise captions is slightly higher at 109895. This small discrepancy suggests that almost no motion-related words are lost during the caption generation process by LM. Therefore, believe approach maintains most of the important motion-related information needed for video understanding.
F.2 MODEL SCALING
We provide a comparison between two versions of VICLIP, -L. (300M) and -B (80M), when trained on Intern Vid in Tables 13 14| and 15 These tables distinctly demonstrate that moving from the base to large model, ViCLIP's zero-shot and linetuned video retrieval performance, well as zero-shot action recognition, can be consistently improved. These tables clearly demonstrate the benefits model scaling, and we aim explore this area further future work as resources permit.
F.3 LINEAR PROBING
we present the linear action recognition results Kinetics-400 in Table T6 It's noteworthy that VICLIP, trained on Intern Dideniii--MM--T/-22000
delivers much higher top-1 accuracy compared when trained on Web Vid- (with more than 10- 10-point increase), mirroring our findings in fine-tuned action recognition settings. Comparing with other approaches, ViCLIP-L offers performance close to TVTSv2-HL -/-B, which incorporate extra learnable parameters for spatiotemporal modeling. Moreover, it significantly outperforms 2id. This result can be attributed to the fact that MAF-based methodologies generally underperform in linear evaluations.
20

---

Published as a conference paper at ICLR 2024
MSR-VTT
LSMDC
DoDeMo
MSVD
ANet
Method
Data
T2V
V2T
T2V
V2T
T2V
V2T
T2V
V2T
T2V
V2T
ViCLIP-B +intern
37.4
36.1
16.5
15.2
16.6
22.6
44.3
67.0
13.3
21.7
ViCLIP-L +InternVid-2000 Vid-200M
39.3
39.5
18.3
16.6
17.1
25.5
47.3
70.0
13.7
21.6
ViCLIP-B
+Intern 1id-IOMFLL
38.6
38.5
18.5
17.0
16.3
25.0
44.8
67.2
13.0
21.8
ViCLIP-L +Intern Iid-IOM-FLT
42.4
41.3
20.1
16.9
18.4
27.9
49.1
75.1
15.1
24.0
Table 13: Scaling model in zero-shot video retrieval on MSR-VTV VTT, LSMDC, DiDeMo, MSVD, and ActivityNet.
MSR-VTT
LSMDC
DiDeMo
MSVD
ANet
Method
Data
T2V
V2T
T2V
V2T
T2V
V2T
T2V
V2T
T2V
V2T
ViCLIP-B +Intern Vid-200M
50.7
49.4
25.3
25.4
41.1
40.8
69.0
69.3
37.7
35.8
ViCLIP-I
+Intern Vid-200M
53.7
53.4
29.3
29.3
51.1
50.8
79.9
78.4
52.8
51.1
ViCLIP-B +Intern 1id-1OM-FLT
49.0
49.2
24.4
23.7
40.0
41.4
72.2
73.7
38.3
37.0
ViCLIP-L
+Intern 10M-FTT
52.5
51.8
33.0
33.0
49.4
50.2
77.2
79.0
49.8
48.1
Table 14: Scaling model in fine-tuned video retrieval on MSR-VTT, LSMDC, DiDeMo, MSVD, and ActivityNet.
F.4 IMPACT OF VIDEOS FROM DIFFERENT LANGUAGE SOURCES
Unlike previous models that benchmarked mostly English-based datasets, Intern Vid encompasses clips from variety languages. This necessitates further analysis to determine the potential impact this diversity might have. Currently, we hypothesize that the language of the video may not significantly impact the generated captions as our deployed image caption models generate English descriptions based purely on input frames. However, in terms of video distributions, there may exist differences (such as in behaviors, activities, and events) between videos stemming from different countries due varied cultural backgrounds.
examine this hypothesis, we select two 2 million subsets Intern one consisting only English videos (Intern Vid-2M-EN and another with only Chinese videos (Intern Vid-2M-CN). It's important to note that whether the videos are in English or Chinese, we generate captions in English. Our ViCLIP-B B model is pretrained on these subsets, and conduct zero-shot experiments described below. Due resource constraints,
trained the ViCLIP-B with batchsize 4096 using A100 GPUs with a mask ratio set to 0.9. The remaining training settings were kept consistent with those outlined in the paper. As given in Table 18] we that the model pretrained with Intern Vid-2M-EN outperformed that with Mice-Vid-2M-C notably in both zero-shot action recognition K400/600//00 and vidco retrieval. This result can be attributed to the fact that Intern AiterVid-2MM Vid-2M-EN has data distribution much closer downstream task data than Intern Vid-2M-CN. all used task videos sourced from English sources.
F.5 TEXT-TO-VIDEL GENERATION
we observe that the t2v baseline using both Oc and Intern Vid-Aes- significantly outperforms others in visual quality and temporal coherence. Note that the 12v baseline using Intern Vid does not contain watermarks, which is a data bias in WebVid10M. We give more visual comparisons between our baseline and other approaches in the supp.
Figure 12,
F.6 VIDEO-CENTRIC DIALOGUE SYSTEM
We give qualitative evaluations of VideoChat VICLIP in spatial understanding (Figure 13), action recognition (Figure [4), temporal understanding (Figure 15). video reasoning (Figure 16). and video creative (Figure 17 tasks.
21

---

Published as a conference paper at ICLR 2024
K400
K600
K700
Method
Data
top-1
AVG
top-1
AVG
top-1
AVG
ViCLIP-B
+Intern Vid-200M
56.58
69.20
53.57
66.20
45.82
58.28
ViCLIP-L
+Intern Vid-200M
59.80
71.09
57.80
69.34
49.30
61.25
ViCLIP-B
PinterVVid-1MMM
58.52
71.11
55.37
68.27
47.09
59.98
ViCLIP-L
+InternVid-IOMee
64.80
75.70
62.20
73.53
54.30
66.38
Table 15: Scaling model in zero-shot action recognition results Kinetics 400/600/700.
K400
Method
Data
top-1 1
AVG
VideoMAE-B Tong et al.
2022)
Kinetics-400
20.4
VideoMAEv2-H Wang et al. 2023
Kinttiss+SthSth--AAA--aaaa Vid2
25.8
TVTS-B (Zeng al. 2023b)
+YT-Tempooal-1180 80M
60.8
TVTSv2-B (Zeng et al.
2023a
+ VT--Temporal-180M-- Wed--
70.1
TVTSv2-H (Zeng
2023a
YT-Teemporal-180M-- Veb
73.1
ViCLIP-L
+WebVid-10M
60.0
82.9
ViCLIP-L ViCLIP-L
+IntennVid-00M--U
71.1
90.4
InteerVid-2000
71.7
90.9
Table 16: Linear action recognition results on Kinetics-400.
K400
K600
K700
Method
Data
top-1
AVG
top-1
AVG
top-1
AVG
ViCLIP
Vinter-id--M--E
40.20
53.68
37.40
51.00
29.60
42.06
VICLIP + InternVid-2MM
35.9
49.73
33.70
47.05
26.90
39.02
Table 17: shoo action recognition results of ViCLIP using different pretraining sources on
Kinetics 400/600/700.
MSR-VT VTT
LSMDC
DiDeMo
MSVD
ANet V2T
Method
Data
T2V
V2T
T2V
V2T T2V
V2T
T2V
V2T
T2V
ViCLIP Vid-2-EN 24.1
24.1
9.8
9.1
10.3
15.6
31.4
50.5
7.1
12.0 9.7
ViCLIP -nteemVid-2M--
22.2
22.2
8.4
8.7
9.9
15.1
29.6
48.5
5.7
Table 18: Results of zero-shot video retrieval of ViCLIP using different pretraining sources VTT, LSMDC, DiDeMo, MSVD, and ActivityNet.
22

---

Published as a conference paper at ICLR 2024
VideoCrafter (WebVid10M)
VideoFusion (WebVid10M)
t2v baseline
(WehVid10M)
t2v baseline (+InterVVid)
Figure 12: Comparison of samples from t2v baseline to others. We provide zero-shot eex-tto-video generation results of different methods trained on both WebVidI0M and the additional Intern Vid-Aes- 18M. The used prompt is: a bald man a black t-shirt is playing a guitar.
where is this video taken from?
ALIA
The video is taken from an aerial view of a skyscraper-filled city, with the Shanghai skyline being the prominent feature in the background. The specific location is Shanghai, China, with several skyscrapers and river visible in the scene. Additionally, few specific buildings and roads are also seen, such as the Oriental Pearl Tower and the Huangpu River.
How many buildings in 00:12?
ALIA
There are several buildings visible the 00:12 frame, including several towers and skyscrapers, as well as a river and a city skyline. they are
The exact number of buildings is difficult to count, interspersed with other objects and structures in the scene, but there are several visible. The buildings include the Oriental Pearl Tower, the World Financial Center, and the IFC Mall, among others.
Figure 13: Video spatial understanding task. This figure demonstrates the spatial understanding and temporal localization capabilities of VideoChat- ViCLIP trained with our dataset.
23

---

Published as a conference paper at ICLR 2024
Describe the activity in the video.
The man is playing the guitar and moving around. The video shows him holding the guitar and playing it.
Figure 14: Video action recognition task. The video comes from Kinetics-400, with the label "playing guitar".
Step-by-step describe the video.
green shirt stands a kitchen with transparent bowl on a table, holding a knife. Next, she uses the knife to cut green apple on the table in the kitchen. Then, she continues to cut the apple, still using the knife. Afterward, she cuts another apple Finally, she cuts the apple and puts table. Throughout the video, other apples are visible on the table, and the woman uses knife to cut them well. The background consists of wooden table, countertops, and kitchen. The video is motion
First, a woman
the table, holding the knife in her hand.
glass blender on the
blurry, green-tinted appearance.
What will she next?
Based on context, it reasonable assume that she may continue to cut apples and prepare them for juicing.
Figure 15: Temporal understanding task. VideoChat ViCLIP can easily handle temporal under- standing tasks and make predictions based these temporal sequences due to its training rich temporal information on InternVid
what is strange about this video?
It shows a man and a lion interacting in a forest. The man is holding lion by the ears, and the lion seems to be okay with being petted by the man. They seem to be enjoying each other's company. The video shows rare and unusual interaction between a man and lion, which is very rare and surprising to see in real life.
Figure 16: Video reasoning task. Our dialogue system has the ability understand common sense and make causal inferences based on the contents of the video.
24

---

Published as a conference paper at ICLR 2024
Write short poem for this video.
In blue flowers she walks, wind in hair. white straw hat, peace in her stare. She breathes, admires the sky. Tall she stands, with eyes up high.
Figure 17: Video creative task. This figure shows the performance VideoChat- ViCLIP on creative and generative tasks within our dataset, like generating poetry based on video content.
25


Reviews:
[{"review_id": "73u7sK1sWS", "reviewer": "Reviewer_gWXX", "summary": "This paper proposes a large-scale video-text dataset for video representation learning. Considering the noise and low correlation in ASR transcripts, this paper utilizes captioning models and a pre-trained language model to generate video descriptions. The authors pre-train the ViCLIP model on the collected dataset and the model shows strong performance on action recognition and text-video retrieval tasks. They also explore potential applications of this dataset on text-to-video generation and video-centric dialogue systems.", "questions": "Refering to the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The dataset is large-scale and diverse, and the generated descriptions have better relevance with videos.\n- The pre-trained model shows good performance.\n- The experiments are extensive.", "weaknesses": "- Some details are not presented clearly. 1) How many descriptions are generated at the coarser level? Are all descriptions used for training the final model at the finer level? 2) Which pre-trained language model is used for processing captions? 3) The details about DIV and FLT are not introduced.\n- A closely related work, CLIP-ViP[R1], is not included in related works and compared. It also adopts a caption model to generate descriptions for video clips. It is corresponding to the coarser level in this paper.\n- The performance of InterVid-10M-DIV, InterVid-10M-FLT is weird. It shows much better performance in zero-shot action recognition in Table 2, but poor in the fine-tuned setting of Table 3. The authors give reasons for false negatives, but what is the training batch size, given the much larger training data, I think the possibility of the same video clip appearing in the same batch is low.\n\n[R1]: CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xt7EedelKQ", "reviewer": "Reviewer_G48V", "summary": "This paper introduces a new video and language dataset, INTERNVID, which includes 234M video-text pairs lasting 760K hours. In addition to the dataset, this paper also introduce a baseline model, ViCLIP, demonstrating its performance on various downstream applications after pre-training on INTERNVID dataset.", "questions": "Given 10M pretrained data, the ViCLIP receives better zero-shot action recognition results from WebVid and better fine-tune action recognition results in Table 2 and 3. And the gain from 50M, 200M INTERNVID pretraining is minor. Does it mean the pretraining data is not the more the better? The performance of vision and language model will be saturated when the pretraining data reach to a certain scale? Could you please provide more insights here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written, and each section is easy to follow. \n\n2. The proposed large-scaled video-text dataset can contribute to the community, scaling up the current model and largely improving the video and language feature representation learning. \n\n3. This paper provides very detailed statistic for the dataset, and also reveal the method for data curation.\n\n4. By leveraging the new dataset, this paper demonstrates extensive experiments on many downstream applications and achieving promising results on many benchmarks.", "weaknesses": "1. There are not many analyses and design justification for the proposed ViCLIP. If ViCLIP is claimed as one of the contributions of this paper. The proposed architecture is not novel and the masking idea needs further justification. For example, the efficiency gain vs. the performance drop, and the ablation over the masking ratio.\n\n2. The video caption is generated by language model from frame-level captions. In this case, will this reduce the number of motion-related words that need to be captured from video-based understanding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tT8nT4UOl8", "reviewer": "Reviewer_6e9v", "summary": "- This paper introduces a web-scale video language dataset InternVid comprised of 234M clips and 760K hours.\n- It introduces ViCLIP, a video-language model based on ViT-L, pretrained with contrastive learning (like CLIP) and masked autoencoder (like VideoMAE, MAE).\n- This work expands the utility of their dataset to video understanding tasks like recognition and retrieval, and video generation.", "questions": "- Will you share the pretrained and finetuned models with the supporting code base (e.g., data processing, pretraining, finetuning, generation)? \n- Could you please share the processed clips, even processing the data by individuals (typically for academic researchers) would be a difficult task considering its massive size. Additionally, we all know the unavailability of videos due to location constraints, permission issues, etc., so even if the full data is not possible to share at least share the 3 10M versions.\n- I suggest releasing the fixed embeddings of the datasets from the trained (e.g., pretrained, finetuned) models.\n- Did you investigate if the InternVid has any sort of bias in the curated clips, could you please share a report with such details? Bias could be of many forms e.g., location/race/gender per action category. A suggested reference: https://arxiv.org/abs/1505.01257\n- Did you investigate, if ViCLIP is robust against some of the OOD setups, some of the popular benchmarks are Mimetics, RareAct etc. For more details please see: https://arxiv.org/abs/2306.02014", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It introduces a web-scale video-language dataset bridging the gap of lack of such datasets, unlike image-text domains.\n- It evaluates on multiple benchmarks in both finetuned and zero-shot setups.\n- The dataset curation is fairly detailed and I also like the hierarchical video caption generation strategy.", "weaknesses": "- The majority of the comparisons are with CLIP which was pretrained with image-text pairs and not video-text; I would like to see how ViCLIP performs compared to other video or video-language models (e.g., pretrained on HD-VILA, HT100M, YT100M), I think that would be a fair comparison than comparing with image-text models.\n- I would encourage authors to dig deeper and find a more concrete argument/explanation: why does pretraining with InternVid-10M-FLT or InternVid-10M-DIV perform better in **zero-shot** than InternVid-200M, but not when **finetuned**. \n- In fig. 7 and 8, the experiments are done only on scaling the dataset size, it would be interesting to see the effect of model scaling in addition to the dataset scaling. I would encourage you to add such experiments in the final version.\n- I would be interested to see linear evaluation (i.e., a single FC layer, or use linear SVM) performance on the downstream benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H1HxdqN6Qp", "reviewer": "Reviewer_1UqW", "summary": "This paper presents InternVid, an large-scaled video-language dataset, consisting of a collection of 7M videos, 234M video clips and corresponding textual descriptions. They consider diversity and data quality by sourcing YouTube contents from various countries and languages. This new dataset enables pretraining a video-language model, named ViCLIP, exhibiting state-of-the-art scores in action recognition and video retrieval tasks. It also demonstrates a credible generative capability in text-to-video generation, supported by qualitative and quantitative assessment.", "questions": "1. Consideration with Table 6\n\n- Comment of VideoCrafter, VideoFusion is absent. Does InternVID also improve VideoCrafter, VideoFusion in IS, FID, FVD, and CLIPSIM? Can you provide further analysis about the choice of InternVID-Aes-18M rather than exploiting the full InternVID-200M? \n\n2. Consideration with Section 5.2 and Figure 13~17\n\n- To insist that InternVID provides more powerful video-text aligned representations for video-centric dialogue systems, authors must compare the results from VideoChat-ViCLIP with that of vanilla VideoChat. Can you provide quantitative and qualitative comparisons, showing improvement of the results after replacement of the previous visual encoder with ViCLIP?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Clarity in writing\n\n- The paper is well-written and easy to understand.\n\n2. Scale and impact\n\n- InternVid includes large-scaled 7.1M videos, making it a significant contribution to the video-language pretraining. Referring to table 1, InternVID comprises 234M 720p video clips with LLM generated captions which shows outperforming accuracy in description. The dataset’s substantial size can potentially have a profound impact on the field.\n\n3. Diversity and Quality\n\n- The effort to collect videos from various countries and languages is commendable. This diversity in data collection may address the issue of bias, a crucial aspect of large-scale datasets, as it mitigates the risk of training biased models.\n\n4. High performance\n\n- The suggested model, referred as ViCLIP, achieve the state-of-the-art in video-language benchmarks such as  action recognition and text-to-video retrieval.\n- InternVID improves text-to-video generation by empowering the previous models with its large scale video-text dataset. It shows quantitative and qualitative (Figure 12) improvement in generation quality.", "weaknesses": "1. Lack of definitions, explanations\n\n- Definition of UMT-SIM is absent. In table 2, justification of the design choice of InternVid-10M-FLT is not explained. (At glance, the result of performance improvement after filtering the dataset with a high UMT-SIM score seems trivial.)\n- Figure 5 introduced 3 schemes of interleaving video clips, text, and ASR text. However, comparison between the methods or further analysis is not given.\n- It is doubtful that CLIP is a suitable choice for comparing their method in action recognition tasks. There are likely more appropriate video models to use in the benchmark for a fairer comparison.\n\n2. Lack of novelty\n\n- The proposed method for generating the dataset is not novel. It simply uses BLIP to generate captions for video frames, and exploits LLM to generate a summarized caption for the video. Also, description of the used LLM is not given.\n- ViCLIP is not a novel architecture and inherits the CLIP model without significant modifications. The paper predominantly focuses on introducing the dataset, which leaves room for exploring further architectural improvement and training techniques.\n\n3. Unclear model selection process\n\n- The paper lacks information on the criteria used to select the image captioning and language models.\n\n4. Absence of ablation study and potential impact\n\n- In contrast to previous models which are benchmarked in datasets mostly composed of English, InternVID consists of clips with diverse languages. Further analysis about the potential impact caused by this aspect should be considered.\n\n5. Minor comments on Figure 2\n\n- Similar colors like green and dark green are used, but this may cause confusion. Using more distinct and discrete colors, such as red and blue, could enhance the clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)