LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Authors: Bin Zhu
Bin Lin
Munan Ning
Yang Yan
Jiaxi Cui
WANG HongFa
Yatian Pang
Wenhao Jiang
Junwu Zhang
Zongwei Li
Cai Wan Zhang
Zhifeng Li
Wei Liu
Li Yuan
Venue: ICLR 2024 poster (2024)
Keywords: LanguageBind
Multi-modal Pretraining
Multi-modal Dataset
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N ≥ 3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining and then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with 10 Million data with Video, Infrared, Depth, Audio and their corresponding Language. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities.

Published as a conference paper at ICLR 2024
LANGUAGEBIND: EXTENDING VIDEO-LANGUAGE PRETRAINING TO N-MODALITY BY LANGUAGE- BASED SEMANTIC ALIGNMENT
Bin 2.,", Bin Lin", Munan Ning!., Yang Yan', JiaXi Cui', Hongfa Wang', Yatian Pang", Wenhao Jiang', Junwu Zhang', Zongwei Li", Wancai Zhang,, Zhifeng LI, Wei Liu', Li Yuan" 1.2,
Pekinn University, Pengcheng Lab, Tennent Data Platform, *National University of Singapore, 'Nari Technology Development Limited Company, 'Guangdong Laboratory of Artificial Intelligence and Digital Economy (S7.
(binzhu, linbin,,iiiiiiii ecst pku. wwt.uuukkuuuuuuu..u. cn
ABSTRACT
The video-language (VL) pretraining has achieved remarkable improvement multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, NV 2 3) beyond vision and language. We thus propose LanguageBind, taking the language the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pre- training and then train encoders for other modalities with contrastive learning. As result, all modalities are mapped a shared feature space, implementing multi- modal semantic alignment. While LanguageBind ensures that we can extend VL modalities N modalities, we also need high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-IOM with 10 Million data with Video, Infrared, Depth, Audio and their corresponding Language. our VIDAL- 10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evi- dence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities.
INTRODUCTION
With the development of the Internet and smartphones, there has been a proliferation of video web- sites and apps (e.g., Youtube and TikTok), leading substantial increase number of videos (Xue
et al. 2022). ) Consequently, a set of video tasks have emerged, such as video search (Smith &
Chang 1997). video recommendation (Deldjoo et al.66 and video editing (Casares al.] 2002
Bonneel et al. 2014). To solve video understanding tasks, video-language pretraining has been em- ployed by training foundation models by combining computer vision (He al. 2016: Dosovitskiy et al. 2020) and natural language processing (Vaswani et al. (017)). These models can capture video semantics and solve downstream tasks (Karpathy al. 2004,]Mitnnn et al.) 20188).
However, current VL pretraining frameworks are often limited to vision and language modalities. The ImageBind (Girdhar et al.) 2023) introduces an indirect alignment method for multi-modal pretraining. It aligns other modalities to images, facilitating comprehensive understanding of various modalities such as infrared (Jia et al. (02I), depth (Kim et al. (0022). audio (Piczak) (015)). and IMU Grauman et al. 2022). In practical tasks such zero-shot retrieval and classification
Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2024
X-Language
Retrieval
D
Depth
wilt having meeting A pope troveling on open body vehicle
Infrared
Image
Language
X-Language
Classification
Audio
Language
moduliit of fe)
modality big
modalitt small tot the modality of [o]
ImageBind
N-th modality
LanguageBind
ImageBind vs. LanguageBind. The ImageBind method relies on images as interme- diaries, while the LanguageBind method dispenses with this requirement. LanguageBind directly aligns all modalities to the language space, thereby enhancing its applicability to downstream tasks. "X" represents all modalities except language, and "c" represents category.
Figure 1:
as shown in Figure I the alignment with language modality is predominantly required for various modalities. While the indirect aligmment of ImageBind may result performance degradation, the LanguageBind method does not need images intermediaries and facilitates straightforward expansion to additional modalities in downstream tasks.
In this paper, we propose the LanguageBind, language-basee multi-modal pretraining framework that can extend video-language pretraining to multiple (N) modalities. As the language modality contains rich semantic information and well-explored (Kenton & 'Toutanova 2019 Dai et al., 2019). we take as the bind across different modalities. This process maps all modalities to a unified embedding space, enabling effective semantic alignment. To improve training efficiency, we employ Low-Rank Adaptation (LoRA) (Hu et al. (021]) for fine-tuning, achieving impressive training results with minimal training iterations.
To further improve the modal integrity in pretraining and validate our LanguageBind, we introduce a dataset with five modalities, the VIDAL-IOM, which includes VL, IL laffaaredlanguuage.. DL (depth-language), and AL. (audio-language) data pairs. The videos of previous datasets are always truncated segments from long videos (Miech et al.) 2019 Xue al., 2022p, resulting fragmented video-text pairs from short videos with com-
semantics. To avoid this problem, construct plete stories. To ensure the quality of the central language modality, we perform multi-view text generation and enhancement VIDAL-IOM.
The proposed LanguageBind ensures that we can extend vision- langun-language to multiple (N) modalities, and our dataset VIDAL-1OM benefits more down- stream tasks beyond VL tasks, including video trieval (Luo et al., 2022), depth classification (Cao) al. 2017), infrared classification (Baffa & Lattari 2018 and audio classification (Palanisamy et al.
SIRVIT
Laqugdiaa
Infrared
UsecUP
PLA Ka
LDMP
2020 As shown in Figure [ LanguageBind achieves superior performances on broad range of 15 tasks. In zero-shot text to video retrieval, I LanguageBind achieves superior performance on four datasets, surpassing Inter (Wang et al.)
Senca
Daprh
AS4A
Cuder
150:36
Audio
2022cb by 1.9% on MSR-VTT (Xu al. (2016). 8.8% on MSVD (Chen & Dolan 2011 6.3% on
Figure 2: LanguageBind achieves superior performances on 15 benchmarks.
DiDeMo (Anne Hendricks et al. 2017). and 4.4%
on ActivityNet (Caba Heilbron et al. 2013). For zero-shot classification on depth and infrared data, LanguageBind achieves substantial performance advantage over ImageBind. LanguageBinnd attains top-1 accuracy of 87. 87.2% and 65.1% 1% on LLVIP and NYU-D, respectively. outperforming Im- ageBind by 23.8% and 11. 1.11% For zero-shot audio classification tasks, LanguageBind outperforms ImageBind with a 23.8% higher top-1 accuracy on the ESC50 dataset.
We summarize our primary contributions as follows:
2

---

Published as a conference paper at ICLR 2024
We propose LanguageBind, the Sangauge-base multi- modal pretraining approach. During the pretraining process, all modalities gradually align with the language modality through contrastive learning, and these modalities are unified within shared embedding space.
We introduce VIDAL-IOM a large-scale five-modal video dataset, containing million data pairs with aligned VL, IL, DL, and AL. To the best of our knowledge, VIDAL-IOM is the first large-scale video dataset with depth and infrared modalities.
Extensive experiments validate the effectiveness of our dataset and approach, achieving remarkable performance in video and other modality understanding tasks.
RELATED WORK
Multi-modal Pretraining Multi-modal pretraining begins with pretraining in vision and lan- guage. CLIP (Radford et al., 202I) pioneered the aligmment of images and texts on a large-scale dataset comprising 400 million samples, effectively establishing bridge between the image and text domains. This alignment benefits a variety of downstream tasks, including zero-shot classifica- tion and image-text retrieval (Li et al., 2023a). CLIP can also be used as foundation for alignment in other modalities, For instance, CLIP4Clip (Luo et al., 2022) aligns video with text, CLAP (Wu*) al., 2023) aligns audio with text, and PointCLIP (Zhang al. 2022) aligns point clouds with text. Recent efforts have undertaken a comprehensive exploration of multi-modal alignment through pre- training. Augmenting the alignment process with additional modalities can enhance the model's ro- bustness while maintaining its performance, as observed in VALOR (Chen et al., 2023a) and VAST (Chen et al. 2023b). However, as the number of modalities increases, the training paradigm required to align them effectively undergoes significant changes. Meta-transformer (Zhang et al., (023) ac- commodates 12 modalities and utilizes distinct tokenizers to harmonize the embedding space across modalities. ImageBind (Girdhar et al. 2023) expands multi-modal alignment pretraining to encom- pass six modalities but may not perform as well in language-related tasks due to indirect alignment. In our work, we propose LanguageBind, direct alignment mechanism designed to align alterna- tive modalities directly with the language modality, which has the highest information density. This direct alignment mechanism yields discernible improvements in downstream task performance.
Multi-modal Datasets Multi-modal datasets serve the foundation for multi-modal pretraining (Zhu
Table 1:
Comparision of existing multi- modal datasets. VIDAL-10M is the currently first accessible multi modal dataset including aligned VI. II, DL, and AL data pairs.
iet al. 20244] Lin et al. 2023). Initially, these datasets only consisted of videos and their corresponding cat-
egories, as shown in Table T HMDB-51 (Kuehne
Datasets
Samples Modality Year
(et al.] (011)) and UCF-101 (Soomro al. 2012) are examples of such datasets, which contain truncated segments from long videos with manual annotation. However, creating these datasets required significant human effort, which limited their scalability and di- versity. To address this issue, researchers turned their attention to the abundance of video-text re- sources available on the internet. Inspired by the success of image-text datasets (Sharma et al.] 2018 Changpinyo et al. 202T), they used baript-based pro- gramming (Schuldt et al., 2004 Kong et al.) 2019 Sigurdsson et al. 2018)) to extract millions of video- text data pairs. However, acquiring data from modal- ities like infrared (Teledyne FLIR, (0]5a and
HMDB-51 UCF-101 101
7
V
2011
13K
V
2012
ActivityNct-200
20K
VT
2015
Web VidVVIMMM 10M
10.7M
VT
2021
HD-VILA-100M
100M
VT
2022
How NowTo-100M
136M
VT
2019
LLVIP
15k
VI
2021
FLIR VI
10k
VI
2015
FLIR V2
12k
VI
2015
NYU-D YouTube-SM
1.4k
VI
2012
6.1M
VAT
2016
AVA
58K
VAT
2017
VIDAL-10M (Ours)
10M
VIDAL
2023
depth (Silberman et al., (012)), which require spe- cialized equipment and manual annotation, has been challenging. This has severely limited the scale
of the data and its alignment with other modalities. Although existing work like ImageBind (Gird- har et al. 2023) has attempted to bind various image-paired datasets and achieve indirect semantic alignment between different modalities, this approach still faces issues of incomplete and indirect data alignment. Thus, there is an urgent need for multi-modal datasets with direct semantic aligned data pairs, especially for modalities with five more types.
3

---

Published as a conference paper at ICLR 2024
Contrastive Leurning
Multi-modue Joint Learning
Langunge Binding
Video Encoder
Infrared Encoder
Audio Encoder
TestE Encoder
Encoder
Encoder
LoRA Fine tuning
Token Masking
Tokenizer
Patch Emhedding
mam in setag deraitedd ircoodile
Modality Extending
kutber abboce
Language 7
Infraved
Deppth
oé modally
Language Encoder
Multi-modal Encoders
Vanalind
Tnummhi
e
Frence
Figure 3: LanguageBind overview. The language encoder parameters are frozen, while the multi- modal encoder parameters can be adjusted using the LoRA technique. By employing contrastive learning between language and other modalities, LanguageBind successfully achieved multimodal joint learning, thereby fostering semantic alignment across different modalities.
3
METHOD
In this section, we present LanguageBind, multi molli-moddl pretraining approach designed to align the semantics of different modalities and enhance cross-modar retrieval and zero-shot classification. As shown in Figure LanguageBind consists of three parts: (a) multi-modal encoders, (b) language encoder, and (c) multi-modal joint learning.
3.1 MULTI-MODAL ENCODERS
For other modalities besides language, employ the 24-layer, 1024-dimensional vision trans- former with a patch size of 14. The encoders are initialized from OpenCLIP-large (Ilharco et al., 2021 Depth and infrared are treated RGB images, which are replicated times in the channel dimension to align with RGB images. Following ImageBind, audio data is transformed into spec- trograms with duration of 10 seconds (128 mel-bins) and repeat and pad the spectrograms. For example, 4-second spectrogram would be repeated twice and then padded with zero for an additional 2 seconds. Similarly,
also replicate it times in the channel dimension. If the duration exceeds 10 seconds, we randomly sample three 10-second audio segments, each from the front 1/3, middle 1/3, and back 1/3 of the original audio, and finally stack them together.
Patch masking To address the inefficiency in processing all tokens within the encoder, we divide the image into patches and take a small portion of patches by encoder mask ML, following MAE He et al. 2022). Given modality m R XF W'C where (H, W) represents the resolution of original data, with C denoting the number of channels. We first transform it into patches using patch embedding layer with non-overlapping filters. This operation produces patches denoted as m' NxC and N H*W represents the resulting sequence length, where S represents the size of each patch. Subsequently, positional embedding is applied to the visible tokens, which are divided by encoder mask. The combined sequence æ is defined as:
x
{m] + P.Jiem.
(1)
where P is sequence of learnable position tokens, and represents the position index at patches.
LoRA fine-tuning We employ the LoRA technique (Hu et al. 202T) to accelerate fine-tuning. For modality-agnoste encoder with weight matrix Wo Raxx we maintain the weight matrix Wo frozen while learning a new weight matrix B.A. For instance, in the case of the modality- encoder h(-) and the forward process can be represented as follows:
h(x=)= Wox BAw
4

---

Published as a conference paper at ICLR 2024
where B R E R**k with being the minimum of d and k. It is important highlight that both Wo and BA possess the same imput and output dimensions, facilitating their summation to produce the final output. Modality extending To extend the LanguageBind method to multiple (N) modalities, the first step involves processing the data into a sequence of tokens. Subsequently, the parameters through token masking and LoRA fine-tuning while keeping the language encoder frozen. Finally, this modality is aligned with the language feature space.
initialized from OpenCIIP. The encoder for different modalities is then trained
3.2 LANGUAGE ENCODER AND MULTI-MODAL JOINT LEARNING
For the language encoder, we utilize 12-layer transformer model with 768-dimensional and ini- tialize it from OpenCLIP. For given text, we initially employ a BPE tokenizer to segment words into relatively common subwords. Each subword corresponds are embedded within a word embedding layer. Ultimately, the tokens are encoded by the language encoder to obtain text logit R Lxx where L represents the length of the sequence. To ensure alignment across different modalities, we implement contrastive learning principles (Radford et al.) 2021]. The objective of this approach is to increase the similarity of paired data, bringing them closer to the same semantic space, while minimizing the similarity of unpaired data. We utilize contrastive learning to bind individual modalities language.
unique token, and these tokens
exp(x; V1/T) 1=1 Σ [x=[exppx]] us/T)*
K
1
K Σ
exply, x4/T)
LM2T = K log
Σ
Lr2M
log
(3)
K
2j=1 exp(y? xTyT)
where F; is the i-th modality data and yj is the j-th text and their features are normalized. and are batch size and the temperature. The direct alignment of each modality M with language T enables us to significantly enhance zero-shot classification and retrieval tasks.
THE VIDAL- 1OM DATASET
In this section, we will describe how to construct our VIDAL-IOM dataset, including million pairs of video-language data, million pairs of infrared-languaee data, 3 million pairs of depth-language data, and million pairs of audio-language data. As shown in Figure the collection process consists of three main steps: visual search term database construction (Section 4. video and audio collection and filtering (Section 4.2) and modality generation and enhancement (Section 4.3
COCO Comminn Objects in Cortest
frressound
YouTube Shorts
Modality G Generation and Enhancement
D Depth
IMA GENET
predictior
Video
TIR predictor
Vision-Related D Datasets
Audio
Infrared
coptionee
Hashtags & Title
Search Term Database
Text
Dataset
Search
Language- based Collection
Video-Text A AdxioTextt
Modality
Five- modal
Text
Terms
Extension
D Dttaset
Figure 4: VIDAL-10M construction. (a) Firstly, search term database is generated by leveraging visually related datasets. (b) Subsequently, relevant videos and audios are collected from the internet and undergo series of filtering processes.
(c) Lastly, perform infrared and depth modality generation, as well as multi-view text generation and enhancement.
4.1 VISUAL SEARCH TERM DATABASE CONSTRUCTION
To build a video dataset with rich visual concepts and diversity, we design unique search term acquisition strategy. We leverage text data including labels and captions from various visual task
5

---

Published as a conference paper at ICLR 2024
datasets ( (YouTube-8h) 8M FAbu-EI-Haija et al. 2016). MSR VTT (Xu et al. 2016), COCO Lin al. 2014). AVA (Gu et al. 2018). HMDB-51 (Kuehne et al., 2011), ImageNet (Dng et al. )009) to create large-scale search term database with diversity and broad upplicability. Then filter these search terms based on their frequency and employ the NLTK toolkit for part-of-speech tagging, followed by tallying the occurrences of keywords (nouns and verbs). A balanced subset of 100,000 search items corresponding to these keywords is then extracted as the final search team database.
4.2 VIDEO AND AUDIO COLLECTION AND FILTERING
During the data collection process, we utilize the aforementioned search terms to retrieve video-text pairs and audio-text pairs from relevant platforms, e.g. YouTube Shorts, and Freesound. Regarding video collection, in order to obtain short videos with high-quality textual descriptions, we imple- mented a filtering mechanism for the title and hashtags. Video samples with titles containing less than 2 words and without video hashtag labels excluded from dataset. Moreover, moved irrelevant words and hashtags, such as "youtube", "fyp". "shorts", etc. Furthermore, to ensure complete, consistent, and precise depiction of the event within a single full video, de- cide to impose a duration limit of 20 seconds. Shorter videos tend to exhibit better scene coherence and event integrity and are more closely aligned with corresponding hashtags and title descriptions. Ultimately, we obtain a short video dataset that encompasses more specific rather than abstract con- tent. Concerning audio collection, we rank the audio list on different audio platforms based on its similarity to the search terms. Additionally, we conduct filtering operations similar to those for videos, taking into account factors such as audio ratings, download counts, user comments, tags, and duration. This comprehensive approach allows us to curate and refine the audio and video content more effectively.
4.3 MODALITY GENERATION AND ENHANCEMENT
Multi-view text generation and enhancement The language modality of VIDAL-IOM consists of multi-view texts, including title, hashtags, keyframe captions, video captions, and enhanced cap- tions. The detailed text generation and enhancement pipeline is illustrated in the Appendix, section A. Hashtags in VIDAL-1OM depicted in the video. These hashtags serve as key indicators, emphasizing the focal points and dy- namic elements of the video. However, hashtags alone may not fully capture the spatial information conveyed by the video frames. To address this limitation, we leverage the image captioning model OFA (Wang et al. (202b) to generate supplementary keyframe captions that enrich the spatial infor- mation at the keyframe level. These captions also contain local temporal information related to the video content, which beneficial for visual-text pretraining. Besides spatial information, temporal information concealed within the video is equally significant, providing crucial insights into the pro- gression and sequencing of events within the video. To further supplement the overall thematic and temporal information of the video, we employ the mPLUG-owl model (Ye et al., 2023) to generate video captions based on the combination of video, title, and hashtags. By leveraging the title and hashtags as accurate video labels, we guide the mPLUG-owl model generate captions that align with the video theme, reducing potential model bias to a certain extent. Furthermore, to extract valuable information from the generated video captions, we utilize the ChatGPT model to refine and enhance the textual description, thereby greatly improving the quality of the text. By incorporating the above text components, multi-view textual descriptions provide comprehensive and detailed representation of the video content.
specifically designed to highlight the main subjects and actions
Infrared and depth modality generation In the field of depth and infrared, creating modal datasets typically requires specialized equipment and human effort, resulting in limited data. Despite the success large-scal pretraining models (Radford et al., 202T Wu et al., 2023 Luo et al., 2022, Chen et al., 2023b) in NLP and CV, there remains a lack of large-scale data in this field. To address this challenge, we propose using advanced generative models specifically to construct a large-scale dataset of depth and infrared. The sRGB-TIR model (Lee et al. 2023) is used for infrared modality generation and the GLPN model (Kim et al. 2022) for depth modality generation, generating depth and infrared from keyframes in videos. While some limitations may exist, our collection of millions of video frames and corresponding texts with highly diverse semantics can significantly reduce the presence of model biases.

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS AND RESULTS
In this section, we evaluate the effectiveness of LanguageBind in various downstream tasks. Firstly, LanguageBind's capability align video and is assessed using zero-shot video-text retrieval. Additionally, we use LanguageBind to enhance the performance of downstream tasks that involve depth, infrared images, and audios. Finally, we conduct ablation experiments to analyze the impact of different parameter configurations and text descriptions on LanguageBind's performance.
5.1 ZERO-SHOT RETRIEVAL IN VIDEO-LANGUAGE
Comparison to prior methods In the zero-shot text to video retrieval benchmark, we utilize ViT- L/14 as the video encoder and add temporal attention layers for fair comparison, which can found in Appendix B According to the results presented in Table 2 the performance of LanguageBind ex- ceeds that of VideoCoca (Yan et al.) 2022) and OmniVL (Wang et al., (022a) by 8.3% and 8.0% respectively MSR-VTI A.R-VTT. In comparison to the ImageBind model utilizing the Vit-Huge architec- ture, the LanguageBind model, employing the Vit-Large model, showcases superior experimental outcomes. Furthermore, compared to models based on CLIP-Large but using more training data, LanguageBind achieves superior performance on four datasets, outperforming InterVideo (Wang al. (202cc) by 1.9% on MSR-VTT, 8.8% on MSVD, 6.3% on DiDeMo. and 4.4% on Neti We also exceed TVTSv2 (Zeng et al. 2023) by 4.4% and 3.2% on MSR-VTT and DiDeMo, respec- tively. Moreover, we outperforms UMT-L et (2023b) on all datasets. For a fair comparison of dataset validity, we use the Vit-B/32 model of CLIP4CLIP to conduct validation experiments using the 100K subset VIDAL-- 10M and the 380k subset of How wwwToll 10M.. As shown in Table2, the VIDAL-100K outperforms the HT100M-380K on both MSRVTT and MSVD datasets, validating the effectiveness of our dataset.
Table 2: Zero-shot text to video retrieval performance of LanguageBind across four datasets.
MSR-VTT
MSVD
DiDeMo
ActivityNet R@s R@10 R@ R@5 R@ 10
Method
Dataset
R@1 R@5 RØ 1010 10 R@I
R@s R@10|R@1
Non-CLIP models
OmniVL
14M
34.6
58.4
66.6
33.3
58.7
68.5
VideoCoCa
100M
34.3
57.8
67.0
34.5
63.2 76.6
CLIP-H/14
ImageBind
36.8
61.8
70.0
CLIP-1/14
UMT
5M
33.3
58.1
66.7
44.4
73.3
82.4
34.0
60.4
68.7
31.9
69.2
72.0
TVTSv2
8.5M
38.2
62.4
73.2
34.6
61.9
71.5
Intern Video
12.8M
40.7
43.4
31.5
30.7
LanguageBind
3M
42.6
65.4
75.5
52.2
79.4
87.3
37.8
63.2
73.4
35.1
63.4
76.6
Table Zero-shot text to video retrieval to verify the effectiveness our dataset.
Dataset
Method
Parameter
Source
R@11
R R@5t
R@ 10
MR.
CLIP4Clip CLIP4Clip
86M
WIT400M, HT HT100M--80L
32.0
57.0
66.9
4.0
MSR-VTT
86M
WIT400M, VIDAL-100K
35.7
60.8
71.5
3.0
CLIP4Clip
86M
WIT400M, HT100M-380k
38.5
66.9
76.8
2.0
MSVD
CLIP4Clip
WIT400M, VIDAL-110K
42.0
70.0
79,2
2.0
5.2
ZERO-SHOT IN MULTIPLE MODALITIES
Zero-shot X-Language classification We compare our model with the recent state-of-the-art multi- modal pretraining models, OpenCLIP (Ilharco et al. 2021) and ImageBind (Girdhar et al. (023) on multi-mode understanding ability tasks in Table 4 For video zero-shot classification, we out- perform ImageBind by 14.0% with a smaller model Kinetics-400 (Kay et al. 2017). and also report the results multi-view/crop (Simonyan Zisserman 2014) on OpenCLIP for further comparison. For infrared, LanguageBind exhibits noteworthy 23.8% performance advantage

---

Published as a conference paper at ICLR 2024
ImageBind the LLVIP and outperforms OpenCLIP all three datasets (LLVIP, FLIR VI, and V2). For depth images, our zero-shot results on NYU-D surpass ImageBind by substantial margin of 11.1% and outperform OpenCLIP by 19.7%. For audio, outperform ImageBind 10.1% on Audioset dataset and 1.1 % on VGGSound dataset. We outperform ImageBind by a large margin of 23.9% on the ESC-50 dataset.
Table 4: Zero-shot X-Language classification. We present the top-1 accuracy results for all datasets, where AS-A represents Audioset Audio-only (Gemmeke al. 2017).
Video
Infrared FLIR V1
Depth NYU-D
Audio ESC-50
Method
Size
K400
K600
LLVIP
FLI V2
AS-A
VGGS
ImageBind
Huge
50.0
63.4
54.0
17.6
66.9
27.8
OpenCIIP
Large
60.7
59.0
82.2
81.2
42.6
LanguageBind
Large
64.0
61.9
87.2
82.9
48.0
65.1
27.7
91.8
28.9
Zero-shot Audio-Language retrieval We compare zero-shot text-to-audio retrieval on Clotho and Audiocaps datasets. LanguageBind outperformes AVFIC (Nagrani et al. 2022) and ImageBind by margins of 9.1% and 6.1% Clotho and by 2.9% and 5.5% on Audiocaps, respectively. Moreover, LanguageBinn passes the powerful baseline of VALOR (Chen al. 2023a by 3.7% on the Clotho dataset.
Table 5: Zero-shot Audio-Language retrieval
Clotho
Audiocaps
Method
R@1
R@ 10
R@!
R 10
AVFIC
3.0
17.5
8.7
37.7
ImageBind
6.0
28.4
9.3
42.3
VALOR
8.4
LanguageBind
12.1
44.0
12.2
53.2
Zeor-shot langauge-based multi-modal joint retrieval In Table 6 we conduct multi-modal joint retrieval explore the complementarity joint space. We report the R@1 scores on MSR- VTT and Place datasets, while reporting accuracy on other datasets. For MSR-VTT, we only evaluate using videos that include audio. Integrating audio embeddings for video-language retrieval further improves performance, increasing it from 41.4 to 42.0. Similar trends have heen observed in other modalities, where each modality has the potential to enhance the performance when combined with other modalities. These results demonstrate that LanguageBind is capable of learning more sistent feature space.
zero-shot retrieval As shown in Table 7 we explore the zero-shot performance of emer- gency coverage in four datasets, including RGB images, audio. infrared, and depth. Due to the novelty of approach, there are "fair" baseline models for comparison. Nonetheless, we com- pare our results with ImageBind, which aligns with images directly. For example, we achieved R @ of 10.6 and 10.0 AVE (Tian et al., | (2088) and VGGS, respectively. On each benchmark, the performance of emergency zero-shot retrieval achieves significant gains, even approaching results obtained by incorporating textual features. These results suggest that LanguageBind aligns various modalities and implicitly transfers text supervision associated with specific modalities and tasks.
Emergent
Table 6: Zeor-shot langauge-based multi- modal joint retrieval. donates that it is not clear whether only videos with audio are in- cluded. donates that dark nighttime images.
Table 7: Emergent zero-shot retrieval. donates that randomly select 10% data to test.
Dataset
Method
Task
Emergent
R @
Ours
10.6
Dataset
Method
Task
Top-1
AVel
ImageBind
V-A
x 36.9
V-T
36.1"
ImageBind
Ours
V
10.0
MSR
A+V-TT
36.8 (407)
VGGSt
ImageBind
V-A
X 28.7
V-T
41.4
Ours
RGB+1
V
7.5
A+V-T
42.0 (406)
RGB+T-11
X
9.1
LLVIP
Ours
ImageBind
D
54.0
I+RGB
9.3
NYU
D
65.1
D+I-RGBB
X
10.6
Ours
RGB-T
76.0
RGB-
17.9
D+RGB-F7
(-.4)
RGB+T->
X
18.3
NYU
Ours
LLVIP
Ours
RGB +T
62.4
D-RGB
24.5
I+RGB
79.3 (4169)
D+T-RRGB
X
25.7
8

---

Published as a conference paper at ICLR 2024
5.3 TRAINING LOss AND ARCHITECTURE
Following ImageBind, we mainly focus on depth and infrared, which are visual and spatial modality, We report R@ score for Clotho and top-1 accuracy for others.
Training epochs. We conduct an experiment in Table to study the effect of training epochs which shows that the LoRA fine-tuning is highly effective. Although 3 epochs of training regimen yield superior accuracy, we chose to optimize for a single epoch, achieving balance between performance and training cost.
Training batch size. In Table [b) we evaluate the effect of batch size on representation learning.
The experiments have shown that a larger batch size is not necessarily better. Training strategy. exhibits the poorest performance, likely due to the lack of prior knowledge from CLIP pretraining. On the other hand, full tuning shows significant improvement compared to training from scratch. This highlights the positive impact of leveraging prior knowledge in the form of pre-trained weights. Meanwhile, the LoRA method stands out for its advantages in terms time and memory cost. requires less time and memory resources compared full tuning. Furthermore, LoRA outperforms full tuning on multiple datasets such as LLVIP, FLIRvI, and Clotho. This indicates that LoRA is only efficient but also effective in learning new knowledge specific to different domains while better retaining the previously acquired knowledge from the pre-trained OpenCLIP models.
Table Sc we compare three different strategies. Training from scratch
Rank of LoRA. As detailed in Table Rd We observe that smaller rank values lead to more significant performance improvements, whereas larger rank tends to decrease performance. This trend may be attributed the potential overfitting of the model.
Temperature for loss. As shown in Table Sc] we find that the Iearnable temperature initiated from 0.07 performs best, outperforming the fixed temperature strategy proposed by ImageBind.
Masked ratio. We explore the impact of different mask ratios in Table 8f The results show that a mask ratio of 0.5 demonstrates the highest performance, requiring only a quarter of the computational resources, aligning with findings in FLIP (Li et al. 2023c).
Table 8: Training loss and architecture design decisions and their impact on zero-shot classifica- tion. Settings for results Section 5.2 highlighted gray
(a) Training epochs
(b) Training batch size
(c) Training strategy
Dataset
Dataset
512
lk
2k
Scratch
Full tuning
LoRA
NYU-D
65.1
64.5
NYU-D
63.9
65.1
64.5
Time
1.4h
1.4h
0.8h
LLVIP
83.9
81.1
LLVIP
80.0
83.9
78.6
Mems
278M
278M
132M
FLIR
82.9
85.0
FLIR
81.6
82.9
85.2
LLVIP
57.1
85.1
87.2
FLIR V2
48.0
44.7
FLIR V2
45.1
48.0
47.9
FLIR V1
74.7
81.3
81.6
ESC-50
86.8
88.9
87.4
Clotho
8.8
9.8
10.1
(d) Rank of LoRA
(e)
Temperature for loss
(l) Masked ratio
Dataset
4
8
Dataset
Learn
0.05
0.1
Dataset
0.0
0.3
0.5
0.7
NYU-D
65.1
64.4
64.7
NYU-D
65.1
63.0
62.7
NYU-D
64.8
65.1
62.7
LLVIP
83.9
78.0
LLVIP
83.9
81.8
83.1
LLVIP
80.3
79.9
83.9
81.5
FLIR V1
82.9
74.4
FLIR V1
82.9
83.3
FLIR V1
83.5
84.2
82.9
81.9
FLIR V2
45.8
FLIR V2
48.0
45.0
43.2
FLIR V2
43.2
44.0
48.0
42.5
CONCLUSION
In this work, we propose the LanguageBind, a language-based semantic alignment method for multi- modal pretraining. We employ contrastive learning to establish modality semantic alignment be- tween the language modality and all other modalities. To improve modal integrity, we also construct the first large-scale multi-modal dataset directly aligned to language modality, VIDAL-IOM, com- prising 10 million aligned VL, IL, DL, and AL pairs. Extensive experimental results, including zero-shot X- X-language comprehension and indirect alignment between different modalities, demon- strate the effectiveness of LanguageBinde multimodal alignment and complementary capabilities, well the effectiveness of VIDAI-10M.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China (2022ZD0 18101), the Nat- ural Science Foundation of China (No.6220201),, Shenzhen Basic Research Program under Grant JCY12022081115173600 and also sponsored by CCF Tencent Open Research Fund.
REPRODUCIBILITY STATEMENT
1. For LanguageBind approach details.
(a) We provide comprehensive overview of the multi-modal encoder, detailing its archi- tecture and functionality in Section
(b) We outline the language encoder in Section 3.2
(c) We expound the methodologies employed for multi-modal joint learning in Sec- tion 3.2
2. For VIDAL-IOM dataset construction details.
(a) describe the procedures employed to construct the search term database Sec-
tion 4.
(b) We provide insights into the strategies used for collecting and filtering video and audio data within VIDAL-I0M in Section 4.2
(c) We elaborate the generation of infrared and depth data, well the processes involved multi-view text generation and enhancement in Section 4.3
(d) We promise to release the VIDAL-1OM dataset upon publication.
3. For setting details.
(a) We describe in detail the training hyperparameters in Appendix B
(b) We deseribe the setup of the downstream task dataset Appendix C
REFERENCES
Sami Abu-al-HHija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasinhan. Noutube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609 1r..::1600900675, 2016.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments video with natural language. Proceedings of the IEEE international conference on computer vision, pp. 1003-5812,20 2017.
Matheus de Freitas Oliveira Baffa and Lucas Grassano Lattari. Convolutional neural networks for static and dynamic breast infrared imaging classification. In 2018 31st SIBGRAPI Conference Graphics, Patterns and Images (SIBGRAPI), pp. 181. IEEE, 2018.
Nicolas Bonneel, Kalyan Sunkavalli, James Tompkin, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. Interactive intrinsic video editing. ACM Transactions on Graphics (TOG), 33(6):1-10, 2014.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen. Estimating depth from monocular images as classification using deep fully convolutional residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 28(11)3314-3118 2017.
Juan Casares, A Chris Long, Brad A Myers, Rishi Bhatnagar, Scott M Stevens, Laura Dabbish, Dan Yocum, and Albert Corbett. Simplifying video editing using metadata. In Proceedings the 4th conference on Designing interactive systems: processes, practices, methods, and techniques, pp- 157-166, 2002.
10

---

Published as a conference paper at ICLR 2024
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1558-33568 2021.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies, pp. 190-200, 2011.
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: V'sion-audio-languaaa onni-perception pretraining model and dataset. arXiv preprint 0004::2300.08345, 2023a.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A iiiio--addo-suuiill--eee omni-modality foundation model and dataset. arXiv preprint arXiv:2305.18500, 2023b.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le. and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.0286)/ 2019.
Yashar Deldjoo, Mehdi Elahi, Paolo Cremonesi, Franca Garzotto, Pietro Piazzolla, and Massimo Quadrana. Content-hased video recommendation system based on stylistic visual features. Jour- nal on Data Semantics, 5:99-113, 13, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255..0 leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Frederic Font, Gerard Roma, and Xavier Serra. Freesound technical demo. In Proceedings of the 21st ACM international conference on Multimedia, pp. 41 411-412, 2013.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190. 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedingy of the IEEE conference on computer vision and pallern recognition, pp. 6047-6056, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.

---

Published as a conference paper at ICLR 2024
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint Sef1::2006006685, 2021.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL httph:tt//// ://cci. org/ If you use this software, please cite it as below.
Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. LIvip: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3496-3504, 2021.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1rXiv:1705.06950, 2017.
Jacob Devlin Ming-Was Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naa,L-HLT, volume 1, pp. 2, 2019.
Doycon Kim, Woonghyun Ka, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, and Junmo Kim. Global-local path networks for monocular depth estimation with vertical cutdepth. arXiv preprint arXiv:2201.07436. 2022.
Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. Mmact: large-scale dataset for cross modal human action understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8658 8
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 201 International conference on computer vision, pp. 2556-2563 IEEE, 2011.
Dong-Guve Lee, Myung- Hwan Jeon, Younggun Cho, and Ayoung Kim. Edge-guided multi-domain rgb-to-tir image translation for training vision tasks with challenging labels. In 2023 IEFE Inter- national Conference on Robotics and Automation (ICRA), ), pp. 8291-8298. IEEE, 2023.
Hao Li, Curise Jia, Peng Jin, Zesen Cheng, Kehan Li, Jialu Sui, Chang Liu, and Li Yuan. Freestyleret: Retrieving images from style-diversified queries. arXiv preprint 1rXiv:2312.02228. 2023a.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. arXiv preprint arXiv:2303. 16058, 2023b.
Yanghao Li. Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer. and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp.
2023c.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Rision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei. Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing. 508: 293-304, 2022.
12

---

Published as a conference paper at ICLR 2024
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m, Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp-
2630-2640, 2019.
Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In Proceedings the 2018 ACM on International Conference on Multimedia Retrieval, pp. 19-27, 2018.
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. European Conference on Computer Vision, pp. 407-426. Springer, 2022.
Kamalesh Palanisamy, Dipika Singhania, and Angela Yao. Rethinking enn models for audio classi- fication. arXiv preprint Miiv:2007.11155: 2020.
Karol Piezak. Ese: Dataset for environmental sound classification. In Proceedings of the 23rd
ACM international conference on Multimedia, pp. 1015-1018, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: a local svm approach. In Proceedings of the 17th International Conference Pultern Recognition, 2004. ICPR 2004., volume 3, pp. 32-36. IEEE, 2004.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556-2565, 2018
Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint (eXi:::804.09626, 2018
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In Computer Esioo-EECY 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
John R Smith and Shih-Fu Chang. Image and video search engine for the world wide web. In Storage and Retrieval for Image and Video Databases V. volume 3022, pp. 84-95. SPIE, 1997.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 121..002, 2012.
Teledyne FLIR. Free teledyne flir thermal dataset for algorithm training. https::/www/ httpss://wwwwwt. flir. foo//omm/a addas//aaa-aattssttttteeee
2015a. Accessed: 2023-09-16.
Teledyne FLIR. Free teledyne flir thermal dataset for algorithm training. https:/ flir.
com/ cem///ee/aaa adas adas dataset- form/| 2015b. Accessed: 2023-09-16.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pp. 247-263, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2024
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for lange-la and video-language tasks. Advances in neural information processing systems, 35:5696-57 10, 2022a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequenee learning framework. arXiv preprint arXiv:2202.03052, 2022b.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang. Zhiyu Zhao, Hongjie Zhang. Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 2022c.
Yusong Wu", Ke Chen Tianyu Zhang*. Yuchen Hui*, Taylor Berg- Kirg-Kirkpa,rick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. In IEEE International Conference Acoustics, Speech and Signal Processing, ICASSP, 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Hongwei Xue, Tiankai Hang. Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5036-5045, 2022.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979 2022.
Taojiannan Yang. Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. arXiv preprint arXiv:2302.03024. 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14177 2023.
Ziyun Zeng, Yixiao Ge, Zhan Tong, Xihui Liu, Shu-Tao Xia, and Ying Shan. Tvtsv2: Learning out-oo-the-box spatiotemporal visual representations at scale. arXiv preprint arXiv:2305. 14173, 2023.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao. and Hongsheng Li. Pointelip: Point cloud understanding by clip. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 8552-8562. 2022.
Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xi- angyu Yue. ceca-traasformmr. A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802 2023.
Bin Zhu, Peng Jin, Munan Ning, Bin Lin, Jinfa Huang, Qi Song, Mingjun Pan, and Li Yuan. Llm- bind: A unified modality-task integration framework. urXiv preprint arXiv:2402. 14891, 2024.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A STATISTICS OF VIDAL- 10M DATASET
Hashlage
pet
hrac
Heshtags
Espiesio
not only
Title: timball exploded in the sky.
KeyErame Captions white with
dock'. .....]
Keytrame Captions large explosion
with
foragoumad" -...]
Video Caption: This video desembee a cine Iennne for
with its distinctive cors laying on piar, looking at the
Video Caption: the vodes, massive
mesing sdd.. It emphasizes the mpportunne
explodes sky duinng
feusee lazge
sot ouly beering,
indusiiiaa accident, creeting dracesiic
also for maitleiting belazce and keeping cool. the best.
and powerful scene.
Hashlage Eirsbali
Explosin
Sky
Hashlags: eveting
Foeris wheel
Title: Baseball player unleacting powerful swings on the feeld
Kochikode
evering vibe
Keyframe Captions: baseball player swinging a but fickd", cene
K Keyframe Captinss ts Perss wheel amasentent park'"
Video Caption: cricket match where
player
procticing his
shot hirting
Caption: beach
swings the ball with bottom edge the bat to soom
ikode, India shows with people enjoying
tides on fenti wheel and other altractions.
boundary.
pairs in VIDAL-10M, with the text compo- nents comprising hashtags, title, keyframe captions, and video caption. Examples are taken from 4 distinct clusters, corresponding Sports, Pets & Animals, News & Politics, and Education.
Figure 5: Examples of seeeo--udd--tee--dppplee
In order to build a video dataset with rich visual concepts and diversity, we develop a unique but simple search term acquisition strategy. This strategy involves obtaining search terms from various visual datasets (as shown Table Subsequently, we use these search terms to gather videos from the YouTube Shorts platform, which has become popular source for video data due to its abundance and diverse content. We collect videos in various categories, including sports, animals, nature, etc., resulting in a large and diverse dataset. Examples of dileo-auddo-texx--pptt------ VIDAL- 10M dataset are shown in Figure Moreover, to ensure data quality, we manually design list of stop words that are filtered from our datasets. These words include terms such "bts", "bmw", and "nff", among others, that are not relevant to our research.
pairs in the
Video categories and duration Furthermore, we analyze the distribution of video categories with varying durations in our datasets, as illustrated in Figure 6 The normal distribution pattern observed in this analysis indicates that dataset covers wide range of concepts. Besides, show the proportions of each category across different duration grades in the VIDAL- dataset in Figure
Table 9: Examples of textual descriptions from various datasets as search terms.
Dataset
Search terms
How to make a delicious chocolate cake.
YouTube-8M
Learn to dance salsa in 10 easy steps.
How to play chess. How to make pizza.
Howto 100M
lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens, coon bear killer whale, killer. grampus, sea wolf, Orcinus orea, giant panda, panda, panda bear
ImageNet
small boat floating on a body of water with city skyline in the background. A man with a red helmet small moped on dirt road.
COCO
Others
15

---

Published as a conference paper at ICLR 2024
Table 10: Stop words in our datasets.
viral
funny
love
fashion
subscribe
nature
asmr
motivation
fitness
art
satisfying
foryou
music india
fun bts
amazing edit
life
roblox vlog
minecraft design
marvel
explore
dubai
foryoupage comedy outfit
ootd
share
indian
lol
creative
relaxing
tattoo
random
instagram
quotes
workout sad
ideas
views bgmi
yummy
respect easy
usa
ronaldo jawellery
memes
happy
nfl
song
mlb
reel
support
nba
wow
status
gree meme
gameplay
top blackpink
whatsappstatus
follow homedecor
history
tutorial bodybuilding
japan
interiordesign
freefire
stunt
foodie animation recipe
skills tips
crazy
pov
editing
aesthetic
style view
london
reaction story
pubg
construction
challenge
healthy
bmw
uk
free
hairstyle
enjoy
motivational
messi
capcut
nailart
entertainment fifa
attitude
europe
health geography
gta
unboxing
adventure
whatsapp
fail btsarny
god inspiration relatable
comment tattoos
fy
highlights amazon
illustration
fortnite
ntb
avaiation
interior decor
travelvlog
canada
btsarmy tranding
time mtb
luxury
vlogs
picsart reels
photoshoot
business
photography
Entertaioment
10
People & Blogs
Travel Events
Science & Technology
Pets Animals
Number(log scale)
Music
Gaming
10a
Autos & Vohicles
Comedy
Education
Howto Style
Sparts
Film & Animation
10"
Politics
Nonprofits & Activism
Distribution of duration(s)
Figure 6: The number of 15 categories with different durations in our VIDAL-IOM datasets. A wide range of concepts covered.
16

---

Published as a conference paper at ICLR 2024
Film Astmation.
Sciano & Tuchandog
News Puñlies Nompeettss Acthsto
Music
Cameds
People Beogo
Rioees
Tranel & Exwete.
Education
1-7
Sports
15-20
Education
VIDAL-10M
Petx Animeé
People & Blogs
Gamning
Hoseo
Honto &
8-14
44%
Gaminn
Gaming
Rntertala....
Pttt 6un
Sperts
Howto &
Peogle
Hloge
Sclencs & Technlogy
Carredy
Atfoss Vehikles.
Norprofits
Sdrace Tedaudop
Pilre Antmaatinn
Nampesflis
Camady
Sports
Travel Exrets
Education
Pets & Antmals
Spoets
People
People Blags
Gurring
Gamiag
&owto& Style
Entertinmmet
Figure 7: The statistical distribution of categories across the three duration grades in the VIDAL- 10M dataset. The colors green, blue, and yellow represent video durations 1-7, 8-14, and 15-20 respectively.
FPS, Aspect ratio and Resolution The first aspect examined in the dataset is the Frames Per Second (FPS) of the videos. FPS refers the number of frames or images displayed per second in video. The aspect ratio of video represents the proportional relationship between its width and height dimensions. It is a critical factor in determining the visual presentation and viewing experience of the videos. The distribution of FPS and aspect ratios in Figure g provides insights into the smoothness and fluidity of the recorded content and sheds light on the various formats and orientations used. Video resolution refers the number of pixels in each dimension that video contains. It directly affects the clarity, sharpness, and level of detail in the visual content, Examining the distribution of resolutions (Figure 99) in the dataset provides understanding of the available video quality and the technological capabilities of the recorded material.
Aspect Ratio (Top 7)
FPS
8691510
1000000
9640943
8000000
6000000
6000000
4000000
4000000
2000000
4266
161806
77745
14905
2000000
25218
104784
1053386
13178
53986
478378
0.10
10.20
40
Figure 8: The distribution of FPS (Frames Per Second) and aspect ratio the videos of the VIDAL- 10M dataset.
17

---

Published as a conference paper at ICLR 2024
Height
Width
3%
4%
1%
5%
27%
1%
67%
92%
256-512 512-768
768-1024
1024+
256-512
512-768 768-1024 1024+
Figure 9; Height and width distribution of videos in VIDAL-IOM dataset.
OF4 cuptions
mPL.UG caption
OFA
1. person helding a yellow shoe on top of tible.
video describus polishing handmade cmoodile leather shoes specialized tool. The shoes Goodyca leather, technique that involves stitching sole part shor.
ChaiGPT
young looking
Video
upper carefally working un the shoes, ensuring they high-quality
CharGPT.PLLU caption
video, wsing
specialized tool to polish handerafted crocodile leather shoes made with Goodyeur technique. He is meticulussly working the shoes achieve
Ed
Tite Polss slsos handuale excoddie laabershoss Goodyaar
webod shoes
Hashings kemostillaehhrr
mPLUG-ow!
high-qualiry
Prompt "(I"
conbine
kaattag surmaise smnenes weghtt bailngg
Language Generation and Enhancement
Figure 10: Multi-view text generation and enhancement pipline. We employ the OFA model to generate keyframe captions and input video, title and hashtags into the mPLUG-owl model to obtain video captions. The video captions are further refined using ChatGPT, resulting in the ChatGPT- mPLUG caption. The final multi-view textual description comprises these components.
B PRETRAINING DETAILS
In this section, we introduce our training configuration.
Video-Language For the video-text retrieval based CLIP4Clip. verify that the VIDAL-IOM dataset highly aligned. We adopted the training framework of CLIP4Clip, and the model is ini- tialized from ViT-B/32, and the rest of the parameters the same the default settings, except for epoch and batch size of 512. For the video-text retrieval based LanguageBind, we add temporal attention before each spatial attention following Aim (Yang al. 2023). The temporal attention is initialized from the spatial attention and LoRA is applied only to the temporal attention. We add tem- poral position embedding before each temporal attention. We show the details of results as shown in Table IT] For zero-shot video classification, The text templates are sourced from OpenCLIP, with modification consisting of the substitution of "photo" with "video" across all templates.
Depth-Language- The model initialized from OpenCLIP with frozen language encoder. For each individual sample, we employ random selection approach to extract either depth image from the video sequence. Subsequently, we resize these frames to have a short edge length of 256 units, followed by central cropping process to attain dimensions of 224x224. Additionally, tripled the number channels in both the depth image. The text templates employed for zero- shot classification are sourced from OpenCLIP, with modification consisting of the substitution of "photo" " with "depth photo" across all templates. This alteration yields an approximate performance gain of 1%.
18

---

Published as a conference paper at ICLR 2024
Table 11:
Zero-shot Video-Text Retrieval Performance based LanguageBind. We show the
details of results.
Text-to- Video
Video-to- Text
Dataset
R@ 1
R@5
R@ 101
MR.
R@ 1
R@51
R 101
MRJ
MSR-VTT
42.6
65.4
75.5
2.0
37.9
63.1
73.3
3.0
MSVD
52.2
79.4
87.3
1.0
68.4
91.7
96.4
1.0
ActivityNet
35.1
63.4
76.6
3.0
32.3
62.2
74.5
3.0
DiDeMo
37.8
63.2
73.4
3.0
37.6
63.7
73.3
3.0
EeraarddLLangugee Following depth-language, it is worth noting that the text templates sponding to infrared images retain the "photo" designation, as no discernible performance improve- ment is observed from this particular modification.
Audio-Language.
The data are preprocessed as in 3.1 Unlike depth and infrared, spectrograms differ much from the domain of conventional visual images. Therefore, it is easy overfit during training, so we increase the training epoch and the rank of LoRA. Additionally, we replace "the/a photo of with "the/a sound of"" across all templates for audio zero-shot classification.
Table 12: Training setting.
CLIP4Clip
LanguageBind
Config
Video
Video
Infrared
Depth
Audio
Vision encoder
ViT-Base/32
ViT-Large/14
Optimizer
BertAdam
AdamW
Optimizer Momentum
31, B2
0.9, 0.98
31. ,8
0.9, 0.98
Epochs
16
8
Learning rate
le-4
le.4
le-4
5e-4
5e-4
Coefficient learning rate
le-3
1e-3
1c-3
1e-3
Weight decay
0.2
0.2
Batch size
512
640
1024
1024
512
Warmup steps
0
2000
200
200
2000
Temperature
learnable cosine decay
learnable cosine decay 77
Learning rate schedule
Max words
32
Max frames
12
8
Mask ratio
0.3
0.5
0.5
0.3
LoRA rank
16
2
16
LoRA alpha
16
LoRA dropout
0.0
0.1
0.1
0.1
C DOWNSTREAM DATASETS
Video-language. We perform video-text retrieval experiments on 2 datasets. et al. (016) comprises 10K YouTube videos, each paired by 200K captions. In our analysis, we present results based on the 1K-A test subset. (b) MSVD (Chen & Dolan 2011) consists of about 120K sentences and reports results test data (670 samples).
(a) MSR-VTY VTT (Xu
Infrared-language (a) LLVIP (Jia et al. 2021 constitutes a dataset for pedestrian object detec- tion within the infrared spectrum. Following ImageBind, we extracted all people from the images, designating all other objects as background elements. This process resulted in dataset comprising 7,622 "background" classes and 7,954 'person' classes, which was subsequently employed for bi- nary classification testing. (b) FLIR v1 (Teledyne FLIR 2015a) offers comprehensive annotations for both thermal and visible spectrum frames. From the test data, derived dataset containing 11,696 images by extracting bounding boxes. This dataset encompasses categories ['bicycle', "car", 'dog'. "person']. (c) FLIR v2 (Teledyne FLIR 2015b) includes 16,696 images after pro-
19

---

Published as a conference paper at ICLR 2024
cessing similarly, which were categorized into 12 classes ['bike", 'bus', 'car', 'hydrant', 'light', 'motor,, 'other vehicle', "person', "sign'. 'skateboard', 'stroller', 'truck"].
Depth-language. We use NYU-v2 Depth-only (NYU-D) (Silberman et al. 2012) to validate by 654 test samples. Through preprocessing, we constrained the depth images to a maximum depth of 10 meters. Following ImageBind, we undertook a category reorganization process, resulting in total of 10 scene categories.
Audio-language. We validate the zero-shot classification capability with the ESC-50 (Piczak) 2015) dataset, which has 2000 test audios, each uniquely labelled. For zero-shot retrieval, we use the Clotho (Font al. 2013) dataset. Each audio has 5 corresponding captions, so use text-to-audio retrieval to validate the model performance. We perpare test data following ImageBind.
D LICENSE
Unless explicitly noted otherwise, our released datasets provided users under the terms of the Creative Commons Atribution-NonCommerrca Shiie 4.0 International Public License ("CC BY-NC-SA 4.0"), in conjunction with the additional terms outlined herein. The CC BY-NC-SA 4.0 license can be accessed at https: //tetttttttt//ee///// incrreativeee vecommons. org/ 1 Licenses /by-yby-eee nc sa/4. o/degalcode By downloading or utilizing our datasets from our website or other sources, you agree to adhere to the terms of CC B Y-NC-SSA 4.0, as well as the terms outlined in our dataset Terms. In the event of any conflict between the terms of CC BY-NC-SA 4.0 and our dataset Terms, the latter shall prevail. We once again emphasize that this dataset is exclusively intended for non-commercial purposes, such as academic research, teaching, or scientific publications. We strictly prohibit any commercial use of the dataset or any derived works, including the sale of data utilization of data for commercial gain.
E ABLATION STUDY
In this section, conduct extensive experiments to investigate the impact of several factors. At first, we examine the effects different enhanced textual inputs on downstream tasks. Furthermore, we assess the impact of data volumes pretraining. In addition, explore various training strategies to enhance zero-shot classification. Finally, we conduct meticulous analysis of model training configurations to ensure robust transferability.
Table 13:
Impact of different text sources. We report the results of text-to-video R@ for zero- shot retrieval and other datasets report top-1 accuracy. MSR-VTT results were tested on 500K subset of VIDAL-1OM. "Raw caption" denotes the title & hashtags.
Modality
Dataset
Raw caption
OFA caption
mPLUG caption
ChaGGPT-mPLUG caption
Video
MSR-VTT
33.5
34.5
35.8
36.4
LLVTP
83.9
87.2
84.6
84.8
Infrared
FLIR VI
82.9
80.6
81.4
81.6
FLIR V2
48.0
45.7
46.8
46.6
Depth
NYU-D
61.5
62.1
63.9
E. IMPACT OF DIFFERENT TEXT SOURCES
In Table 13 we conduct various experiments to explore how different text sources impact language modality. We verify the effectiveness of LanguageBind, trained with text from multiple sources, across various modalities. While some text sources yield good results, we discover that single text source may not be universally suitable for all downstream tasks and datasets. In terms of video and depth modalities, the ChatGPT enhanced caption proves to be advantageous. For infrared images, the OFA performs best in the LLVIP dataset, while the raw caption achieves the highest accuracy in FLIR v1 and v2. That's why our VIDAL-IOM provides multi-vies -iel textual descriptions, allowing for flexibility in selecting appropriate text source that caters to diverse task requirements.
20

---

Published as a conference paper at ICLR 2024
E.2 SCALING THE SIZE OF DATASET
We analyze the impact of different data amounts on MSR- VTT and report the @@l score for zero-shot retrieval shown in Figure Our findings indicate that an increase in data amount leads significant improvement in recog- nition performance. Specifically, the performance of 3M ChatGPT-enhamn text surpasses that of 500k and 100k data by 0.9% and 1.6%, respectively.
37
VET
35
R@1
33
Furthermore, the trends observed in both video-to-text retrieval and text-to-video retrieval consistently demon- strate that the interaction between modalities plays piv- otal role in enhancing the learning process. Consequently, Figure 11: Scaling pretraining data size.
10
100
500
3000
Data size (K)
with the expansion of data size, the textual descriptions within the VIDAL- 1OM dataset align more closely with the video content and demonstrate increased scalability.
21


Reviews:
[{"review_id": "vHedvDpdIA", "reviewer": "Reviewer_fQRK", "summary": "The authors introduce LanguageBind, an alternative approach to ImageBind where language is the primary modality that all other modalities are aligned to (instead of images). They also introduce a new dataset called VIDAL-10M that contains language-aligned data for visual, infrared, depth, and audio modalities. LanguageBind achieves state-of-the-art zero-shot classification on various infrared, depth, and audio benchmarks, as well as zero-shot video-text retrieval. Finally, they analyze the impact of scaling their dataset size on MSR-VTT R@1 and provide some training ablations that measure changes in zero-shot classification on NYU-D as a function of training epochs, batch size, LoRA rank, loss temperature, and masking ratio.", "questions": "### Audio Processing\n\nI don't understand how the authors are processing the audio. \n\nThey state \"For example, a 4-second spectrogram would be repeated twice and then padded with zero for an additional 2 seconds.\" Why isn't a 4-second spectrogram simply padded with zero for the remaining 6 seconds? ImageBind does not repeat spectrograms. From ImageBind Appendix B.1: \"For audio, we process each raw audio waveform by sampling it at 16KHz followed by extracting a log mel spectrogram with 128 frequency bins using a 25ms Ham- ming window with hop length of 10ms. Hence, for a t second audio we get a 128 ×100t dimensional input.\" \n\nI'm also confused about this sentence: \"If the duration exceeds 10 seconds, we randomly sample three 10-second audio segments, each from the front 1/3, middle 1/3, and back 1/3 of the original audio, and finally stack them together.\". What is being stacked along what dimension exactly? \n\n### VIDAL-10M\n\n* How are the multi-view text annotations used during training? Randomly sampled at each step? There's also ambiguous wording later on like in section 6.1: \"allowing for flexibility in selecting an appropriate text source that caters to diverse task requirements.\" How are the authors selecting \"an appropriate text source\" during training?  \n* There already exist short-video datasets like WebVid-10M, as the authors mention. Why not just use the sRGB-TIR/GLPN models, as well as OFA/mPLUG-Owl on those existing datasets instead of constructing this new one? At first, I thought the motivation for VIDAL-10M was to obtain a multimodal dataset with more modalities than existing datasets, but the \"new\" modalities (infrared, depth) are just generated by these models. Not clear why you need to collect the audio/video in the first place if that's the case. WebVid videos have an average duration of 18 seconds, which seems similar to VIDAL-10M. Perhaps I'm missing some of the details here, but if so it might be beneficial to highlight these differences more clearly.  \n\n### Miscellaneous \n\n* Section 3.2: this does not say what pooling method is used to go from text logics of length L to single normalized text vector. Typically this is done with either CLS token pooling, max pooling, or mean pooling, but the authors do not mention that here. \n* Table 3: ImageBind uses OpenCLIP. How are their numbers on LLVIP (63.4) worse than the reported OpenCLIP numbers in this table (82.2)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* While ImageBind was able to perform zero-shot classification by pairing its text encoder with other modalities, even if they hadn't been observed together during training, LanguageBind takes an alternative approach by obtaining the text-aligned data via synthesizing the rarer modalities from visual information in their collected text-paired data, and then training the model to align each modality separately to text. This is a subtle but interesting distinction. \n* The authors describe their data collection pipeline and state they will release upon publication, which would be valuable for the broader community. \n* The authors overcome the lack of pair infrared/depth data with other modalities by utilizing pretrained generative models for synthesizing a large-scale dataset is an interesting research direction that is currently gaining popularity. The VIDAL-10M dataset could be a fruitful playground for future research on scaling synthetic data generation. \n* The scaling curves presented in Figure 5 are promising. They suggest one could continue scaling the techniques in this paper to continue advancing the state-of-the-art. \n* Typically, models like CLIP are trained with the goal of producing a strong image encoder. It is interesting that LanguageBind is able to achieve competitive results by aligning to the text encoder instead.", "weaknesses": "* __Lack of ablations__: The authors only provide a limited set of ablations for a single modality (depth) on a single dataset (NYU-D). It is not clear whether the ablated decisions would impact other datasets or modalities. This is especially true because of the results in Table 4, which suggest each modality and dataset responds differently to the kinds of text annotations used, as stated by the authors. Providing a small amount of ablations on just one of these combinations makes the paper seem incomplete. Furthermore, since this paper is explicitly comparing to ImageBind, which provides extensive ablations, this paper would be much more convincing with a broader set of ablations to match. \n* __Model release__: it is unclear whether the authors intend to release their models, which is a bit unexpected since they state they will release the dataset, and the model weights should be fairly small since they are mostly LoRA modules applied to OpenCLIP.\n\n**Update**: the authors have incorporated more ablations in the rebuttal, along with a statement on model release, that adequately address my concerns. I have increased my score on \"soundness\" to \"good\" and my overall rating to \"accept, good paper\" to reflect this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVKAdlkTuo", "reviewer": "Reviewer_ZKDg", "summary": "The paper proposes LanguageBind, a method for training encoders of multiple modalities in a joint embedding space by aligning them to a frozen text encoder. Additionally, the authors introduce VIDAL-10M, a multimodal dataset that includes data for 4 modalities with paired textual descriptions. The authors rely on multiple third-party tools for creating VIDAL like OFA, mPLUG-Owl, and ChatGPT, in addition to modality-specific generation modules to collect data for the infrared and depth modalities. Various techniques are utilized to train LanguageBind like LoRA tuning, masking, and initializing from pre-trained CLIP checkpoints. The authors provide zero-shot retrieval and recognition experiments to showcase the effectiveness of their method.", "questions": "- The paper only includes LoRA results. Why are not there any full fine-tuning results given the authors collected a decent-sized dataset (10M across modalities) ?\n- Similar to the previous point, what happens if the modality encoders are not initialized with openCLIP checkpoints similar to ImageBind where only the text and image encoders are pre-trained ?\n- In the ablation, the performance drops when moving from 0.5 -> 0.3 masking which is counter-intuitive. What is the authors' explanation? What happens with no masking is used?\n- As stated above, what is the value of training all modalities in a joint embedding space if all use cases have to do with text only?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The VIDAL dataset is potentially interesting. In particular the utilization of multiple captioning models to enhance the textual descriptions for spatial and temporal information (OFA and mPLUG) as well as ChatGPT for refining the descriptions. This is further confirmed by the results of the video modality in Table.4\n2. Overall, the results reported by the authors for the different modalities+text benchmarks are strong and reflect a good performance of the model.", "weaknesses": "1. The authors try to draw parallels to ImageBind (method name, comparisons, frequent mentions in the abstract and throughout the paper). However, LanguageBind is much closer to standard CLIP training where a joint encoder is trained between textual descriptions and sensory data for a certain modality. There have been various examples of such methods ever since CLIP was introduced for image-text pre-training such as AudioCLIP, PointCLIP, VideoCLIP (only to mention a few). This is very important because the paper only includes evaluations testing the performance of each modality and text which is different than ImageBind's proposal of testing alignment that emerged indirectly by training the modalities jointly. For LanguageBind, the benefit of training all modalities in a joint embedding space is not showcased. \n2. The technical contributions of the paper are weak in terms of novelty and potential interest to the wider research community. The method is more a bag of well-established tricks (e.g. masking from FLIP, LoRA tuning, fine-tuning openCLIP checkpoints)\n3. While the VIDAL dataset is potentially interesting, the fact that all modalities other than Video and Audio are automatically generated by off-the-shelf models is concerning in terms of its long-term impact. (the dataset will likely be outdated once higher fidelity generation models are developed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ylHEwF1ZG", "reviewer": "Reviewer_Bowz", "summary": "The paper presents multi-modal pretraining approach with modalities N=5 (video, text, audio, depth, infrared) by using language as bind across different modalities. A frozen text encoder from a pretrained VL model is used as the feature extractor for the text modality and aligned with other modalities (pair-wise) using contrastive loss. It also introduces a dataset called VIDAL-10M with 10M data pairs from VL, DL, IL,and AL. The dataset and method is evaluated on standard retrieval benchmarks to show the effectiveness of the pretraining data as well as the technique.", "questions": "I would like authors to discuss all the points described above.\n\nFinal Rating: After reading the rebuttal and comments from other reviewers, I have decided not to change the score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper attempts to learn a unified embedding space for 5 modalities where the modalities are guided by language during pre-training. Such an embedding space can be very useful for tasks involving: i) multi-modal data for ex: video containing audio, ii) tasks where paired data is not available, for ex: Video-Infrared, Video-depth etc \n\n2) The paper introduces a dataset with 10M paired data from AL, VL, IL and DL which is important for driving research in the multimodal learning area as many of the real-world applications contain multimodal data. It follows a careful approach by leveraging existing vision and language models (OFA, mPLUG-owl, chatgpt etc) to collect a balanced (in topic) and diverse (in semantic) data-pairs.\n\n3) The introduced dataset and the pretrained model is shown to be useful for:\na) cross-modality video retrieval task where it outperforms its counterparts (ImageBind, CLIP-straight, CLIP4clip).\nb) AL, DL, IL zero-shot classification tasks.\nThis shows that the model has learned good representations in the joint embedding space.", "weaknesses": "1) It is not clear from the text or Table2, the size of the pretraining data used for MSR-VTT and MSVD datasets. For a fair comparison, all methods should be pretrained with same amount of data but here CLIP-Straight is trained with WIT400M only (initialized from CLIP but no fine-tuning), CLIP4clip is trained with WIT400M+HT100M-380k whereas the proposed technique (although CLIP4clip technique is used) is pretrained with WIT400M+VIDAL-10M. It would be a fair comparison if all methods use similar sized data, i.e what would be the performance of other technique like CLIP4clip if additional data (not VIDAL-10M) is used for training.\n\n2) One of the goals of learning a model from multimodal data is  that the data can use all available modalities to learn stronger representations but there are no experiments to demonstrate this, for ex: instead of using just video -> text retrieval, it would interesting to show that video+audio -> text retrieval has better performance. \n\n3) There are few other advantages of multimodal learning in situations where:\na) one of the modalities is corrupted \nb) one of the modalities has some weaknesses (videos taken in the dark, OR audio from multiple sources) \nc) one of the modality undergoes a domain change while the other doesn't (eg: videos under weather changes etc) \nbut none of these has been addressed in this paper. It would be interesting to see results on at least one of the above scenarios.\n\n4) It would also be interesting to see an experiment where the model is evaluated on retrieval task where the modalities doesn't contain text. For ex (video<->audio, video<->infrared). This will evaluate the quality of learned representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s8dwyyZpg9", "reviewer": "Reviewer_ePEX", "summary": "This paper proposes a novel pretraining paradigm called LanguageBind, which takes the language as the ind across different modalities. To this end, authors curate a large-scale multimodal dataset. Extensive experiments for different modalities demonstrate the effectiveness of the proposed paradigm.", "questions": "See the above weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and contains sufficient details and thorough descriptions of the experimental design. \n2. Extensive experiments are conducted to verify the effectiveness of the proposed method and dataset.", "weaknesses": "1. In table 2, while authors demonstrate the improvements over ImageBind on T2V and V2T tasks, these two models are trained with different backbones, model initializations, finetuning techniques, and training data. This leads to an unfair comparison, especially considering the proposed model is leveraging more video data. \n\n2. Based on my understanding, LanguageBind is initialized from OpenCLIP and continues to train on the VIDAL-10M dataset. Compared to OpenCLIP, it is difficult to tell whether the performance improvement comes from the proposed dataset or the new pretraining paradigm. \n\n3. In table 4, do the authors have any intuition why raw caption works best for the Infrared modality?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)