CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment
Authors: Hongwei Xue
Yuchong Sun
Bei Liu
Jianlong Fu
Ruihua Song
Houqiang Li
Jiebo Luo
Venue: ICLR 2023 poster (2023)

Abstract: Pre-trained image-text models, like CLIP, have demonstrated the strong power of vision-language representation learned from a large scale of web-collected image-text data. In light of the well-learned visual features, there are works that transfer image representation to the video domain and achieve good results. However, adapting image-text pre-trained models to video-text pre-training (i.e., post-pretraining) has not demonstrated a significant advantage yet. In this paper, we tackle this challenge by raising and addressing two questions: 1) what are the factors hindering post-pretraining CLIP from improving performance on video-text tasks, and 2) how to mitigate the impact of these factors. Through a series of comparative experiments and analyses, we find that the data scale and domain gap between language sources have large impacts. By these observations, we propose an Omnisource Cross-modal Learning method equipped with a Video Proxy mechanism on the basis of CLIP, namely CLIP-ViP. Extensive results show that our approach improves the performance of CLIP on video-text retrieval by a large margin. Our model achieves state-of-the-art results on a variety of datasets, including MSR-VTT, DiDeMo, LSMDC, and ActivityNet. We release our code and pre-trained CLIP-ViP models at \url{https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP}.

Published as a conference paper at ICLR 2023
CLIP-VIP: ADAPTING PRE-TRAINED IMAGE-TEXT MODEL TO VIDEO-LANGUAN ALIGNMENT
Hongwei Xue'; Yuchong Sun?; Bei Liu Jianlong Fu? Ruihua Song", Houqiang Li', Jiebo Luo University of Science and Technology of China, Hefei, China, RRemmnn University of China, Beijing, China, MMicrosoft Research, Beijing, China,
UUniversity of Rochester, Rochester, NY
ABSTRACT
Pre-trained image-text models, like CLIP, have demonstrated the strong power of vision-language representation learned from large scale of web-collected image-text data. In light of the well-learned visual features, there are works that transfer image representation to the video domain and achieve good results. However, adapting image-text pre-trained models to video-text pre-training (i.e., post-pretraining) has not demonstrated a significant advantage yet. In this pa- per, we tackle this challenge by raising and addressing two questions: 1) what are the factors hindering post-pretraining CLIP from improving performance on video-text tasks, and 2) how to mitigate the impact of these factors. Through series of comparative experiments and analyses, we find that the data scale and domain gap between language sources have large impacts. By these ob- servations, we propose an Omnisource Cross-modal Learning method equipped with Video Proxy mechanism the basis of CLIP, namely CLIP-ViL ViP. Ex- tensive results show that our approach improves the performance of CLIP on video-text retrieval by large margin. Our model achieves state-of-the-art re- sults on a variety of datasets, including MSR-VTT, DiDeMo, LSMDC, and Ac- tivityNet. We release our code and pre-trained CLIP-ViP models at https: / //github. com www.////w/ooottooooooooooooooote /ttp//Peettaaaaeeeeeeeeeeeeeeeee Xee.eeerirttttreeeeeeeeeeeeeeeee wereeeeeeeeeeeeeeeeeeeeee
INTRODUCTION
In the past few years, vision-language pre-training has achieved great success on cross-modal rep- resentation learning from large scale of web-crawled data (Radford et al., 2021; ;; Jia et al., 2021; ; Li al., 2021; Wang et al., 2021b; Zellers al., 2021: 2022: Bain al., 2021). Among them, image- text pre-trained models (Radford et al., 2021: Jia et al., 2021) have shown powerful capability for various downstream tasks, including visual understanding (Gu et al., 2021; Wang et al., 2021a; Rao et al., 2022). image-text generation (Patashnik et al., 2021; Mokady et al., 2021) and so on (Guzhov al., 2022; Zhang et al., 2022). In light of the well-learned and enriched visual representation, some works directly adapt image-text pre-trained models to video-text downstream tasks without further pre-training on video data (Luo et al., 2021; Fang et al., 2021; Gorti et al., 2022; Zhao et al., 2022), while still outperforming models pre-trained on video data (Xu et al., 2021b; Bain et al., 2021).
Utilizing an existing powerful image-text pre-trained model for further video-text pre-training (i.c., post- pretrainri))) is able reduce the required training cost by making good use of the knowledge learned from images. However, adapting image-text pre-trained models to video-text data for post- pretraining has not demonstrated significant advantage yet, and thus is still under-explored. A preliminary study is conducted by CLIP4Clip (Luo et al., 2021) which adopts McanPooling by averaging multiple frame features based the CLIP model on subset of Howto 100M (Miech al., 2019). While the improvement over directly using the image-text pre-trained model is marginal for
Equual contributon. This work was performed when Hongwei Xue and Yuchong Sun were visiting Mi- crosoft Research as research interns. *Correponning authors.

---

Published as a conference paper at ICLR 2023
either zero-shot fine-tuning settings. In this paper, we aim to explore how to effectively adapt the image-text pre-trained model (e.g., CLIP) to video-language representation learning for video-text tasks (e.g., text-to-vided retrieval).
To unleash the power of video data to adapt image-text pre- pre-trained models for post-pretraining we conduct several preliminary experiments to figure out the challenges that hinder post-rretainning First, we explore post-pretrainigg an image-text pre-trained model (i.e., CLIP) with MeanPooling on video-text datasets with different scales, including Web (Bain et al., 2021) and HD- VILA-100M (Xue et al., 2022). The result shows that the scale of data is critical for video-text post-pretraiiing Data on a small scale makes the model easy to over-fit the new data while the knowledge learned from image-text is suppressed and the performance is reduced. Second. we investigate the language domain gap between pre-training data and downstream data. By calculating the Normalized Mutual Information (NMI) on clusters of text features, we find that there is large domain gap between subtitles that are used in large-scale video-text pre-training data and descriptive texts in downstream tasks.
To mitigate the impact of the above factors, we propose CLI to adapt the pre-trained image-text model CLIP for video-text pre-training. First, we introduce auxiliary captions that have smaller language domain gap with downstream data into existing large-scale video-text data. Instead of us- ing video captioning model which may cause data leakage by training the same dataset with video-text downstream tasks, and considering a better visual captioning capability, we adopt an im- age captioning model to generate auxiliary caption of middle frame in each video. In order to adapt a Transformer-basee vision encoder to process both images and videos with minimal modifica- tion, we then propose video proxy tokens and design proxy-guided video attention mechanism for the Vision Transformer (ViT). Specifically, during attention computation in each block, video proxy tokens can interact with all tokens, while patch tokens only interact with video proxy tokens and patch tokens within the same frame. Our vision encoder only increases negligible parameters and calculations compared to the vanilla Vision Transformer while increasing the generality and extend- ability. To facilitate cross-modal representation learning from both caption- frame and video-subtitle data types at the same time, we propose an Omnisource Cross-modal Learning (OCL) method for pre-training and study series of variants to find the best fusion strategy.
Our experimental results show that our approach improves the performance of CLIP on text-to-video retrieval tasks by large margin. We also conduct ablation studies to verify the effectiveness of each part in our approach. Our contributions are summarized as follows: (1) We are one of the first to explore factors that hinder video post-pretraining on pre-trained image-text models; (2) We propose CLIP-VIL ViP that can effectively leverage image-text pre-trained model for post-pretraining; (3) We conduct extensive experiments to verify the effectiveness of our method. Our model outperforms the state-of-thee results by a large margin on four widely-used benchmarks.
2 RELATED WORK
Vision-Language Pre- Training End-to-end models (Lei et al., 2021: Xue et al., 2022: Zellers et al., 2021: Fu et al., 2021; Huang et al., 2020; 2021b; Xue et al., 2021; Li et al., 2021; Kim et al., 2021; Huang et al., 2021a; Sun et al., 2022) for vision-language pre-training are replacing the traditional approach using pre-extracted visual features by off-the-she the-shelf models (Sun et al., 2019; Xu et al., 2021b; Zhu & Yang, 2020; Li et al., 2020b;a; Chen et al., 2020). Training end-to-end models on large-scale web-collected data also gradually demonstrates the big advantages (Radford et al., 2021; Jia et al., 2021; Xue et al., 2022; Zellers et al., 2021; 2022). Unlike images that have alt- texts, scale video datasets suitable for pre-training usually use subtitles as text sources (Miech et al., 2019; Xue et al., 2022). Subtitles are much noisier than alt-texts, according to (Miech et al., 2019), typical examples of incoherence include the content producer asking viewers to subscribe to their channel, talking about something unrelated to the video, or describing something before or after it happens. Bain et al. collect a video dataset WebVid (Bain et al., 2021) with textual description annotations. Their texts are well aligned with the video and avoid suffering from ASR errors. However, the vast majority of WebVid videos are sourced from a stock footage website, so scaling up is under limitation. The sudeo-sub data is more easily accessible the web and thus suitable for scaling up. In this paper, we investigate the unfavorable factors of video-subtitae data and explore how to mitigate the impact of these factors.
2

---

Published as a conference paper at ICLR 2023
Avg. of Rerallg 1,310
66.3
65.7
65.
NMI Soore
MSR-VIT
DiDeMo
Mean
64.5
6177
HD-VI AD-VILAuub
0.831
0.684
0.758
635
6300
62.8
0.317
0.621
625
WebVid
0,420
0.488
0.454
Postpredtrinnng Step Peresdage
100%
COCO
0.373
0.758
0.566
-WebVid-25M
-HD-VILA-IM -HD-VILA-10M
CC12M
0.445
0.673
0.559
Figure 1: The curve of finetuning results dur- ing post-pretraining The x-axis indicates the percentage of training steps. The y-axis indi- cates average value of @ecall 1, and 10. [Best viewed in color]
Table I: Normalized Mutual Informa- tion (NMI) score of language features ex- tracted on series of data and downstream tasks. Larger value indicates larger do- main gap.
Pre-trained Models for Video-Text Retrieval The great success of the CLIP has demonstrated its unprecedented power on varies downstream tasks, including vision understanding (Gu et al., 2021; Wang et al., 2021a; Rao et al., 2022), image-text generation (Patashnik al., 2021; Mokady al., 2021) and so on (Guzhov et al., 2022; Zhang et al., 2022). By contrastive learning on large-scale image-text pairs, CLIP learns enriched visual concepts for images. Recently, some works directly transfer CLIP to video-text retrieval without further pretraining (Luo et al., 2021; Fang et al., 2021; Gorti et al., 2022; Zhao al., 2022; Wang et al., 2022c). Their work takes the performance of video-text retrieval to pre-trained on video data (Xu et al., 2021b; Bain et al., 2021; Xue et al., 2022: Ge et al., 2022; Wang et al., 2022a). They transfer CLIP from views of feature aggregation (Luo et al., 2021; Zhao al., 2022; Fang et al., 2021; Gorti et al., 2022) or representation alignment (Fang et al., 2021; Gorti et al., 2022; Wang et al., 2022c). In parallel with these works, study post-pretraining with video data on top of CLIP in an effective way and our model can be combined with other approaches effectively.
video data (post-pretraining)
level, outperforming existing models
PRELIMINARY ANALYSIS
In this section, we first study the impact of the data scale for adapting image-text pre-training to video-text post-pretraining, and then investigate how the language domain gap affects the adaption.
3.1 POST-PRETRAINING WITH DIFFERENT DATA SCALES
To study the effectiveness of different data scales, we use the CLIP-ViT-B/32 model (Radford et al., 2021) the base image-text pre-trained model and adopt MeanPooling for video adaption like CLIP4Clip (Luo et al., 2021) by averaging multiple frame features as video feature. Two open- domain video-text datasets
used: WebVid-2.5M (Bain al., 2021) with 2.5 million pairs and HD-VILA-IOOE 1-00 (Xue et al., 2022) with 100M pairs. We also adopt a subset of HD-VILA-100M containing random 10% data (namely HDD-ILA-II 100MM) of steps on all settings, equivalent to one epoch on HD-VIL VIL- 1-00.......... We uniformly sample 12 frames from each video and apply the same hyper-parametest as described in Section 5 for all settings.
middle setting. We run the same number
During post-pretraining, we evaluate the pre-trained models by fine-tuning on MSR-VTT text-to- video retrieval task. Figure shows the performance trend. We observe an overfitting phenomenon that continuous post-pretraining leads to a performance drop. And the drop is more significant for smaller data (e.g.. Ved and HD-VILL-100T image-text pairs, further training on small data makes the model tend to overfit the new data while the implicit knowledge learned from the image-text pairs degrading. As consequence, the per- formance will drop, even worse than using CLIP directly. Thus we adopt HD-VILA-1O0M, due to its large scale and diverse category.
As CLIP is pre-trained on 400 million
3.2 LANGUAGE DOMAIN GAP WITH DOWNSTREAM DATA
is intuitive that pre-training on data with the same domain as downstream data can benefit down- stream tasks. For most video-text tasks like video-text retrieval, texts are descriptive sentences of videos (i.e., captions). While for auto-transcribed subtitles and they indicate very different relevance to visual information compared to descriptive texts. Meanwhile, auto-trant itto-transcribed subtitles suffer from irrelevance, misalignment, and ASR errors (Miech al., 2019). To better explore the language domain gap between pre-
which we will use for pre-training the texts are
3

---

Published as a conference paper at ICLR 2023
Video Proxies
Prame Patches
$
Proxies
Prmm:
Patches
Expns F
Eppur Aprnse bprie
Quecy
Omnisource Cross-modal Leamingg in Batch
Proxy- Proxy-guided tinnion
s/99
M//f
Text Encoder
Vision Encoder with Proxy-guided ttention
Subtitle: "[CLS] Josh Donaldson lats a foul ball to the first base side
Prame stl
Prame #112
Prome PY
Caption: "(CLS) man is bitting baschal!"
Frame:
Pos embed
Pos. umbeed
token
Token
Visuall
Video
Prory
Frame
A Atentio
mask
Posétive
Negativee
Contrastive
candidales
Figure 2: The framework of CLIP-ViL with text encoder and a vision encoder. Taken features V. S, C of videos, frames, subtitles, captions as input, series of Omnisource cross-modal learning variants are studied to explore better representation learning losses: (a) LV + FFecs (b) Lv+s + Lvesc; (c) Lv+>s + Lve + CF (d) Lv Crec. In the vision encoder, Video proxy tokens and the ViP-guided attention mechanism is proposed to transfer CLIP into the
video domain. [Best viewed in color]
training data and downstream data, measure the inconsistency by calculating the dissimilarity between their language features. For downstream language data, we choose two typical video-text retrieval datasets: MSR-VTT (Xu al., 2016) and DiDeMo (Anne Hendricks et al., 2017). For pre- training language, we select four types: video subtitles of HD-VILA- 100M (HD-VILAgus). video captions of Web 2.d- image captions of MS-COCO (Lin et al., 2014), and web-collected alt- texts of Conceptual Caption 12M (Changpinyo et al., 2021). In addition, we analyze auto-generated captions of HD-VILA- 100M OHD.V which will be introduced in Section
We use Transformer Encoder initialized from CLIP (Radford et al., 2021) to extract text features. To quantify the domain gap of languages between pre-training and downstream data, we first mix their text features and then K-means to get two clusters. Then calculate the Normalized Mutual Information (NMI) between cluster labels and ground-truth -ouun labels of pre-training or down- stream. A larger NMI value means that the types of features there is larger domain gap. For each comparison, we randomly sample 1000 texts from each type of data for 10 times and adopt the average of 10 results. We report the results in Table 1. Compar- ing the values of all pre-training data types, we find that the NMI score between HD-VILAssu and downstream data is much larger than others, especially for MSR-VTT downstream dataset. This indicates that direct training with subtitles may introduce inconsistency with downstream tasks.
easy to be distinguished, thus
APPROACH
In this section, will introduce the proposed CLIP-ViP video pre-training framework. To bridge language domain gaps between image and video datasets, we first introduce an in-domain auxil- iary data generation method. Then, we propose a novel Video Proxy mechanism enable the Vision Transformer (VIT) model for both image and video encoding. We further present an Om- nisource Cross-modal Learning (OCL) method which can jointly learn cross-modal representation from video-text and image-text pairs.
4.1 IN-DOMAIN AUXILIARY DATA GENERATION
Motivated by the analysis in Section 3, we introduce auxiliary captions into large-scale video-subtitle data to reduce the language domain gap between pre-training and downstream data. We adopt
4

---

Published as a conference paper at ICLR 2023
image captioning model for two reasons. 1) Most SOTA video captioning models are trained with video-text datasets (c.g., MSR-VTT, ActivityNet) which are also used for downstream tasks. We avoid data leakage perform pre-training agnostic to downstream data. 2) The performance of existing video captioning models lags far behind that of images. Thus, we choose a powerful image captioning model OFA-Caption (Wang et al., 2022b) generate one caption for the middle frame of each video in ID-VILA-I0OM We use the default setting of the OFA-Caption model. As a result, we generate 100M sentences with a max length of 16 words. This method can be applied any video data and we will release the generated captions to facilitate future research.
4.2 VIDEO PROXY MECHANISM
Since video is ordered sequence of frames, it is critical to learn the frame aggregation and tem- porality when transferring to the video domain. Meanwhile, to keep the high generality and ex- tendability of the Vision Transformer (ViT) backbone, we aim to find simple but effective way to transfer ViT to enable both image and video encoding with minimal modifications. Given a video containing T frames: {f1. f2.. we follow CLIP to divide each frame into N patches: positional embedding to each flattened
{.ft f?. f?.... f. E [1,T]]. Then we add 2D patches:
g(ff) = Linear(fT) + (n) + (ts,
(1)
where Linear(*) is linear layer, (n) and Pos,(t) is the learnable spatial and temporal posi- tional embedding, respectively. The whole video can be divided into x N patch tokens.
To model spatial information from multi- frames, one simple way is directly feeding all tokens into CLIP's vision encoder and conducting attention across all tokens. However, this method introduces significant conflicts with CLIP. As CLIP is pre-trained on image and text pairs, it has difficulty han- dling interactions of tokens between images/frames during training. We also verify it by experiments as Full Attention setting in Table 2. Instead, introduce Video Proxy token to act as a proxy that helps each local patch perceive video-level temporal information.
Before feeding into CLIP, we concatenate patch tokens with set of learnable parameters called video proxy tokens: P = (PI, (p..p...... PM J. where M is the number of video proxy tokens. Then P2....,PPd
all T x N + .M tokens will be fed into the ViT of CLIP. The output of the first video proxy token will be regarded as the video's 's representation. We also design a proxy-guided attention mechanism for the vanilla ViT. In the attention score calculation of each block, video proxy tokens attend to all tokens, while patch tokens only attend to tokens in the same frame plus video proxy tokens. This mechanism can be formulated as attention mask Mvip:
Mvip kq = { if E or E P or (q. k) in the same frame,
otherwise,
where q and A is the key and query in attention calculation. Patch tokens can obtain global informa- tion from video proxy tokens while reducing inconsistencies with the original CLIP's calculation. Our experiment in Section 5 demonstrates the superiority of this mechanism.
For the input type of the image/frame, we use linear interpolation to get middle temporal positional embedding, then treat the image/frame as a special single-frame video. This method enables joint training on both videos and images in the same batch, as our proxy-guided attention mechanism reduces the difference in calculations between video and image.
4.3 OMNISOURCE MODAL LEARNING
To learn rich lideo-language alignment from video-subtitle pairs and reduce the language domain gap with downstream data by corresponding auxiliary frame-caption pairs, we study joint Cross- Modal Learning on the omnisource input. Following most works of learning multimodal alignment on dual encoders (Radford et al., 2021; Xue et al., 2022; Li et al., 2021; Luo et al., 2020; Xu et al., 2021b; Luo et al., 2021), we use info-NCE loss to perform contrastive learning. There are two formats of visual source: video sequences and single frames, and two types of text source: subtitles and captions in our work. We denote them by V, F, S, and C respectively. We define source wise info-NCE loss by:
B
e
5/T
1 B
L =
log
Lizu =
log
(3)
B i=1
Σ
Σ
B
1//T
i=1
1=
5

---

Published as a conference paper at ICLR 2023
Model
R@1
R@5T
RØ 10
Menn f
MeanProl
43.4
70.9
81.1
65.1
Seq Transformer
44.6
71.2
81.8
65.9
Attention
42.8
2 Video Pruxy Tokens
45.8
71.3
81.7
66.3
Proxy
72.1
Video Proxy Tokens
45.7
72.7
81.7
66.7
Table 2: MSR- VTT text-to-video retrieval results of finetuning CLIP by different settings. Mcan
indicates the average value of Recall @ 1, and 10. All results are based on CLIP-ViT-B/32.
MSR-VTI Retrieval
DiDeMo Retrivall
Model
R@ 1
R@51 1
R@10
Meun
R 7
R
R 10
Mean 1
CLIP-MeanPool
43.4
70.9
81.1
65.1
40.6
67.5
77.2
61.8
CLIP-VPP
46.5
72.1
82.5
67.0
40.6
70.4
79.3
63.4
Lvos
47.7
72.1
82.4
67.4
44.6
73,9
81.9
66.8
Lvess + Leec
74.5
84.4
Lvos + Lvoc
49.6
74.2
84.0
69.3
48.5
76.6
83.6
69.5
Lvus + Lvac + Ceac
49.6
48.5
76.8
84.1
69.8
Lv- Ep
49.6
74.5
84.8
69.6
48.2
76.7
84.4
69.8
Table 3: Ablation study of different losses. We report eext-to-video results of models finetuned on MSR-VTT and DiDeMo. Mean t means the average of Recall 1, 5 and 10. All results are based on CLIP-ViT-B/2.. All post-pretraining steps are equivalent to one epoch HD-VILA-100M.
where and are the normalized embeddings of i-th visual feature in X E (V, F) and j-th text feature E {S, C} in a batch of size B. т is a learnable temperature. The overall alignment loss CxeY is the average of Lu and Lizv. For example, Lv represents info-NCE loss within video-subtitle pairs in a batch, which pulls aligned pairs together in embedding space while pushing apart misaligned pairs.
We study the reasonable variants of OCL: (a) Lv + CF Simple combination of two source- wise losses video-subtitle and frame-caption pairs; (b) Lv Lv As there is also tent correlation between videos and its middle-frame captions, we explore to add a loss on video- caption pairs to baseline loss Cv+ss; (c) Lv + Lv + LF Combination of (a) and (c); (d) Lv+s, + CF+.0: A video corresponds to both a subtitle and auxiliary caption. Compare to (c), the numbers negative pairs in Lezi can be expanded. The Co21 in LV++sO rewritten as:
Luz = 2B
1
Σ 1=1
B
(log
c 4//
+ log
E e9 6// + + esun/T)
(4)
ΣB= eai #5/T + e e///T
where E S and C E C. The Lezo in Lvesoo is equal to (c). We compare all variants with the baseline Lvess and report results Section
EXPERIMENT
5.1 EXPERIMENTAL DETAILS
Video-Text Pust-Pretraining To pre-train the proposed CLIP-ViP model, we uniformly sample 12 frames and resize all frames to x24x224 from video clips with average length of 13.4 seconds. The sampled frames can well cover the semantics conveyed from video clip. For text, we adopt the CLIP's tokenizer split sentence into word tokens with max length of 70. We AdamW optimizer (Loshchilov & Hutter, 2019), and empirically set an initial learning rate as 5e-6 and a fixed weight decay as 5e-2. For the learning rate schedule, we adopt cosine decay with warm- up strategy. We train our model with 32 NVIDIA Tesla V100 GPUs in batch size of 1024. The contrastive similarity calculated on gathered features from all GPUs. We set training steps to one epoch HD-VILA-11O 100M for all ablation studies and three epochs for the full setting.
Fine-tuning Training. To better adapt CLIP- ViP to downstream tasks, we reuse most hyper- parameters of post-pretraining in fine-tuning with some exceptions. 1) Batch size: fine-tune

---

Published as a conference paper at ICLR 2023
MSR-VTT Retrieval
DiDeMo Retrieval
Post-pretrain Data
R@ f
R@51 f
R0
Mean t
R@1
R@5
R@10
Mean
w/o Post-pretrain
46.5
72.1
82.5
67.0
40.6
70.4
79.3
63.4
HD-VIL .uub
47.7
72.1
82.4
67.4
44.6
73.9
81,9
HD-VILAcap
45.9
73.0
81.8
66,9
44,9
74.4
82.3
67.2
HD- ViLhhoo
49.6
74.5
84,8
69.6
48.2
76.7
84.4
69.8
ImageCaption
45.6
70.7
81.1
65.8
43.7
69.5
77.9
63.7
HD-VILA ImugeCaption
83.5
68.6
47.0
75.3
84.1
Table Ablation study of post-pretrain data. We report text-to-video results of models finetuned on ESR-VTTT and DiDeMo. Mean t indicates an average of Recall @ 1, 5 and 10. For all results, the model is designed with 4 video proxy tokens and pre-trained on steps are equivalent to one epoch on HD-VILA-100M.
All post-peetraining
our model with a batch size of 128 for all downstream tasks for a fair comparison. 2) Learning rate and weight decay: we empirically set them to le-6 and 0.2, respectively. 3) Number of epochs: due to the various scales of downstream datasets, we set epoch numbers to 5, 20, 10, and 20 for MSR- VTT, DiDeMo, LSMDC, and ActivityNet, respectively. 4) Frame number: for fair comparison, we set frame number to 12 except for ActivityNet Captions (set to 32) as its videos are much longer (180 seconds on average). Note that the seper-parameters of downstream training the same in all settings in the ablation study.
Downstream Datasets. To evaluate performances of video pre-training models, we conduct text- to-video retrieval experiments on four typical datasets. (a) MSR-VTT (Xu et al., 2016) contains 10K YouTube videos with 200K descriptions. We follow previous works (Yu al., 2018: Liu al., 2019) to train models on 9K videos, and report results on the 1K-A test set. (b) DiDeMo (Anne Hen- dricks et al., 2017) consists of 10K Flickr videos annotated with 40K sentences. We follow (Liu et al., 2019; Zhang et al., 2018) to evaluate paragraph-to-vieee tions of a video as one query. (c) LSMDC (Rohrbach et al., 2016) consists of 1 118,081 video clips sourced from 202 movies with caption corresponding each clip. Evaluation is conducted on a test set of 1,000 vidcos from movies disjoint from the train and validation sets. (d) ActivityNet Captions (Krishna et al., 20) contains 20K YouTube videos annotated with 100K sentences. We follow the paragraph-to-videe retrieval setting (Zhang et al., 2018; Liu et al., 2019) to train models on 10K videos and report results on the vall set with 4.9K videos.
retrieval and concatenate all descrip-
5.2 ABLATION STUDIES
Video Proxy Mechanism. For the vision encoder, we evaluate our proposed Video Proxy (ViP) mechanism with different numbers of proxies and compare it with different model structures (i.e. MeanPool, SeqTransformer, Full Attention) by fine-tuning the same pre-trained model on MSR- VTT retrieval task. MeanPool simply takes the average of frame features as the representation of the whole video. For SeqTransformer,, we follow the seq Transf type in CLIP4Clip (Luo et al., 2021) and the residual connection in their implementation. Full Attention setting takes all patch tokens as the input of the vision encoder and attention is conducted across all tokens. All models are initial- ized with CLIP-ViT-B/32. The results are shown in Table 2. Compared to the MeanPool baseline which completely disregards temporality, Seq Transformer improves the average Recall 1,5,10 by 0.8%. Full Attention type leads to a significant performance drop and we observe a worse initial status and slower convergence than other settings during the experiment. This consistent with our analysis in Section 4.2 that directly using CLIP for all patches' attention computation will de- crease the advantage of CLIP. In our method, different numbers of video proxy tokens all result in significant performance gain on R@I (c.g., 3.1% by 4 proxies), while only increasing negligible parameters: 3K compared to 86M of ViT backbone. Compared with other settings, our methods in all settings have the most improvement which indicates that our proposed video proxy mechanism can effectively leverage the image-text pre-trained model for video-text post- post-pretraining.
Omnisource Cross-modal Learning. To verify the effectiveness of the proposed Omnisource Cross-modal Learning (OCL) and compare its variants, we set a post-pretraining and fine-tuning pipeline and adopt the same hapee-paaammeters for all experiments. Lvess is the baseline contrastive
7

---

Published as a conference paper at ICLR 2023
MSR-VTT Retrieval
ActivityNet Captions Retrieval
Method
R@1
R@5
R@10
Mean
R@
R@51
R@10
Mean
ClipBERT (Lei 2021)
22.0
46.8
6.0
49.0
63.5
6.0
VLM (Xu et el., 2021a)
28.1
55.5
67.4
4.0
MMT (Gabour et 2020)
26.6
4.0
28.7
61.4
3.3
Support Set (Patrick
2021)
30.1
58.5
69.3
3.0
29.2
61.6
3.0
Frozen (Bain et al., 2021)
31.0
59.5
70.5
3.0
28.8
60.9
3.0
VideoCTII (Xu
202
30.9
55.4
66.8
HD-VILA (Xue ot ol., 2022)
35.6
65.3
78.0
3.0
28.5
57.4
4.0
Florence (Yaan al.,
37.6
63.8
All-in-Oue (Wang et al., 2022a)
37.9
68.1
77.1
22.4
53.7
67.7
5.0
BridgeFormer (Go al., 2022)
37.6
64.8
75.1
3.0
CLIP-VET-B/322
CLIP4Clip (Lai 2021)
44.5
71.4
81.6
2.0
40.5
72.4
2.0
CenterCIIIP (Zhao al., 2022)
44.2
71.6
82.1
2.0
43.9
74.6
85.8
2.0
XPool (Gorti ct, 2022)
62.2
Cidee (Fang al,, 2021)
45.6
72.6
81.7
2.0
CLiP2NideorBoogoln
2022)
47.2
73.0
2.0
CLIP2TV (Gao al., 2021)
46.1
72.5
82,9
2.0
44.1
75.2
2.0
DRL (Wang al., 2022c)
47.4
74.6
83.8
2.0
44.2
74.5
86.1
2.0
CAMol (Cheng 2021)
47.3
74.2
84.5
2.0
51.0
77.7
Ours
50.1
74,8
84.6
1.0
51.1
78.4
88.3
1.0
Ours
55.9
91.3
CLIP-WT-B/I6:
CenterCLIP (Zhao et 2022)
48.4
73.8
82.0
2.0
46.2
77.0
87.6
2.0
CLIPZTY (Gan al. 2021)
49.3
74.7
83.6
2.0
(Wang al., 2022c)
50.2
76.5
84.7
1.0
46.2
77.3
88.2
2.0
DRLf(Wang et al,, 2022c)
53.3
80.3
87.6
1.0
Ours
57.7
80.5
88.2
1.0
61.4
85.7
92.6
1.0
Table 5: Comparison with SOTA models in MSR-VTT (Xu et al., 2016) and ActivityNet (Krishna et al., 2017a) text-to-video retrieval tasks. and respectively denotes that the method uses DSL (Cheng et al., 2021) and QB-Norm (Bogolin et al., 2022) as post-processing operations.
loss on video-subtitle pairs. After introducing auxiliary captions, we study four variants of OCL Loss: (a) Lv (b) Lv Ly (c) Lv Lv F (d) L srrescttlee F as explained Section 4.3, We pre-train models with each loss function for only one epoch due to the costly training, then finetune two video-text retrieval datasets: MSR-VTT and DiDeMo. We compare the results with CLIP-MeanPool and CLIP using the proposed Video Proxy mechanism without post-pretraiie prettraren (i.e., CLIP-ViP! The results listed in Table 3. On MSR-VTT dataset, we find that Lves brings very little improvement: 0.4% on average of Recall @1 1, 5, 10. This is due to the large domain gap between MSR-VTT and post-pretraining data. Combined with auxiliary captions, four variants of OCL loss all bring significant improvements: over 3% on Recall 1 and over 2.3% on average of Recall @ 10. On DiDeMo dataset, based on the improvement brought by Lv+ss, OCL further improve the results by large margin: 8% on average of Recall @ 1, 5, 10. Finally, LV+ CFHC performs best which is applied our final setting.
Auxiliary Data. In this part, ablate the contribution of large-scale noisy data and auxiliary data. For uni-source, we use video-subtitle pairs and video-caption data for post-pretraining by vanilla contrastive loss. For data combination, we apply OCL under Lv CF setting to post-pretrain on the combined data. From Table 4, Omnisource post-pretraining results are much better than two uni-source results. On MSR-VTT, both uni-source post-pretraining show limited improvement: 67.4% and 66.9% compared with 67.0%. While the Omnisource pestraentninnng brings significant improvement of 2.6%. On DiDeMo, the benefit of data combination is also obvious, with nearly double the improvements brought by uni-source. These results show that the auxiliary data together with our designed joint learning method can effectively adapt the image-text pre-trained model video-text post-pretraining
As the generation of auxiliary captions is based on OFA-Caption (Wang et al., 2022b), a powerful image-text pre-trained model, we also explore only including existing data in post-pretraining. We choose image-text pairs of several widely-adopter datasets: MS-COCO, Visual Genome (VG) (Kr- ishna al., 2017b), Flickr-30K (Young et al., 2014), SBU (Ordonez al., 201 1), CC3M (Sharma et al., 2018) and CC12M as our auxiliary data (namely ImageCaption). To ablate the contribution of these data, add experiments of post-pretraining ImageCaption alone and HD- HDD-IILA- VILAA-10MM
8

---

Published as a conference paper at ICLR 2023
DiDeMo Retrieval
LSMDC Retrieval
Method
R@1
R@5
R@
Mean
R@
R@51
R@10
Mean
MMT (Gabeur 2020)
12.9
29.9
40.1
19.3
ClipbERT (Lei et al., 2021)
20.4
48.0
60.8
6.0
Frozen (Bain al., 2021)
31.0
30.8
40.3
20.0
HD-VILA (Xue et al.. 2022)
28.8
57.4
69.1
4,0
17.4
34.1
44.1
15.0
All-in- One (Wang el al., 2022g)
32.7
61.4
73.5
3.0
FridgeFormer (Ge
2022)
37.0
62.2
73.9
3,0
17.9
35.4
44.5
15.0
CLIP-WT-B/32:
CLIP4Clip (Loo et al., 2021)
43.4
70.2
80.6
2.0
21.6
41.8
49.8
11.0
CenterCLIP (Zhao al., 2022)
39.8
49.8
XPool (Gorti al., 2022)
22.7
42.6
51,2
10.0
CLIPZTY (Gao al., 2021)
45.5
69.7
80.6
2.0
DRL (Wang al., 2022c)
47.9
73.8
82.7
2.0
24.9
45.7
55.3
7.0
CAMoE (Cheng et al, 2021)
43.8
71.4
25.9
46.1
53.7
Ours
48.6
84.4
45.3
54.4
8.0
Ours
53.8
79.6
86.5
1.0
26.0
46.4
54.9
8.0
CLIP-VTT-/16:
CLIP4Clip by (Zhaoo et al., 2022)
24.1
45.0
55.1
8.0
CenterCLIP (Zhao al., 2022)
24.2
46.2
55.9
8.0
DRI. (Wang al., 2022c)
49.0
76.5
84.5
2.0
26.5
47.6
56.8
7.0
Ours
55.3
82.0
89.3
1.0
30.7
51.4
60.6
5.0
Table 6: Comparison with SOTA models in DiDeMo (Anne Hendricks et al., 2017) and LSMDC (Rohrbach et al., 2016) text-to-video retrieval tasks. * denotes using post-processing DSL (Cheng et al., 2021).
combined with ImageCaption. From Table 4, post-pretraining on ImageCaption alone results in performance degradation on MSR-VTT and marginal improvement on DiDeMo. In contrast, Im- ageCaption yields significant performance gains on both datasets when used auxiliary data for HD-VILA- A-00M. This further illustrates the importance the combination of large-scale noisy data and auxiliary data.
5.3 COMPARISON TO STATE-OF THE-ART MODELS
We compare model under full setting (in three epochs) with state-of-the-art works on the text- to-video retrieval task. The results of fine-tuning on four datasets (i.e., MSR-VTT, DiDeMo, Ac- tivityNet Captions, LSMDC) CLIP-based works. We only add results with DSL (Cheng et al., 2021) to make fair comparison with some methods using post-processing operations e.g., DSL (Cheng et al., 2021) or QB-Norm (Bogolin et al., 2022). Our model achieves the best results on all datasets in both CLIP-ViT-BA. and CLI--V.T-//16. Note that some existing methods are also applicable on top of our models as our modification to the CLIP model is minimal. Note that even without post post-processing (c.g., DSL), our results still surpass methods using post-processing operations on most datasets. Besides, adding DSL will greatly improve the performance of our model since our model has good bidirectional vision-lant lannu-la correspondence. The good results on the ActivityNet Captions dataset also indicate that our model can generalize well long videos. Overall, the improvements different datasets demonstrate the superiority of the video- lideo-language representation learned by our CLIP-Vii ViP model.
shown in Table 5 and 6, respectively. We clarify the backbone for
CONCLUSION
In this paper, we study further pre-training gpost-pretraining image-text models like CLIP on large- scale video data. We first conduct preliminary analysis reveal the factors hindering video post- pretraining. Motivated by findings, we propose CLIP-ViP which includes Omnisource Cross- modal Learning method and Video Proxy mechanism. The Video Proxy mechanism can better model videos containing temporal information while reducing conflicts with the pre-trained CLIP model. The Omnisource Cross-man modal Learning alleviates the problem caused by the domain gap between vidco-subtitle and downstream data. Extensive results show that approach improves the performance of CLIP on video-text retrieval by a large margin and also achieves new state-of- the-art results four widely-used ladeo-language benchmarks.

---

Published as a conference paper at ICLR 2023
REFERENCES
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Damrlll, and Bryan Russell. Localizing moments in video with natural language. In ICCV, pp. 5803-5812, 2017. 4, 7.9
Max Bain, Arsha Nagrani, Gäl Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 1, 8,
Simion-Vlaa Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation. In CVPR, pp. 5194-5205, 2022. 8,
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pp. 3558-3568, 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pp. 104-120, 2020.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. arXiv preprint arXi:22109.04290, 2021. 8,
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106. 1099, 2021. 3,
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Vi- olet: End-to-end languaan transformers with masked visual-token modeling. arXiv preprint arXiv:2111.1262 2021. 2
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. ECCV, pp. 214-229, 2020. 8, 9
Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. Clip2tv: An empirical study on transformer-based methods for video-text retrieval. arXiv preprint arXiv:2111.056/6 2021. 8,9
Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridging video- text retrieval with multiple choice questions. In CVPR, pp. 16167-16176, 2022. 3, 8.9
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text-video retrieval. In CVPR, pp. 5006-5015, 2022. 8,
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Hcuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pp. 5842-5850, 2017. 13
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2021. 1,3
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP, pp. 976-980 IEEE, 2022. 1,3
Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi- directional image and text generation. In ACM MM, pp. 1138 1147, 2021a. 2
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi- modal transformers, arXiv preprint arXiv:2004.00844 2020.
Zhicheng Huang. Zhaoyang Zeng, Yupan Huang. Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to mon-to--nd pre-training for languae representation learning. In CVPR, pp. 12985, 2021b. 2
10

---

Published as a conference paper at ICLR 2023
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsure Sung, Zhen Li, and Tom Duerig. Scaling up visual and assio-language representation learning with noisy text supervision. In ICML, pp. 4904-4916, PMLR, 2021. 1.2
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and -nd--anndd-gggee transformer without convo- lution or region supervision. In ICML, pp. 3583-55944 PMLR, 2021.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Nicbles. Dense-captioning events in videos. In ICCV, pp. 706-715, 2017a. 8,8
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 1 123(1):32-75 2017b. 8
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In /CCV, 2011. 13
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and- lnd-lang learning via sparse sampling. In CVPR, pp. 7331-7341, 2021. 2, 8,
Gen Li, Nan Duan, Yuejian Fang. Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, pp. 11336-1134, 2020a. 2
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021. 2,
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pp. 121-137, 2020b. 2
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, pp- 740-755. Springer, 2014.
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In BMVC, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal under- standing and generation. arXiv preprint arXiv:2002.06355 2020. 5
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2 104.08860, 2021. 1,3,5, 7,
Antoine Miech, Dimitri Zhukov, Jean- han--Batitte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howto100ml Learning text-video embedding by watching hundred million narrated video clips. In ICCV, pp. 2630-2640, 2019. 2,
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint 1tXiv:2//1.097:: 2021. 1,
Vicente Ordonez, Girish Kulkami,, and Tamara Berg. Im2text: Describing images using million captioned photographs. NeurIPS, 24, 2011.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In ICCV, pp. 2085-2094, 2021. 1,3
Mandela Patrick. Po- Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F. Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2021, 8
11

---

Published as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763. PMLR, 2021. 1, 2, 3, 4,5
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR. pp. 18082-18091, 2022. 1.3
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Joseph Pal, H. Larochelle, Aaron C. Courville, and Bernt Schiele. Movie description. ICV, pp. 94-120, 2016. 7,9
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018. 8
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, pp. 7464-7473, 2019. 2
Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre- pre-training with multimodal temporal contrastive learning. In NeurIPS, 2022. 2
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre- training. arXiv preprint MrxXix:2203.07000 2022a. 3, 8,9
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint
2021a. 1,
Peng Wang. An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple
sequence-to-seqquecee learning framework. arXiv preprint arXiv:2202. 03052, 2022b. 5, 8
Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and ShnnSh Hua. Disentangled representa- tion learning for text-video retrieval. arXiv preprint arXiv:2203.07111 2022c. 3, 8,
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang. Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In CVPR, pp- 14733-14743, 2022d. 14
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2021b. 1
Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichten- hofer, Florian Metze, and Luke Zettlemoyer. Vlm: Task-agnostic video-language model pre- training for video understanding. arXiv preprint arXiv: 0999999 2021a. 8
Hu Xu, Gargi Ghosh, Yo- Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, pp. 6787-6800, 2021b. 2, 3,
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pp. 5288-5296, 2016. 4, 7,8
Hongwei Xue, Yupan Huang. Bei Liu. Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing modallty: Visual parsing with self-attention for In NewrlPS, 2021. 2
pre-training.
Hongwei Xue, Tiankai Hang, Yanhong Zeng. Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, pp. 5036-5045, 2022. 2, 3, 8, 9, 14
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. ACL, 2:67- 78, 2014. 8
12

---

Published as a conference paper at ICLR 2023
Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, pp. 4714474777 2018. 7
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprini arXiv:2111.11442 2021. 8
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In NeurIPS, 2021. 1,2
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. CVPR. pp. 16375-16387, 2022. 1,
Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In ECCV, pp. 374-390, 2018.
Renrui Zhang. Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In CVPR, pp. 8552 8562, 2022. 3,3
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Centerclip: Token clustering for efficient text-video retrieval. arXiv preprint arXiv:2205.0088 2022. 1, 3, 8,
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In CVPR, pp- 8746-8755, 2020. 2
A MORE DOWNSTREAM TASKS
We add various suitable downstream tasks verify that our model can be generalizable other tasks. Specifically, we add four zero-shot evaluations on video action recognition and video- language multi-choice tasks:
HMDB51 Action Recognition HMDB51 (Kuehne al., 2011) is video dataset containing alistic videos from various sources, including movies and web videos. There are 51 action cate- gories (such as "jump". "kiss" and "laugh"). We evaluate our model by zero-shot classification on HMDB51. To adapt to our model, we formulate video classification as video-text alignment, using text prompts "a person [label]". The prediction made on top-1 similarity between query video and prompted texts. We directly test our model without finetuning to report zero-shot accuracy results.
SomethingSomethingg22 (SSv2) Action Recognition. SomethingSomethinggW (Goyal et al., 2017) dataset is large collection of labeled video clips that show humans performing pre-defined basic actions with everyday objects. It requires models to develop fine-grained understanding of basic actions that occur the physical world. To adapt our model, we design multi-choice task on SSv2. The model has to choose the ground truth annotation among all categories. We directly test our model without finetuning report zero-shot accuracy results.
MSR-VTT Multi-Choice (MC). MSR-VTT MC is benchmark build MSR-VTT video dataset. Given a video query and five descriptive sentences, MSR-VTT MC requires the model choose single best answer in the candidates. We directly our model without finetuning to report zero-shot accuracy results.
LSMDC Multi-Choice (MC). LSMDC MC is a benchmark build on movie data. Similar with MSR-VTT MC, it requires the model to choose single best answer from 5 candidates. We report zero-shot accuracy on 10000 test videos.
From the results in Table 7. performance drop on most tasks: HMDB51 SSv2 and MSR-VTV MC. This phenomenon is con- sistency with assumption subtitles usually have unique form. By final setting, CLIP-ViP
find that post pretraining CLIP subtitle alone will lead to
13

---

Published as a conference paper at ICLR 2023
Model
HMDB51
SSv2
MSR-VTT MC
LSMDC MC
CLIP-McanPool
40.3
25.8
89.9
64.3
CLIP-ViP w/o post-pretraining
41.1
26.1
89.4
63.5
CLIP-ViP MiP w/ video-subtitle
36.6
28.0
86.6
67.0
CLIP-ViP full
44.8
33.0
90.3
67.6
Table 7: Evaluation on more downstream tasks. We report zero-shot accuracy (%) on HMDB51, SSv2, MSRVTT-MC, and LSMDC MC. All models are base size with patch size of 32. Details of post-pretraining are the same as in Section 5.2.
achieves better results than CLIP on all evaluations, especially on SSv2 which requires more under- standing temporality (Wang et al., 2022d). These results demonstrate that CLIP- ViP outperforms CLIP on various video benchmarks and further show the generalization of our model CLIP-ViP.
B EFFICIENCY OF POST-PRETRAINING
In light of the good image representation of CLIP, our post-pretraining significantly reduces the training cost, Compared with HID-VILA (Xue al., 2022) which trains video-language representa- tion from scratch, CLIP-ViL ViP outperforms HD-VILA by large margin with about 1/30 training time. To make more fair comparison with training from scratch. We conduct experiment of training from scratch with the same setting and training time (one epoch). Then we finetune on MSR-VTT Retrieval to evaluate the pre- pre-trained model.
Model
MSR- R@I
MSR-VTT R@S
MSR-VTT R 10
CLIP-ViP (from scratch)
28.5
55.2
68.6
CLIP-ViP
49.6
74.5
84.8
Table 8: Comparison with training CLIP-ViP from scratch. We report finetuning results on MSR-
VTT Retrieval. All models are base size with patch size of 32. Details of post-pretraining are the same as in Section 5.2.
From the results in Table 8, we can see that there a huge performance gap. Without basing on CLIP, CLIP-VIP can not leverage the rich knowledge entailed in CLIP, thus leading much lower training efficiency.
14


Reviews:
[{"review_id": "y3lf8eYmj76", "reviewer": "Reviewer_39Ux", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attributes the weakness of post-pretraining of text-video models (based on the text-image model) to the domain gap between video subtitles and image captions. The authors develop several methods to improve the effectiveness of post-pretraining: data augmentations via video captions, video proxy tokens, and omnisource cross-modal learning.\n", "summary_of_the_review": "The paper does an excellent work in the post-pretraining of text-video models. I hope the authors can resolve my concern about the domain gap assumption.\n", "strength_and_weaknesses": "Strength:\nThe paper is well organized.\nThe performance is very impressive.\n\n\nWeaknesses:\n\n1. One primary concern: is the domain gap between language sources really the underlying problem? Do the authors really solve this problem in this work?\n\n1.1) In Table 1, we can see $HD-VILA_{cap}$ has a far smaller NMI than $HD-VILA_{sub}$. If the domain gap is the real problem and $HD-VILA_{cap}$ has the same amount of data as $HD-VILA_{sub}$, in Table 4, $HD-VILA_{cap}$ should have much better performance than $HD-VILA_{sub}$, but we can this is not the truth. Indeed, $HD-VILA_{cap}$ has worse performance than $HD-VILA_{sub}$, and $HD-VILA_{cap}$ is even worse than w/o post-pretrain. This violates the domain gap assumption of the authors.\n\n1.2) In Table 1, CC12M has a smaller NMI than $HD-VILA_{sub}$. In Table 4, we can see only ImageCaption pre-training does not help fine-tuning. There are two possibilities: a) the domain gap assumption does not hold; b) we should also consider the domain gap of visual signals. It is also weird why $HD-VILA_{sub}$ + ImageCaption get such good performance as the two work ordinarily without each other.\n\n1.3) Following 1.2, could the authors provide analyses of domain gaps of visual signals across different datasets?\n\n1.4) I think there is a publication that can explain the improvement in performance when using $HD-VILA_{sub}$ + $HD-VILA_{cap}$ [1]. I purely consider $HD-VILA_{cap}$ as a bootstrapping of captions or data augmentations. Hope the authors can turn my mind around. Besides, I think one way to verify this idea is by adding HowTo100M + HD_VILA result in Table 4. If this does not work, I may lean toward the domain gap assumption.\n\n2. What is the difference between video proxy tokens and video prompts? I think they are just the same thing with various names. \n\n\n[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. https://arxiv.org/abs/2201.12086\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "x5-BLn3ekv", "reviewer": "Reviewer_XjSo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for adapting image-text pre-trained models to video-language alignment tasks (specifically text-to-video retrieval). It argues there is a domain gap between text used in video-language pre-training versus downstream tasks, and uses an image captioning model to caption middle frames of the video and use them alongside with the subtitle text in training. It also adds additional global video tokens to the visual encoder and uses an attention mask to mask attention from tokens of each frame to those of other frames. Experiments on text-to-video retrieval show that the proposed approach outperforms competing methods. ", "summary_of_the_review": "Overall, the proposed approach is sound and shows reasonable performance gain in the experiments. However, it is mainly evaluated on a single task which makes it questionable to what extent the method is generalizable to other tasks.  ", "strength_and_weaknesses": "**Strengths**:\n\n* The paper is well-written and the proposed approach is described clearly. \n\n* The authors motivate the problem well with the experiments on the impact of data scale and language domain gap.  \n\n* CLIP-ViP outperforms baselines on various datasets such as MSR-VTT, DiDeMo and ActivityNet. \n\n**Weaknesses**:\n\n* The authors only evaluate their approach on the text-to-video retrieval task. It is unclear whether the method also provide benefits in other tasks such as video action recognition. Given that we can formulate video classification as video-text alignment (using text prompts), does the method perform well on this task as well?\n\n* The authors argue that there is a discrepancy between language data in pre-training and downstream tasks, and propose their captioning method accordingly to bridge this gap. As the downstream task is limited to text-to-video retrieval, it is questionable how generalizable the captioning approach is. Providing results on other downstream tasks can clarify this. \n\n* Using the additional captioning model adds extra compute to the process that needs to be discussed. \n \n* Regarding the plots in Figure 1, does this overfitting hold for different values of learning rate and optimization schedule? \n\n* There are a few errors in the manuscript, e.g. section 4.3. \"we definite\" -> \"we define\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OWPhVrZc2e", "reviewer": "Reviewer_P4Rx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new framework called CLIP-VIP to transfer the image-text model to video-language alignment. It has been found that the scale of dataset is crucial for post-pretraining. Therefore, the authors apply HD-VILA-100M due its large scale and diverse category. The other factor is the language domain gap measured by the NMI score in Table 1. Therefore, CLIP-VIP introduces an additional captioning model (OFA-Caption) to augment text data as another source. To enable cross-model post-pretraining based on video-text pairs, the authors invent video proxy tokens to summarize the content of the whole video temporal-spatially. The model is trained by the info-NCE loss in a self-supervised way.\n\n", "summary_of_the_review": "Based on all the comments above, I am leaning toward weak rejection. I will consider upgrading the score after a strong rebuttal from the authors.", "strength_and_weaknesses": "Pros:\n+ The proposed CLIP-VIP model successfully extends a pre-trained CLIP model to the video-text tasks with only limited revisions, which marginally increases negligible parameters and computations.\n\n+ This paper has systematically explored the crucial factors for model transfer between image-text and video-text tasks, which may shed light on the following works.\n\n+ It has achieved very competitive results on video-retrieval tasks on many benchmarks. Moreover, there are solid ablation analyses for different components as a thorough evaluation.\n\nCons:\n- This paper is not written and organized well. Without extra reading efforts, I cannot easily understand the motivation and the proposed method. Some of the sentences are pretty long and unfriendly to read. The introduction section focuses too much on algorithms/methods instead of its motivation. As a result, I am still confused about the major contributions of this paper.\n\n- The major technical contribution of this paper seems to be proxy video tokens which are simple and trivial. Moreover, how to assign the number of proxy video tokens needs some manual effort, which degenerates its value in practice.\n\n- The post-training is costly, considering the 32 GPUs in use. So, I have not seen the extraordinary benefits of model transfer in efficiency. The authors are suggested to compare the cost of training from scratch and post-training to highlight the advantages of model transfer.\n\nQuestions/Other Concerns:\n\n1. How to assign the M (number of proxy video tokens) in practice? What will be if M is small or large?\n\n2. How many GPU hours are needed for post-training? Is it more efficient than other methods?\n\n3. What is the meaning of single frames (F) in page 5? What is the difference between F and video sequences (V)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LqJZsYP8BS", "reviewer": "Reviewer_qsPb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an adaptation of the ViT based CLIP model for video-text alignment that learns the spatio-temporal structure of the videos during the alignment. The authors start by analysis the problems with current approaches that straight forwardly apply CLIP on video for video-text alignment. They find that data scale and domain gap are big factors in the performance of models. Based on these findings, the authors design a video proxy to allow the model to understand spatio-temporal aspects of video while also combined captioning and subtitle language information from videos for alignment. In the experiments, the authors show multiple ablations to validate the different components of their method and also outperform that state of the art in many datasets.", "summary_of_the_review": "All-in-all, I like this paper and I think it should be accepted in the current form. There is analysis of problems from previous methods while tackling this problem. There is novelty in the proposed method and objective strategies, and the method outperforms previous baselines.", "strength_and_weaknesses": "Strengths:\n+ New adaptation of ViT for video-text alignment\n+ Captioning strategy for full videos using middle frame.\n+ Info-NCE adaptation to include video subtitles and video caption.\n+ Outperforms baselines and informative ablations\n\n\nQuestions/comments:\n- How are negatives chosen?\nThere are two different text inputs being contrasted, subtitles and captions. I may have missed it from the text, but I couldn't find any details on the strategies used for getting the negatives for Info-NCE. Could the authors provide these details?\n\n\n- Using DSL posprocessing on all baselines?\nThe best reported results in this paper are after using the DSL posprocessing. The author mention that they use this for fairness against other baselines. However, I am not sure if this was used in all the baselines. Can the authors comment on whether this makes sense doing or not?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8EtXRuT67YU", "reviewer": "Reviewer_XvUh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper starts with two questions that try to answer what and how to apply CLIP to video-text tasks. They build a model based on the observations and propose new cross-modal learning methods for video-text retrieval. Their method was tested and showed promising results on those datasets. Adapting the image-caption pretarined model to video is a challenge. The findings in this paper can serve as a good baseline for the bridge of image-caption pretrained models and vision-text tasks.", "summary_of_the_review": "The proposed model achieved good performance on test datasets. The additional computation and proxy-guided attention need to be discussed more.", "strength_and_weaknesses": "- As for Further pretraining on small data, some details are missing; for example, did the authors try to freeze the backbone and optimize the head only? \n\n- The adopted OFA was powerful and could generate diverse and well captions. However, such a strong captioning model also brings another problem, what if we do not use OFA, but use previous models such as BUTD, SCST, etc. Then what are the effects of image captions on the final performance?\n\n- The proxy-guided attention looks new and interesting, and the results shown in Table 2 look promising. Showing some visualized attention samples will help the understanding of what the proxy learned.\n\n- The cross-modal learning part is straightforward and combines several contrastive learning tasks. The novelty of this part is somehow limited, especially considering that in table 3, some losses do not bring a big performance. They simply combine some possible losses together and lack strong motivation.\n\n- Given that they have already adopted OFA as the caption generation module and they have already got a large amount of data, then what if they train the model without the initial weight from CLIP? I think such a comparison can also show how much benefit their model leverages from the CLIP model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster