COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Authors: Sihan Chen
Xingjian He
Handong Li
Xiaojie Jin
Jiashi Feng
Jing Liu
Venue: ICLR 2024 poster (2024)
Keywords: Vision-Language Foundation Model
Video-Language Pretraining
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Due to the limited scale and quality of video-text training corpus, most  vision-language  foundation  models employ  image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations. To address this issue, we propose COSA, a COncatenated SAmple pretrained vision-language foundation model. COSA can jointly model visual contents and event-level temporal cues using only image-text corpora.  We achieve this by sequentially concatenating multiple image-text pairs as inputs for pretraining. This transformation effectively converts existing image-text corpora into a pseudo  video-paragraph corpus, enabling richer scene transformations and explicit event-description correspondence. Extensive experiments demonstrate that COSA consistently improves performance across a broad range of semantic vision-language downstream tasks, including paragraph-to-video retrieval, text-to-video/image retrieval, video/image captioning and video QA. Notably, COSA achieves state-of-the-art results on various competitive benchmarks. Code and model are released at https://github.com/TXH-mercury/COSA.

Published as a conference paper at ICLR 2024
COSA: CONCATENATED SAMPLE PRETRAINED VISION- LANGUAGE FOUNDATION MODEL
Sihan Chen 12- Xingjian He Handong Li Xiaojie Jin 3
Jiashi Feng" Jing Liu'
'School of Artificial Intelligence, University of Chinese Academy of Sciences
2 Institute of Automation, Chinese Academy of Science 'Bytedance Inc.
(s dsihhan chen, xingjian, he, jliu) @nlpr. ia La.ac. 1 ihandcng2023 aa.aaac.com en, (jinxiao jie, jshfeng) @bytedance.com
ABSTRACT
Due to the limited scale and quality of video-text training corpus, most vision- language foundation models employ image-text datasets for pretraining and pri- marily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations. To address this issue, we propose COSA, a COncatenated SAmple pretrained vision-language foundation model. COSA can jointly model visual contents and event-level temporal cues using only image-text corpora. We achieve this by sequentially concatenating multiple image-text pairs as inputs for pretraining. This transformation effec- tively converts existing image-text corpora into a pseudo video-paragraph cor- pus, enabling richer scene transformations and explicit event-description cor- respondence. Extensive experiments demonstrate that COSA consistently im- proves performance across a broad range of semantic vision-language down- stream tasks, including paragraph-to-video retrieval, text-to-video/mmage retrieval, video/image captioning and video QA. Notably, COSA achieves statc-of-the- art results on various competitive benchmarks. Code and model are released at wlhttps:///ithhb. com/ TXH- mercury/COSA
INTRODUCTION
Image-text and video-text pretraining models have garnered increasing attention in recent years due to their strong capabilities in bridging the domains of vision and language. These models contribute significantly to various vision- vision-language tasks such cross-modality retrieval, vision captioning, and visual question answering (VQA). In the early stages, these models were independently trained using distinct modalityy corpora, network architectures, and training objectives. However, considering that a video can be perceived as a sequence of multiple images (Lei et al., (202)), and an image can be seen frozen video (Bain al. 2021). most researchers lean towards training unified foundation models using corpora from both domains, employing sparsely sampled frames as the video representation.
Nevertheless, the majority of large-scale vision-language foundation models are usually trained solely on image-text corpora, disregarding the joint modeling of images and videos. This limitation stems from the restricted scale of video-text corpora, which is associated with the costly processes of video uploading. storage, and downloading: currently, open-sourced web-crawled video-text corpora, such as WebVid OM (Bain et al. 2021), are still two orders of magnitude smaller than its image-texi counterpart (LAION-2B (Schuhmann et al., 202T)). which implies the dominance of image-text data in joint modeling. On the other hand, image-text pretraining can also bring benefits to video-related tasks. In fact, some large-scale image-text foundation models (Radford et al. 2021 Wang et al. 2022bb have achieved competitive even superior performance on video tasks compared to dedicated video-language pretraining models.
Sihan Chen did this work when interning at Bytedance Inc. Corresponding authors

---

Published as a conference paper at ICLR 2024
Single Sample Training (SST)
Amage Text Corpus
Short Farm Vidne feut Coruos
Tasag meald dkgge
aerea sour chil maae AEStOA
Concafenate Sample Training (COSA)
Image- Tast Corpus
Pacagroph
of sulivip
United
Vemage resut for
"eugee resilt Bar dlgg of encres ceriic Fllecing Dication
service.
Bosing Nocatios Antoriaet of cuitaay posk car discced
Figure I: Visualizations of the traditional image-text and video-text model pretraining pipeline and proposed unified foundation model COSA, which transforms the image-text corpus into synthetic video-text corpus through online sample concatenation.
Despite the fact that image-text foundation models have acquired broad range of semantic correla- tions between visual concepts and language, they totally overlook temporal contexts within videos, which are crucial for video-language reasoning. This raises the question that whether can utilize large-scale image-text corpora to train a unified foundation model capable of capturing both static and temporal information during the pretraining phase.
Towards this goal, we propose a simple yet effective approach: training a foundation model with concatenated samples. As illustrated Figure [1, instead of modeling independently modeling cross- modaltty samples via traditional single sample training (SST), COSA models the semantic and event-level temporal information among multiple samples by conducting online sample concatenation to transform image-text corpora into -idro-paragraph corpora, and thus further enhance the learning of both image-language and video-language representations. Specifically, at each training step, for every image-text training sample, we randomly select some other samples from current batch and concatenate them (both images and texts) together. Ablation studies have demonstrated that model trained with COSA evidently outperform model trained with SST on 12 benchmarks including both video-text and image-text tasks. We attribute it to two main reasons: 1) Concatenated samples build connections between visual concepts and words within sample group, which is more difficult than build them within single sample, due to that both negative unmatched visual concepts and textual words become more, and thus helpful for learn more discriminative representations. comparison to current vidco-text corpora (Bain et al.] [202T), the on-the-fly concatenated corpus offers richer scene transformations, reduced vision redundancy due to sampling randomness, and longer temporally fine- tne-grained captions that describe every frame in sequential order, which encourages models to learn better event-level temporal understanding capabilities and thus further improve performance on video-text tasks.
In
Enhanced by COSA, we deliver unified vision-language foundation model. possesses a simple architecture, takes concatenated samples as input during pretraining with training objectives modifid to build connections among multiple cross-modalit samples. Extensive experiments have shown that COSA model is capable of handling both discriminative and generative tasks such cross- coos-myd retrieval, captioning, question answering, and achieves new state-of-the-art results across a wide range of benchmarks under different model scale settings.
2 RELATED WORK
Temporal Learning in Video-Language Pretraining (VidLP). Temporal information is the essence that distinguishes videos from images. Early methods aimed to enhance temporal learning in VidL. by either employing models pretrained on action classification tasks as vision backbones (Bain et al.) 2021 Li et al., 2022a Fu et al., 2021 2022) or designing specific training objectives such as frame order modeling (FOM) (Li al. 2020 Zellers et al. 202 1) or masked video modeling (MVM) (Fu ct al. 2021 2022). We divide temporal leamnnn into two aspects, namely event-level and aevion-level temporal modeling, in which the former emphasize on the differences and relations across multiple events/scenes, while the latter concentrates the temporal relations of consecutive frames in single event/scene.
2

---

Published as a conference paper at ICLR 2024
CIC
[DIT
COM
GMLM
Vision encoder
Text encoder
Vision encoder
Text encoder
result the secret service Filming location the snine are mall and clouds this location. sult subway train, Car countnn dollecs.
Amage dage the service Filiin location the skite are realy this blse and mhe clouds this aubwayy train, airessed in sul counting dollars.
Figure 2:
Model architecture and training objectives of COSA. Circles and squares in the figure represent global and patch features, respectively.
However, Singularity (Lei et al. 2022) reveals that temporal modeling in pretraining may not be necessary, as sampling a single frame can yield comparable performance to utilizing multiple frames. They attribute this the static appearance bias in video-language downstream benchmarks. We hypothesize that another contributing factor lies in the limitation of current video-text corpus (WebVid-2.5MM (Bain et al. 202 for both action-level and event-level temporal learning. In terms of action-level temporal learning, the corpus contains fewer annotated actions compared to recognition datasets like Kinetics-400 (Kay et al. 2017). and there is significant caption redundancy as actions are often associated with several predicates rather than the entire sentence. For event-level temporal learning, the corpus typically constrained to a single scene and description, whereas rich scene transformations and event-sentence correspondence are crucial. To address these limitations, OL (Wang et al. 2022d) extends the idea of UniCL (Lu et al. 2022) to the video domain and incorporates Kinetics-400 into the training corpus. HiTeA (Ye et al.
2022 tackles caption redundancy through specific fine-grae tiee-gr alignment designs. Compared to above two methods which target at enhancing action-level temporal learning, LF VILA (Sun et al. (202)) constructs mag-ffrm dataset from the large-scale HD VILA_ IILA_100M (Xue et al.] 2022a) dataset to enhance event-level temporal learning. In contrast to the aforementioned methods, this work proposes concatenated sample learning to dynamically compose pseudo video-paragraph corpora from existing image-text or video-text corpus, and simultancously enhance image-text and event-level temporal video-text learning.
Unified Vision-Language Foundation Model. The Frozen model (Bain et al. (021) was the first to leverage mixture of image-text (CC3M (Sharma et al. 2018) and video-text corpus (WebVid2.5M (Bain et al. )02T)) for training VidLP models by sparsely yy sampling the video corpus and treating the image corpus as frozen videos. Subsequent VidLP methods have followed this route to support both
image-text and video-text tasks, with advanced vision backhones (Fu et al. [021 2022
Ye et al.
2022] or specially designed training objectives et 2022a Ye et al. 2022
al. 2022b).
Regarding large-scale unified vision- vision-language foundation models, they can be classified into two types: co-training methods and two-stage training methods. Co-training methods train on both image and video corpora simultaneously and a typical representative method is Flamingo (Alayrac et al.
2022b. Two-stage training methods first train models on image-text corpora and then adapt them for video-text tasks. Image-text models can be trained via contrastive learning (Radford et al. 1021
Jia et al. 202T), generative learning (Wang et al.] 2022b; (Cen et al. 2022 Wang et al. 2021), or combination of both them(Yy Yu al. 2002)). Regarding adapting image-text models to video-text ones, CLIP-hased models (Luo et al.] 2022: Xue et al.) 2022b) employ temporal fusion transformers and frame-word fine- -inener alignment learning to enable text-to-video retrieval. GIT (Wang et al.) 2022b) directly finetune models by concatenating multiple frame features as input to the text decoder and achieves state-of-the -feee----re-rtttt performance on most video captioning benchmarks. VideoCoCa (Yan) et al. (022) adapts the CoCa model (Yu et al.) (202) through continuous learning corpora. MaMMUT (Kuo al. 2023) applies a similar temporal perceiving capability enhancement during finetuning as Tube (Piergiovanni et al. 2022). In comparison to the above methods, COSA simultaneously model image-text and video-text learning during the pretraining stage, with image-text corpora utilized only.
video-language
METHOD
3.1
SIMPLE MODEL ARCHITECTURE
COSA adopts a simple architecture, consisting of a vision encoder (ViT (Dosovisskiy et al. 2020 and a text encoder (BERT (Devlin et al., 2018)), which facilitates both single-modalle encoding and croastmy interaction. Specifically, we introduce additional cross- cross-attention layers between the self-attention and forward layers of BERT. These cross attsnntttnnioon layers are activated during the forward pass for cross-modality understanding or generation, while deactivated during the forward
3

---

Published as a conference paper at ICLR 2024
Table 1: Model configurations of COSA. CC14M is a combination of CC3M (Sharma et al. 2018). CC12M (Changpinyo et al., (021), COCO (Lin et al. 2014). SBU (Ordonez et al.) 2011). and VG (Krishna et al. 2017b) datasets. The largest 1.2B COSA model is trained with data bootstrapping strategy inspired by BLIP (Li et al. 2022b), and captions in training corpora (marked with **) are generated by through
additionally trained captioner, more details can be found in Appendix. The 400M image-text pairs used in CLIP and LAION-40OM (Schuhmann et al. 2021) used in EVAClip are also included in the training examples statisties. LAION-102M randomly sampled subset
from LAION-40MM
Model
Vision Ficeuler
Param
Training corpora
Example
Step
COSA-B
Swin-BeLio et 2021
251M
CC3M- Vib2.
SM
100K
COSA-B
Swin- or 2021
251M
webvii- SM
17M
160K
COSA-L
CLIP/I
468M
WCIHM SM
417M
160K
COSA
VVACIipVVI--77--SS 202-
1.2B
415M
150K
pass for single-modality encoding. The text encoder can serve as single-modality encoder, cross- modality encoder and decoder.
3.2 CONCATENATED SAMPLE TRAINING
Online Sample Concatenation. As illustrated in Figure unlike traditional foundation models that take a single image-text pair or frames-text pair as input (SST, single sample training). COSA employs online transformation to convert them into pseudo video-paragee patropprrgrrphh corpus. To illustrate this process, let's consider the image-text corpus. Given a batch containing ny image-text samples, for the i-th sample S, with its image and text denoted I; and T; respectively, we randomly select ne (n. = 3 by default) other samples from the current batch and group them together. Consequently, the single sample (I, Ti) is transformed into grouped sample (fgroup [T, If,,-.. Ijn.]; T group [T) TT..T...... T), Subsequently, we concatenate both the images and texts in the same order to establish their event-level temporal correspondence. Specifically, for the grouped images, we individually feed them into the vision encoder, incorporate learnable temporal embeddings into the output features, and concatenate them along the sequential dimension. Regarding the grouped texts, we directly concatenate them into paragraph where the temporal relations can be encoded by BERT's vanilla position embedding layer. Through the process of in-order sample concatenation and the use of position embeddings. the model can learn explicit scene-level temporal alignment. This concatenation procedure is applied to all samples in the batch. COSA can also be applied on video-text corpus and in this scenario, we sample single frame from each video and perform the same concatenation process.
The concatenated images can be viewed as series of snapshots from multiple clips within a pseudo video, where each clip captures a different scene accompanied by its corresponding descriptions. is noted that these pseudo videos exhibit rapid scene changes with reduced visual redundancy but little semantic coherence due to the random sampling process. However, our experiments demonstrate that the cost of sacrificing clip coherence is negligible, and random sampling yields superior results compared to other settings where clip relevance is intentionally enforced.
Training Objectives. We employ four training objectives. namely Image-Text Contrastive (ITC), Image-Text Text Matching (ITM), Masked Language Modeling (MLM), and Generative Modeling (GM), to enhance the model's cross-modality alignment, understanding, and generation capabilities. These objectives are modified to accommodate the use of concatenated samples as input, as opposed to single samples.
Concatenated Image-Text Contrastive (CITC). The pseudo video frames are individually passed through the vision encoder, and the average pooling of the output [CL.S] token features serves as the global representation of the pscudo videos. The [CLS] token feature from BERT is utilized as the paragraph representation. A bi-directional contrastive loss is employed to bring paired concatenated samples closer together while pushing away unpaired ones.
Concatenated Image-Terr Text Matching (CITM). This task requires the model to determine whether a pseudo video and paragraph correspond to each other. Hard negative mining is used following the approach in (Li et al. 10) The [CLS] token feature from the multimodal encoder (text encoder) used to compute a binary classification loss through two-layer MI..
4

---

Published as a conference paper at ICLR 2024
Table 2: Performance comparison text-to-video retrieval benchmarks. For fair comparison, evaluation results before employing post-pree post-processing methods such as dual softmax (Cheng et al. 2021 are compared. Recall @ 1 1,5,10 are used as evaluation metries.
Method
Example
MSRVIT
DiDcMo
LSMDC
ActivityNel
R@l
R@5
R@10
R@1
R@:
R@10
R@1
RB5
R@10
R@1
RØ5
R@10
ClipBet et
2021
5.4M
22.0
46.8
59.9
20.4
48.0
60.8
21.3
49.0
63.5
Frozen Binn al 2024
SM
31.0
59.5
70.5
34.6
65.0
74.7
15.0
30.8
39.8
Bridgerormer
2022a
5M
37.6
61.8
75.1
37.0
62.2
73.9
17.9
35.4
44.5
MeLES/ et
37.7
63.6
73.8
36.5
63.9
74.0
0A-1rais Winl e mi
20122
5M
35.8
63.4
76.5
34.8
64.4
75.1
18.2
34.3
43.7
Clover
2124
5M
38.6
67.4
76.4
45.1
74,3
82.2
22.7
-2.0
52.6
VIOLETVZ
37.2
24.0
LF-VILA Stan ot 4022
8.5M
35.0
64.5
75.8
35,3
65.4
COSA-B(OUIS)
SM
42.2
69.00
79.0
57.8
80.6
87.99
27.3
46.7
55.2
55.6
80.7
89.1
OmniVL Want st
17M
47.8
74.2
52.4
85.4
HFTcA Ye er ale puze
17M
46.8
71.2
81.9
56.5
81.7
89.7
28.7
50.3
39.0
49.7
77.1
86.7
SINGULARITY cer all Rizza
17M
41.5
68.7
77.0
53.9
79.4
86.9
47.1
75.5
85.5
VINDLU-B Cheined
79.6
59.2
LAVENDER
2006g
3IM
40.7
66.9
77.66
53.4
78.6
85.3
26,
46.4
57.3
COSA-B(OuES)
17M
16.9
72.1
811.3
64.1
86.1
90.6
31.2
50.9
57.8
59.3
83.8
90,9
All-in-one Wang
1022a
13M
37.9
68.1
77.1
32.7
61.4
73.5
22.4
53.7
67.7
400M
44.5
71.4
851.6
43.4
70.2
890.6
22.6
41.0
49.1
40.5
72.4
X-CLIP e PUs4
400M
19.3
75.8
84.8
47.8
79.3
26.1
/18.4
466.7
46.2
75.5
mPLUG et pizS
417M
53.1
77.6
84.7
56.4
79.1
85.2
34.4
55.2
65.1
UMT-L Li
58.8
81.0
70.4
93.5
73,0
89.1
AALOKETIOR ar 2024
433.5M
544
79.8
87.6
57.5
83.3
88.8
31.8
52.8
62.4
63.4
87.8
94.1
CLIP-VII VIP Andee 20220
SOUM
54.2
77.2
84.8
50.5
78.4
87.1
29.4
50.6
59.0
53.4
81.4
90.0
COSA-LIOUIS)
54.4
77.2
68.3
91.7
67.4
COSA(Ours)
41SM
57.9
79.6
86.1
70.5
89.3
92.4
39.4
60.4
67.7
67.3
89.0
95.0
Concatenated Masked Language Modeling (CMLM). We randomly mask 15% of the tokens in the paragraphs and employ BERT's vanilla prediction layer to reconstruct the masked tokens given the context of the pseudo videos.
Concatenated Generation Modeling (CGM). We randomly mask 60% of the tokens in the paragraphs and utilize the same prediction layer as in CMLM to reconstruct the masked tokens in the context of the pseudo videos. CGM incorporates causal attention masks in the self-attention layers of BERT to prevent information leakage and enhance the model's capabilities for text generation.
In addition to the four training objectives that utilize concatenated samples as input, also include two vanilla objectives (ITC and ITM) that take raw single samples as input to enhance the model's ability to process single samples.
4
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We train COSA models using the PyTorch framework and 64 Tesla V100 cards. To ensure fair comparison with ofate-of-the-art video-language pretraining models and vision-language foundation models, train four model variants with different parameter and data sizes, as illustrated in Table 1. All models utilize BERT-Bar Base as the text encoder. The initial learning rate is set to le-4, and a 10% warm-up strategy with linear decay schedule is employed. The batch size is set to 2048. For ablation studies, we utilize the frozen CLIP/ViT-B/16 (Radford et al. 2021) as the vision encoder for efficiency purposes, and the models are trained on CC3M for 30K steps with batch size of 1024, unless specified otherwise. Six training losses are equally weighted. For pretraining and video-text tasks finetuning, we use 224 image resolution, while for image-text tasks finetuning, we use higher resolution, which is presented together with Other details in Appendix.
4.2 COMPARISON TO STATE- OHE--E-RTTTT
Text-to- Video Retrieval. We evaluate eext-to-vides retrieval on four benchmarks, including MSRVTT (Xu et al. 2016). DiDeMo (Anne Hendricks et al.) 2017). LSMDC (Rohrbach et al.) 2017). and ActivityNet (Krishna et al. 2017a). ITC and ITM are used as the training objectives. During testing, we rank all candidates using similarity scores computed by ITC and then rerank the top-50 candidates via ITM. As shown in Table 2 COSA-B (5M) outperforms previous models of similar scale by a significant margin all four benchmarks, especially for the paragraph-to-viee -oviidoeo retrieval benchmarks (DiDeMo and ActivityNet). This demonstrates the effectiveness of concatenated sample training. It noted that VILA et al. 2022) is customized designed for paragraph-to-viee retrieval with
5

---

Published as a conference paper at ICLR 2024
Table 3: Performance comparison open-ended video QA benchmarks. Accuracy is used as the evaluation metric.
Method
Example
MSVD
ClipBERT Lei et
2021
5.4M
37.4
60.3
ALPRO
5M
45.9
VIOLETY
2029
5M
44.5
54.7
72.8
Clover Hunig 20222
5M
51.9
COSA-R(OBIS)
SM
46.2
54.3
73.4
46.5
OnniVV Wang el 2022d
17M
44.1
$1.0
HITAA CoTEAAooe cla
17M
45,9
55.3
73.2
46.4
SINGULAREI Liv 2022
17M
43.5
43.1
VINDLU-BI Cheng
17M
43.8
44.6
LAVENDER &re 20220
30M
45.0
56.6
73.5
JustAsk Yang dea ZUP4
69M
41.5
46.3
38.9
COSA-B (Ous))
17M
55.5
47.3
MERLOT Zellers
2021
180M
43.1
69.5
41.4
All-in-one Wang at
228. SM
46.8
48.3
66.3
FrozenBil
410M
47.0
54.8
68.6
43.2
mPLUG-2 X 2056
417M
48.0
58.1
75.4
UMT-L di ZUA60
425M
47.1
55.2
47.9
PALOR-L/CCBSS ci 2025
433.5M
49.2
60.0
78.7
48.6
Video Wanerr 20zde
646M
47.1
55.5
72.2
GIT WWag erale 211220
1.7B
56.8
(lamingatsons))0 (Nllyya el 2022
2.3B
47.4
VideoCoCa (2. Yan HI2Z
GIT2 (S.IB) (ang caii 20222
12.98
45.6
58.2
74.9
COSA-L(Our)
COSA(Our))(1.22)
415M
49.2
60.0
79.5
49.9
Table 4: Performance comparison video captioning benchmarks. BLEU@4 (Papineni al. 2002 (B (B@4) and CIDEr (Vedantam et al. 2015) (C) are used as metrics. Results models ut lizing additional modalities such audio (Chen et al. 2023) and subtitle (Tang et al. 202 Li et 202
are denoted by dashed color. Following (Tagg et al. 202T: 202T: Chen et al. 2023 Wang et al. 2022b).
we employ SCST finetuning (Rennie al. 2011 on VATEX benchmark, and the corresponding results are marked with
Mathod
Example
MSRVTT
MSVD
VATEX
YouCook2
TVC
B684
C
B684
B84
C
Be4
C
H64
C
HiTe Ye
SM
62.5
145.1
COSA-BIORIFS
5M
48.1
64.3
66.9
146.0
40.5
72.6
7.8
100.9
15.6
59.2
LAVENDER
30M
COSA-B(OUES)
17M
48.5
64.7
68.7
150.9
41.4
74.4
8.6
106.2
16.2
61.0
VALUE al 80216
136M
58.1
1244
130.3
11.6
50.5
CLIP4C Capanisr
VALOR-I Chen
433.5M
54.4
74.0
80.7
178.5
45.6
95.8*
GIT Wang ae 20560
1.7B
53.8
73.9
79.5
180.2
41.6*
91.5*
10.3
129.8
16.2
63.0
Hammingoug Alyyin 2023
118.6
GIT2 (5.1B) Wanget 20-20
12.9B
54.8
75.9
82.2
185.4
12.7*
94.5*
9.4
131.2
16.9
66.1
COSA-L(OUEE))
417M
53.2
72.1
74.0
169.1
45.4
84.4
10.2
125.8
17.7
67.5
COSA(OurS(1.28)
415M
43.7*
96.5*
70.7
specialized framework and corpus, but fall behinds COSA-B which takes simple unified architecture well as common corpora. In addition, COSA (17M) evidently surpasses VINDLU et al.) 2022) which have searched a best training recipe targeted at retrieval tasks. When compared to large-scale models, COSA achieves the best R @@@ results DiDeMo and ActivityNet datasets and comparable results on other benchmarks.
Video QA. We evaluate video QA on four benchmarks, including MSRVTT-QA (Xu et al. 2017). MSVD-QA (Xu et al.] 2017). TGIF-QA (Li et al., 2016), and ActivityNet-QU (Yu et al. 2019), 9), and formulate as an open-ended generative task to predict answers with questions as prefixes. As shown in Table COSA-B (5M) and COSA-B (17M) outperform all models of similar scale by significant margin. Under the large-scale pretraining scenario, COSA achieves state-of-the-an accuracy on benchmarks, using only 415M training data and 1.2B parameters.
Video Captioning. We evaluate video captioning on five benchmarks, including MSRVTT (Xu et al. 2016), MSVD (Chen & 2011), VATEX (Wang al. 2201)), YouCook2 (Zhou et al. 2018). ) and TVC (Lei et al.] 2020). The GM objective is used, and captions are autoregressively generated during inference. From the results in Table
that COSA-B (5M) outperforms

---

Published as a conference paper at ICLR 2024
Table 5: Performance comparison on text-to-image retrieval, image captioning, and image QA benchmarks. "ZS' stands for zero-shot evaluation. CIDEr (C) and SPICE (S) are reported for captioning. SCST finetuning is employed COCO caption, and corresponding results are marked with
MSCOCCO-Re
Flickr0K-Rei (2S)
MSCOCO-Cap
VQNv2
Method
Example
R@1
R@5
R@
R
R@5
RØ 10
C
day
std
OFA OFA/Wang
2022e
18M
154.9*
26.6
82.0
82.0
BETTX Wane
ATG21
21M
67.2
87.7
92.8
81.5
97.8
147.6
25,4
84.19
$4.03
BLIP
L St 20ccb
129M
65.1
86.3
91.8
$6.7
97.3
98.7
136.7
78.25
78.32
B1
129M
68.3
87.7
92.6
89.7
98.1
98.9
145,8
82.19
82,30
mPLUG
92.6
137.7
VAI Chen al Mikes
433.5M
61.4
84.4
90.9
152.9
25.79
78.46
78.62
Florems
900M
63,2
85.7
76.7
93.6
80.16
80.36
ea
78.6
78.8
Sim Wann C 2031
1,8B
143.3
25,4
$0,03
80.34
ALIGN dhre KiP4
L.8B
59.9
83.3
89.8
75.7
93.8
96.8
2022
82.0
82.1
CoCar2.18 Yo 2010
4.88
80.4
95.7
97.7
143.6
24.7
82.3
82,3
GIT2(5. Wang cl 20226
12.90
152.7
26.4
81.7
81.9
COSA(OurS)L.225)
415M
68,5
85,0
93.0
90.2
95.4
99.4
150.6
27.0"
80.16
80.51
Table 6: Experiment results of SST (baseline), COSA, and COSA variants on a broad range of video/image- video/mmage-languuge language benchmarks. R@ CIDEr, and Acc are reported for retrieval, captioning, and QA tasks, respectively.
Exp
Video Ret
Video Cup
Video QA
Image Ret
Image Cap
DiDeMo
ActiviyNee
MSRVTT
VATEX
YouCook
VATEX
MSRVIT
TGIF-QA
MSCOCO
MSCOCO
TVC
VIST
SST
49.2
43.8
39.
62.4
85.4
67.6
53.9
45.9
73.0
53.9
124.7
14.8
COSA
43.5
63,9
91.9
68.1
125.4
COSA-copy
43.5
39.3
42.3
60.8
84.6
65.8
52.0
45.4
72.2
51.6
122.7
11.9
COSA-shuffle
50.9
48.1
44.7
64.0
88.8
67.7
54.4
46.6
73.9
54.5
124.9
14.2
VIOLETv2, and HiTeA which utilizes a more advanced vision backbone (MViTv2 (Fan et al. (021)) COSA-B (17M) surpasses LAVENDER, which trained on 30M vision-text pairs. When compared to large-scale foundation models, COSA achieves comparable performances on the MSRVTT and MSVD datasets. Additionally, COSA outperforms GIT2, which specializes in captioning tasks, and achieves new state-of-the-art performances
both open-domain benchmarks (Wang et al.) 2019) and domain benchmarks (Zhou et al. 2018 Lei et al. 2020) (cooking and TV), using only 24% of its parameters and 3.2% of the training examples. It is worth noting that both GIT2 and COSA do not any video-text corpus in the pretraining stage, but COSA can generate online pseudo video-parage paragraph pairs. Furthermore, utilizing the vision modality alone, COSA achieves higher CIDEr scores than VALOR and CLIP4Caption++ which additionally incorporate audio and subtitle modalities.
Image-Texx Benchmarks. We evaluate COSA image-text benchmarks, including text-to-image retrieval on MSCOCO (Lin et al., (014) and Flickr30K (Plummer et al., 2015) (zero-shot), captioning on MSCOCO, and image QA on VQAv2 (Goyal et al. (017). As shown in Table 5] COSA surpasses BLIP-2 and BETT-3, and achieves new state-of-the-ard results on the MSCOCO and Flickr30K benchmarks. Additionally, COSA achieves the best SPICE score (27.0) on MSCOCO caption benchmark and comparable results on VQAv2 benchmark.
4.3 ABLATION STUDY
In this subsection, we conduct extensive ablation experiments to demonstrate the effectiveness of COSA when compared with SST and the method's robustness to different vision backbones and training corpora. We also explore the influence of training objectives, iterations, concatenation num- bers, and sampling methods. Besides above mentioned benchmarks, we further evaluate the model on the Visual Storytelling Benchmark (VIST), which requires models to perceive the relationships among multiple images and describe them in sequential paragraphs, thus demanding strong narrative abilities.
COSA vs SST. As presented Table COSA consistently improves performance on 12 benchmarks of video-text well as image-text tasks, with significant margins compared the SST baseline.
7

---

Published as a conference paper at ICLR 2024
Table 7: Experiment results of different training objectives for COSA. R, C, Q in the parentheses represent retrieval, captioning, and QA tasks, respectively.
ITC
ITM
MLM
GM
CTTC
CITM
CMLM
CGM
DiD(R)
ANET(R)
MSR(R)
TVC(C)
(GIF())
49.2
43.8
39.4
53.9
73.0
53.0
50.6
42.8
55.7
73.7
54,8
51.2
43.5
55.4
73.7
52.3
49,9
42.3
55.5
73.6
Table 8: Experiment results of changing dataset and vision backbone (Enc).
Exp
Enc
Data
#Frame
DiD(R)
ANET(R)
MSR(R)
YOU(C)
VIST(C)
TVC(C)
MSR(Q)
SST
Swin-B
CC3M
35.3
30.4
34.7
91.2
14.3
51.6
44.6
COSA
Swin
CC3M
45.1
41.7
36.2
94.6
22.3
53.1
SST
CLIP-B
WebVid2.5M
1
43.6
40.2
40.7
89.3
12.2
53.5
46.0
SST
CLIP-B
WebVid2.5M
4
51.4
47.3
42.5
90.8
11.9
54.0
46,2
COSA
CLIP-B
WehVid2.5M
54.7
52.2
44.0
93.4
17.4
55.0
46.4
We further investigate two COSA variants named COSA-copy and COSA-shuffle. COSA-copy simply duplicates single sample (image and text) multiple times and concatenates them, instead of performing cross-sample concatenation as in COSA. On the other hand, saossesamples from a batch, but with the captions randomly shuffled before concatenation to disrupt event-level temporal correspondence. From the results we can find that 1) Both COSA and COSA- shuffle outperform SST with huge margins all benchmarks, demonstrating the effectiveness of modeling lasoun- connections within sample group instead of single sample. 2) Due to maintaining the same concatenation order for image and text, model can learn better event-level temporal correspondence between visual scenes and descriptions, and thus improve the performance on most benchmarks especially on paragraph-to- vi-ve retrieval benchmarks such as ActivityNet, whose video and caption are relatively longer and contain more complicated scenes. 3) achieves lower performance than SST on most benchmarks, demonstrating that the improvement of COSA over SST comes from multiple samples concatenation, instead of longer visual and text sequences.
concatenates
Training Objectives. As depicted in Table compared to the SST baseline, incorporating COSA while modifying the MLM and GM objectives CMLM and CGM leads to evident improvements across all five benchmarks (b vs a). The default objectives (c) yield the best results for most benchmarks.
Generality. We further explore the generality of COSA with respect different vision encoders and training corpora. The previous experiments employed CLIP-B as the vision backbone and were conducted the CC3M dataset. In this experiment, we replace them with Swin Transformer(Lin jt al. 202T) and Web respectively. From the results in Table X is evident that COSA consistently outperforms SST under different backbone and corpus settings. Additionally, we observe that sampling multiple frames for SSL can improve downstream performance (except for VIST), but still much less effective when compared with proposed COSA.
Equal samples seen. In the previous experiments, models were trained for 30K steps for efficiency purposes. One might argues that the improvements COSA could be attributed to faster convergence rather than cross-sample learning and introduced event-level temporal correspondence, as the model can see each sample multiple times per iteration. However, the results Figure 4 demonstrate that as training iterations prolong, the performance gap between COSA and SST remains consistent, even when SST has already converged. Moreover, pretraining with COSA for 30K steps outperforms pretraining with SST for I 120K steps on all benchmarks, which demonstrates COSA still outperforms SST with same samples seen and much less iterations.
Concatenate Number. Figure shows the performance of models trained with different concatenate numbers (n.) ranging from (SST) to 4. Based on the results, we choose n. = 3 as it consistently performs well across all benchmarks.
Training Speed. We have made statistics that transferring from SST to COSA averagely cost 0.9x more training time when four samples concatenated. We think this is unavoidable cause the frames become more and sentences become longer, but affordable because we only need to pretrain once, and the finetuning, testing, inference speed of COSA and SST totally the same.
8

---

Published as a conference paper at ICLR 2024
Table 9: Experiment results of different sampling methods for online sample concatenation.
Sampling
DiD(R)
ANet(R)
MSR(R)
VAT(R)
TVC(C)
VIST(C)
MSR(Q)
random
54.8
51.2
43.5
63.9
55,4
20.7
46.5
vision similarity
53.7
50.9
43.0
63.6
55.1
20.4
46.4
text similarity
48.2
45.4
42.6
62.4
55.2
20.2
46.3
DIDeMo-RET
ActiviryNer-BET
VATEX-KET
A25
201
VIST-CAP
TVC
TCIF-QA
107
102
C1D2
CIEE
79.5
11.
16
55.
3
Cosconsationo 1311
Conconsation sprt
Figure 3: Experiment results of different concatenation numbers used in COSA.
Sampling Choice. By default, we randomly choose samples within a batch for grouping and concatenation. We also experimented with two alternative choices: selecting the top ne most similar samples based on the similarity matrix computed by either the image or text encoder output features. The results in Table demonstraate the importance of reducing semantic relevance among grouped samples through random sampling. More explanations and visualizations could be found in Appendix.
YOUCOOK-CAP
TVCCAP
VISECAP
0061
AST
005A
500
5.1
CDO-
545
309C
12NG
30K
JMA Sepp
1206
30X
Strps
VATEO-REY
MSDOCC-RFT
COSA
SST + COUDA
5S7
COSA
66
38-
50.5
74-
R021
302
146
23.5
30K
AN6 Stng
12W6
35K
JMN Strgs
1206
31K
100K
Jonnn
Figure 4: Experiment results of SST and COSA when training for more iterations.
5 CONCLUSION
This paper introduces COSA, a concatenated sample pretrained unified vision-language foundation model. COSA transforms image-text corpora into pseudo video-paragran corpora, enabling explicit event-sentence correspondence and promoting event-level temporal learning in foundation models. Extensive ablation studies demonstrate the effectiveness of COSA in improving various downstream tasks, including video-language tasks and image-text tasks. COSA achieves state-of-the-as results on multiple benchmarks. However, it is important note that the proposed COSA has only been proven effective the languaee pretraining field, and we hope that it can inspire further research on co-learning multiple samples in other AI communities.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Major Project (No.2022ZD01 18801), National Natural Science Foundation of China (U21B2043, 62206279).
REFERENCES
Hann-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pp. 5803-5812, 2017.
Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7728-1738, 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In
Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies, pp. 9002000, 2011.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10638-10647, 2020.
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor:
omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345. 2023.
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0667 06794, 2022.
Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. Vindlu: recipe for effective aare---nd-aanguge pretraining. arXiv preprint arXiv:2212.0505// 2022.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. arXiv preprint arXiv:2109.04296 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 11929, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6824-6835. 2021.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv 2021.
preprint arXiv:2//1.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. An empirical study of end-to-end video-language transformers with masked visual modeling. arXiv preprint arXiv:2209.0155 2022.
10

---

Published as a conference paper at ICLR 2024
Yuying Ge. Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridging video-text retrieval with multiple choice questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16167-16176, 2022a.
Yuying Ge, Yixiao Ge, Xihui Liu, Alex Jinpeng Wang, Jianping Wu, Ying Shan, Xiaohu Qie, and Ping Luo. Miles: Visual bert pre-training with injected language semantics for video-text retrieval. arXiv preprint urXiv:2204. /2408, 2022b.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Jingjia Huang, Yinan Li, Jiashi Feng, Xiaoshuai Sun, and Rongrong Ji. Clover: Towards unified Video-language alignment and fusion model. arXiv preprint (xXix2207.007885 2022.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2758-2766, 2017.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- Asioon-anguage representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137, 2015.
Will Kay. Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: arXiv: 1705.0655. 2017.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. 706-715, 2017a.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32 73, 2017b.
Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. Mammut: A simple architecture for joint learning for multimodal tasks, arXiv preprint arXiv:2303. 16839, 2023.
Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. Tvr: A large-scale dataset for vidco-subtitle moment retrieval. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 447-463. Springer, 2020.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan. Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Cliphert for video-and- lanaaauaee learning via sparse sampling. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7331-7341 2021.
Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for video-and-lanne language learning. arXiv preprint arXiv:2206.03422, 2022.
Dongxu Li, Jumnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video-and- languae pre-tratee gre-trainin with entity prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4953-4963 2022a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9700 9705, 2021a.

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp.
PMLR, 2022b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping sinuuge-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training -effininnfiii video foundation models. arXiv preprint 1605, 2023b.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchi- cal encoder for videot language omnireepresnntatioon pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2046-2065, 2020.
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, al. Value: multi-task benchmark for video-and-language understanding evaluation. arXiv preprini arXiv:2106.04632 2021b.
Linjie Li, Zhe Gan, Kevin Lin, Chung- Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Laven- der: Unifying video-language understanding as masked language modeling. arXiv preprint arXiv:2206.071 076600 2022c.
Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4641 4-41-4 2016.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLwrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022 2021.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916. 2022.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Yiwei Ma. Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to- end multi multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 638-6477 647, 2022.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 1m2text:
Describing images using million captioned photographs. Advances neural information processing systems, 24, 2011.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Rethinking video vits: Sparse video tubes for joint image and video learning. arXiv preprint arXiv:2212.03229, 2022.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phree correspondences for richer maage-to-seentene models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-26499 2015.
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7008-7024, 2017.
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision, 123(1):94-120,, 2017.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.021 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565, 2018
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15388 2023.
Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. arXiv preprint arXiv:2210.0603// 2022.
Mingkang Tang, Zhanyu Wang, Zhaoyang Zeng, Fengyun Rao, and Dian Li. Clip4caption++: Multi-clip for video caption. arXiv preprint (eXiv:2110.05200 2021.
Ramakrishna Vedantam, C Lawrence Ziticck and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4575, 2015.
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge. Xudong Lin, Guanyu Cai, Jiamping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. arXiv preprint urXiv:2203.07300 2022a.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022b.
Jinpeng Wang, Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Object-aware video-language pre-training for retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3313-3322, 2022c.
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint arXiv22209.07526 2022d.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through learning framework. In International Conference on Machine Learning, pp. 23318 23340. PMLR, 2022e.
a simple serueneeeeeeeeee
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022f.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li. Yuan-Fan Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for Madeo--an--angusee research. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 45911, 2019.
13

---

Published as a conference paper at ICLR 2024
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Intervvideo: General video foundation models via generative and discriminative learning. arXiv preprint arX1v:2212.03111, 2022.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprini arXiv:2 (00::::000.00044,,, 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. mplug-2: A modularized multi-modal foundation model across text, image and video. arXiv preprint arXiv:2302.00402 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 52885588855 2888-,29,, 2016.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5036-5045, 2022a.
Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-traines trained image-text model video-language representation alignment. arXiv preprint arXiv:2209.064/ 2022b.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979 2022.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1686-1697, 2021.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.08155, 2022.
Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. Hitea: Hierarchical temporal-awe aware video-language pre-training. arXiv preprint arXiv:2212.14546 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2005.01917, 2022.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 9127-9133 2019.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprini arXiv:211i..145: 14332 2021.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-23653 2021.
Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In Secroy AAAI Conference on Artificial Intelligence, 2018
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PRETRAINING SETTINGS
Table 10: Pretraining Settings for COSA models. means that captions in those corpora has been replaced by descriptions generated by separately trained captioner.
Model
Vision Encoder
param
Sample
Training Corpus
Batch Size
Steps
Epoch
CC3M
2048
50000
35
COSA-B
Swin-B
251M
5M
Wid
2048
50000
41
CC4M
2048
60000
25
COSA-B
Swin-B
251M
17M
CC12M
2048
60000
12
Wed
2048
40000
33
CC4M
2048
60000
25
COSA-L
CLIP-VIT-L
468M
(17+400)M
CC12M
2048
60000
12
WebVid2.5M
2048
35000
29
CC4M
2048
50000
20
COSA
EVAClip-Vi--G
1.2B
(15+400)M
CC12M
2048
50000
10
LAION-102M
2048
50000
The specific pretraining settings of all scales of COSA models are presented in Table I0 It is
noted that the training corpus of the largest one (COSA) has been clean by a separately trained captioner. Specifically, the captioner takes the same model architecture of COSA and is pretrained with generative modeling loss (GM) only on the same corpus with original captions (CC4M, CC12M and LAION--12M4 without concatenated samples applied. After pretraining, the captioner is finetuned with GM loss and MSCOCO. At last we use trained captioner to generate high-quality captions for CC12M and LAION-102M with top-10 10 sampling method.
mixture of downstream of datasets including MSRVTT, VATEX, MSVD
A.2 DOWNSTREAM DATASET DESCRIPTIONS
COSA is evaluated on a series of downstream datasets including MSRVTT, MSVD, LSMDC, DiDeMo. VATEX, YouCook2, ActivityNet Caption, TGIF, TVC, MSCOCO, Flickr30K and VQAv2. Train/val/test splits of different benchmarks of those datasets are presented in Table II
Specifically, MSRVTT Xu al. 2016) dataset consists of 10K video clips and 20K captions. Our evaluation encompasses text-to-vides retrieval, video captioning, and video QA tasks on this dataset. For retrieval, we adopt the 11KAA split' and for captioning and QA, we adhere to the standard split. MSVD(Chen & Dolan 2011) dataset comprises 1,970 videos, each accompanied by approximately 40 captions. Our eval evaluation involves video captioning using the official split and video QA using the split proposed by Xu et al. al. et al. 2017). LSMDC (Rohrbach et al. (017) dataset consists of 118,000 clips from 202 movies, with each clip paired with a single caption. We evaluate text-to-video retrieval on this dataset using the official split. DiDeMo (Anne Hendricks et al. (0117) dataset includes 10K long-form videos sourced from Flickr. For each video, four short sentences are annotated in temporal order. We concatenate these short sentences and evaluate iparagraph-to-vidde
retrieval and the official split is employed. VATEX Wang et al. 2019) dataset contains 41 ,,250 video clips sourced from the Kinetics-600 (Ka al. 2017). accompanied by 825,000 sentence-level descriptions. Our evaluation focuses on text-to-video retrieval and video captioning tasks. Regarding 2020)
captioning, we employ the official split, while for retrieval, we follow the HGR (Chen split protocol. YouCook2 (Zhou et al. a (2018) dataset comprises 14K video clips extracted from 2K instructional cooking videos on YouTube, Each video showeases multiple actions performed by chef, along with corresponding textual descriptions and temporal annotations. We assess video captioning on this dataset using the official splits. ActivityNet Caption(Krishni et al., 2017a) dataset encompasses 20K long-form videos sourced from YouTube whose average length are 180 seconds.
We evaluate text-to-vidd video retrieval and video QA on this dataset. For retrieval, we use the official split, while for video QA, adopt the split proposed by Yu et all et al. 2019). TGIF (Jang cet al. 2017) dataset features three video QA benchmarks, including TGIF-Action, and Fram- with the first two being multiple-choice QA and the last one being open-ended
15

---

Published as a conference paper at ICLR 2024
Table 11: Downstream datasets splits.
#Videos/#lmages
#Captions/#QA A-pais
Task
Benchmark
Train
Val
Test
Train
Val
Test
MSRVTT
9000
1000
180000
1000
DiDeMo
8394
1065
1003
8394
1065
1003
Text-to-Vide Retrieval
ANET
10009
4917
10009
4917
LSMDC
101046
7408
1000
101046
7408
1000
MSRVTT
6513
497
2990
130260
9940
59800
YouCook2
10337
3492
10337
3492
Video Captioning
MSVD
1200
100
670
48774
4290
27763
VATEX
25991
3000
6000
259910
30000
60000
TVC
86603
10841
174350
43580
MSVD-QA
1200
250
520
30933
6,415
13157
TrIF-FrumeQA
32345
7132
39389
13691
Video QA
MSRVTT-QA
6513
497
2990
158581
12278
72821
ANET-QA
3200
1800
800
32000
18000
8000
MSCOCO
113287
5000
5000
566747
25010
25010
Text-to-Image retrieval
Flickr30K
29000
1014
1000
145000
5070
5000
Image Captioning
MSCOCO
113287
5000
5000
566747
25010
25010
Image QA
VQAv2
82783
40504
37K/81 K
4437570
2143540
1.IM/4.5M
QA. We evaluate VAST on the TGIF-Frame benchmark using the official split. TVC (Le et al. 2020h dataset is multi-channel video captioning dataset, comprising 108,000 video moments and 262,000 paired captions. Additional input in the form of video subtitles can be utilized. We evaluate video captioning on this benchmark using the official split with vision information utilized only. MSCOCO (Lin et al. (014)) dataset contains 123K images, each paired with five annotated captions. We evaluate tex--to-image retrieval and image captioning tasks on this dataset using the Karpathy split KKarpathy& & Fei-Feil 2015). Flickr30K (Plummer et al., 2015) dataset consists of 31 images, with each image paired with five annotated captions. We assess text-to- tottto retrieval on this dataset using the Karpathy split (Karpathy & Fei-Feij 2015). VQAv2 (Goyal et al. 2017) dataset comprises 1.1 million questions and 11.1 million answers related to MSCOCO images and we utilize train+val set for training and evaluate the online test server.
A.3 VISUALIZATIONS OF DIFFERNET SAMPLING METHODS
In our experiments, COSA trained with random sampling works better than the one trained with relavant sampling. As shown in figure we observe a real video-language sample from ActivityNet caption dataset, which shown in the top green part of figure, we can find that the 1) sampled vision frames relevant and coherent, and 2) the corresponding texts very informative and can strictly match to different frames in temporal order. Motivated by this, an ideal large-scale pretraining dataset for video-language tasks should also possess those two characteristics help model learn the capabilities of perceiving evnets happed in videos and their relative order. In this work, we propose COSA imitate those pretrain dataset from image-text data. And the examples of COSA with different sampling methods are shown in the bottom pink part of the figure, from which we can find that even though the random sampling example shows little relevance between four independent scenes, but it has the strenth that each sentance can strictly correspond to one image, which satisfy the second charatersic. By contrast, even though relevant (vision/text similarity) sampling examples have shown vision relevance some extent, they have strong semantic redundancy, and one sentence can correspond to many frames, which could confuse models' event-level temporal learning. And we think that's 's why it performs weaker than random sampling. In addition, considering that the scene changes are somewhat hard random sampling, we think other soft sampling methods could be a future research direction.
16

---

Published as a conference paper at ICLR 2024
Real video-paragraph example from ActivityNet- caption dataset
train by spare. puts away equipment gives driving away.
road.
ground,
Pesudo video-paragraph example from CC3M COSA
random sampling
drowning persistently
walking everyday length elderly playar during day tournament
outdoors, tannis player compates against
vislon-similarity sampling
and person promiero, person promiere ame empire theater, astronaut and person attend premiere ame empire theater, pop artist and dramatist walk red carpet during festival.
text-similarity sampling
city, square part post
castle, town foreground.
Figure 5: Visualization of different sampling methods of COSA. The top green part in the figure is the example from real video dataset (ActivityNet-captions and bottom pink part in the figure are examples of COSA with different sampling methods.
17

---

Published as a conference paper at ICLR 2024
Table 12: Downstream task finetuning settings of COSA models. Ir, Bs, Epo, F_train, F test and Res denote learning rate, batch size, epoch, training sampling frames, testing sampling frames and resolution, respectively. Task
Benchmark
Lr
Bs
Epo
train
F
test
Res
MSRVTT
2e-5
64
3.6
16
224
VATEX
2c-5
64
2.5
16
224
Text-to-Vidde Vidco Retrieval
DiDcMo
2e-5
64
40
8
32
224
ANET
2e-5
64
20
8
32
224
LSMDC
2e-5
64
5
32
224
MSRVTT
2e-5
128
10
8
8
224
YouCook2
3c-5
64
30
16
224
MSVD
lc-5
64
1,2
8
224
Video Captioning
VATEX
2e-5
64
10
20
224
VATEX(SCST)
7c-6
64
5
20
224
TVC
3e-5
64
40
8
8
224
MSVD-QA
le-5
64
18
224
TGIF-FrameQA
2e-5
64
10
4
224
Video QA
MSRVIT-QA
2e-5
64
4,5
8
224
ANET-QA
2c-5
64
10
16
224
Text-to-Image retrieval
MSCOCO
lc-5
256
5
384
MSCOCO
le-5
64
5
480
Image Captioning
MSCOCO(SCST)
2.5e-6
64
2.5
480
Image QA
VQAv2
20-5
128
20
384
A.4 FINETUNING SETTINGS
Specific finetuning settings of COSA model such learning rate, batch size, training epochs and sampled frames are presented in Table
A.5 QUALITATIVE COMPARISON ON VIST BENCHMARK
The qualitative comparisons between model trained with SST (Single Sample Training) and proposed COSA (Concatenated Sample Training) prediction visual story telling benchmark (VIST) is shown in Figure 6, from which we can find that COSA can generate comprehensive captions (storys) for multiple frames under both zero-shot and finetuning settings, while SST can only generate one sentence deseribing limited contents under zero-shot setting, and is easy to generate duplicated captions under finetuning settings.
18

---

Published as a conference paper at ICLR 2024
10
sunshine.
SSZIEn this grew up, where grow up. this where i grew up. this where am growing up.
grew up. grow up.
COSA(28) the front of house. gr graduation ceremony. a city sunshine.
cathedral from outside the cathedral. photo a graduast and student
COSA(FT)
was very thamselves, building was very beautiful.
buildding was very nice. architscture was very old, the students very proud
buildiing a lot because looks angle looks even better. a couple of my classmates are going to graduate. this one beautiful building.
GroundTrutt where go school.
SST22)
center, watches
SS7(F7) want baseball game, there were lot people there. some them were crazy. some the players didn't crowd crazy.
COS4/2S) baseball players against team. general the scoreboard.
game. watch baseball game. person, mascot,
game
COSA/ED) the baseball game weekend. there great night.
people took lot pictures. got
players.
GroundTrvth:: we are going to watch baseball game today: the players are doing great job, we even got to interact with the crowd. the mascot was great, and very funny. final score is here, what a great day.
SST12
graduate calebrates after receiving his diploma.
finally graduate college. group friends and family gathered pool graduation ceremony. boy very finally graduate. the family was very proud him.
COSA(ZS): person
during graduation ceremony. person, family during graduation ceremony. person, and person people.
receiving
receiving diploma school's
commencement ceremony. pool
COSA(FI) graduation day is finally hare. all the students line receive their diplomas. the students wave to the crowd. family and friends for after the ceremony, the family relaxes pool.
GroundTruth: day finally aniivd.
thrilled finally graduation. cap off day, hung
started
everyone posed
SST/ZS
photo.
SST(ED)
people event today, characters were very popular, characters were very popular. characters were very funny. the characters very popular.
COSA people walking in street. parson, left, person, right, enjoy day a bicycle with attached person with the girls the party
COSA(FT) there were lot of people at the festival today, there were a lot funny characters. some of them were very cute. they rode bikes around the city. was great time.
Ground/rrth the streets ware during the some came dressed huge cartoon outfits. they were fun and take pictures with. we got around using rent-able bikes. it was memorable day.
Figure 6: Qualitative comparison between SST and COSA on the visual story telling (VIST) bench- mark. 7S and FT denote zero-shot and finetuning, respectively.
19


Reviews:
[{"review_id": "gXWVrc0GdP", "reviewer": "Reviewer_kord", "summary": "This paper proposed to concatenate image-text samples to mimic video-paragraph corpus in vision-language pre-training. The method is simple and the evaluation is conducted on various image/video datasets to demonstrate impressive performance.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is simple and easy to reproduce. Meanwhile, the performance gain is impressive.\n2. The experiments are conducted on many benchmarks across image-text and video-text tasks, as well as different data scales. Also the ablation is comprehensive and covers most of the aspects of this method.", "weaknesses": "1. It makes sense that pseudo video-paragraph data in pre-training can mitigate the gap between pre-training and fine-tuning in image-text pertaining. However, intuitively, the discontinuity of semantics in pseudo video-paragraph data should hurt compared with relevant video-paragraph data because in downstream videos, image and text are indeed relevant. But in Tab9, it seems random sampling is better than relevant sampling, which is kind of counter-intuitive. Can the authors explain more about it?\n\n2. When having seen the same number of samples, whether COSA is better than SST in `image-text downstream tasks`? Basically, I want to see the comparison like Figure 4 in image-text downstream tasks. I am okay with this observation not holding anymore in image-text downstream tasks because essentially video-paragraph and image-text are different domains.\n\n3. I want to see how this method performs in zero-shot image-text tasks. Considering the domain gap, I suspect it might perform worse than some image-text pre-trained methods that COSA can outperform when finetuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OUYXw0B4aO", "reviewer": "Reviewer_vqzh", "summary": "The work proposes the vision-language foundation model, which can jointly model visual contents and event-level temporal cues using only image-text corpora. Extensive experiments demonstrate that COSA consistently improves performance across a broad range of semantic vision-language downstream tasks.", "questions": "see weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes the effective method for video-text and image-text tasks.\n2. The experiment is very adequate.  The model consistently improves performance\nacross a broad range of semantic vision-language downstream tasks.", "weaknesses": "1. The reasons for the improvement brought by Concatenation lack detailed analysis. Why is there also improvement for image-text tasks? Why is it necessary to include the video dataset (web2vid)? Why wasn't the 1.2B model included in the video dataset?\n2. The data shown in Table 1 is confusing. The data for COSA-L is 417M, while the data volume for COSA is 415M.\n3. The results in Table 7 and Table 7 are also confusing. The best performance is based on 6 pretraining task? Which pre-training tasks were used in the overall experimental results of COSA? Is the WebVid2.5M dataset more important，the results for COSA 4 frames?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jyfSb5qPmA", "reviewer": "Reviewer_gWLj", "summary": "This paper proposed a new vision-language pre-training framework, called COSA. In particular, COSA augmented original image-text pairs by concatenating multiple examples as pseudo video-text pairs. Extensive experiments were conducted covering both video-language and image-language tasks, and demonstrated the effectiveness of the proposed method.", "questions": "- Is it possible for the authors to include tasks which rely much on temporal information like temporal action localization? This would provide better understanding of the proposed method.\n- It would be better if results with different initializations can be presented to remove my concern about better CLIP initialization.\n- It was worth trying data augmentations like mixup and it might lead to similar performance gain as demonstrated in the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow. In addition, the proposed method was supported by comprehensive experiments together with ablation studies, which made the paper a complete work.\n- The method COSA itself was simple yet effective to improve the learned representations for downstream tasks, and at the same time, it did not introduce extra computational costs.", "weaknesses": "- The method was more like a trick of data augmentation instead of a significant technical contribution, as it just simply concatenated images and their corresponding captions and it was not very surprising to observe performance improvements.\n- As it was mentioned in the paper that apart from modified objectives, COSA also included original objectives for pre-training on image-text pairs. It was a complicated design to have so many training objectives and it was unclear how they were weighted (seemed to be equally weighted). Even though there was an ablation study of training objectives in Table 7, it still did not explain well the contributions of each item.\n- The method leveraged the average pooled [CLS] token for each image as the final representation for the pseudo video. In this way, there was actually no temporal information considered. And the selected downstream tasks were less dependent on temporal information in the meanwhile. It would be better if tasks such as temporal action localization were included to show whether COSA can improve those tasks. In addition, since temporal information did not play any role in current method, I am afraid that using augmentations like mixup for videos/images might lead to similar performance gain, as shown in [1].\n- Previous works showed that using CLIP initialization could lead to better performance. Among compared baseline methods, some of them such as MILES [2] actually used ViT trained on ImageNet for image classification and it was not a fair comparison to COSA with CLIP initialization.\n\n[1] Hao, Xiaoshuai, et al. \"Mixgen: A new multi-modal data augmentation.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\n[2] Ge, Yuying, et al. \"Miles: Visual bert pre-training with injected language semantics for video-text retrieval.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdYW13uAEI", "reviewer": "Reviewer_bdF4", "summary": "This paper proposed a concatenated sample pretrained vision-language foundation model. By sequentially connecting multiple image-text pairs as pre-training inputs, it can jointly model visual content and event-level temporal cues using only image-text corpora. Extensive experiments show the effectiveness of the proposed method.", "questions": "1. In Table 9, why is it better to concatenate random images for training than to concatenate only semantically similar images?\n2. It is better to give the weights of the 6 losses (training objectives) and the size of the input image in the implementation details.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tMotivations: This paper presents a very important problem of how to capture time-level event clues using only image-text data. In the case of insufficient quality and quantity of video data, it provides a very important help for the pre-training of vision-language foundation model. The proposed method is very simple and effective. I think this work is easy to follow and most of the techniques are correct.\n\n2.\tExtensive experiments: A large amount of experimental evidence is provided in this paper, which fully verifies the effectiveness of the proposed method.", "weaknesses": "1.\tTechnical contributions: The proposed method is simple and effective. However, the proposed method is not surprising enough, because using pictures to enhance video pre-training has been quite explored in the field of CV/ vision-language pre-training field. This paper combines many existing pre-training methods, so the technology sharing is limited.\n\n2.\tMore explanation: The continuous frames in the video are similar, and there are relations between different events. Establishing event-level correlation includes two parts: (1) the first part is to distinguish between different events. (2) the second part is to make temporal inferences between similar or related frames. The proposed method randomly splices several pictures, but there is no correlation between these pictures, so the model can only distinguish different events, but can not make the model time sequence inference between frames. Therefore, I do not think that the proposed method fully corresponds to its motivations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)