Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Authors: Chenyu Yang
Xizhou Zhu
Jinguo Zhu
Weijie Su
Junjie Wang
Xuan Dong
Wenhai Wang
Bin Li
Jie Zhou
Yu Qiao
Jifeng Dai
Venue: NeurIPS 2024 poster (2024)
Keywords: vision pre-training
compression learning
interleaved image-text data
Primary area: machine_vision

Abstract: Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent on the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of a causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets (e.g., LAION), but can also leverage interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch, showcasing the potential of vision model pre-training with interleaved image-text data.

Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Chenyu Yangl. Xizhou Zhu Jinguo Zhu Weijie Su²5, Junjie Wang Xuan Dong!.3, Wenhai Wang24 Lewei Lu', Bin Li,, Jie Zhou', Yu Qiao', Jifeng Dai 1,20a
TTinnguua University 2 #OpenGVLab, Shanghai Al Laboratory
Time Research *The Chinese University of Hong Kong
University of Science and Technology of China "Xi'an Jiaotong University
'Beijing University of Posts and Telecommunication
x--ngcy22,xdoo2211 ammannl aamiil..tiinnuua.......a. edu.
(zhuxizhou, jzhou, desiighha aataaa.dduccom lechatelia@stu .ft..eeu....u.......... edu. jackroosdmail. wstc. edu. en, jjwangdbupt. edu.
whwangēie cuhk. edu. hi, binli@ustc.en edu. en, qiaoyu@pjlab.org org. luotto@sensetime.com
Abstract
Recently, vision model pre-traninng has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (I.CL. for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method only matches the performance of CLIP paired mrettrinnnn datasets (e.g., LAION), but can also leverage interleaved pre- pre-training data (e.g., MMC4) to learn robust visual representations from scratch, showcasing the potential of vision model pre-training with interleaved image-text data. Code is released https: // thub. com DLaDaGGGLLLb.L..
Introduction
Over the past decade, ImageNet [34] pre-trained vision models have significantly advanced computer vision, continuously achieving breakthroughs in various vision tasks [7, 24, 10]. The success of ImageNet has inspired further exploration of better methods for pre-training vision models from scratch. Recently, the focus of pre-training has shifted from manually annotated data to large-scale, web-crawled image-text data. A key milestone in this shift is CLIP [55], which utilizes image-text pair data hundreds of times larger than ImageNet, delivering superior performance across various tasks and progressively becoming the mainstream method for vision model pre-training. Building on this trend, there increasing interest in exploring interleaved image-text data, which more prevalent the Internet. Unlike the structured image-text pairs used in CLIP, this interleaved data is free-format and non- non-paired, larger in scale, and richer in textual information. Fully exploiting these interleaved image-text data is necessary for further improving vision model pre- pre-training scale.
Equaa contribution. This work is done when Chenyu Yang, Jinguo Zhu, Junjie Wang and Xuan Dong are interms at Sense Time Rescarch. Caatehhhooororoonninnnn author: Jifeng Dai <daijifeng ssinghua.cdu.cnz.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Visual Embedding
Text Embedding
Discardable after Training
Frozen Fine-tuned
Vision Model Pre-training
Vision-language Alignment
Contrastrer Loce
Commantion Lowe
Gonerative
Geverotiv
Vision Encoder
Text Encoder
Vision Encoder
Language Model W
Pre-trained
Vision Encoder
Model
Image
Interleaved Image- Text
Interleaved Image-Text
(e) CLIP Pre-training
Our Latent Compression I. Learning
(c) Multi-modal Incremental Training
learning vision models from scratch
learning vision models from scratch
learning vision models from scratcc
trinning interleaved data
training, interleaved
training on interkkaved data
Figure I: Comparison of different training frameworks, (a) Contrastive learning framework from CLIP [55] pre-trains vision encoders from scratch with image-text pairs, but does not support interleaved data. (b) Our proposed LCL pre-training frameworkm can effectively pre-train vision encoders from scratch with interleaved image-text data. In these two frameworks, the text encoder or the language model that provides supervision can be optionally discarded during the transfer stage. (c) Multi-modas incremental training process uses interleaved image-text data align the pre-trained vision encoder and the language model, but it cannot pre-train vision encoders from scratch.
Currently, no pre-training method can effectively utilize interleaved image-text data to pre-train vision models from scratch. In preliminary attempts [2, 29, 44] of using interleaved data, vision models were already pre-trained by CLIP on paired image-text data. Subsequent training on interleaved data primarily serves to align the pre-trained vision models with language models, thereby enhancing the multi-modal capabilities the entire vision-language network. Therefore, it remains an important and open problem of how to effectively learn robust visual representation from scratch on interleaved image-text data.
A recent study [30] in Natural Language Processing (NLP) suggests that the success of modern language models originates from the compression of training datasets into model parameters. We believe that such compression learning is also applicable to the multi-modal field, except that the data be compressed expands from structured plain texts to interleaved image-text data, where the images are of raw pixels and unstructured. Such compression learning should be revised to accommodate to the image data. Raw pixels are unstructured and often contain unnecessary and unpredictable details. Such details are irrelevant to the high-level semantic tasks and should be discarded in compression learning. Thus, we argue that compression learning on interleaved text-image data should be applied to latent image representation to better extract semantic abstracts.
In this paper, we propose novel visual pre-training framework, named Latent Compression Learning. We first theoretically demonstrate that effective latent compression learning can be performed by maximizing the mutual information between outputs and inputs of a causal attention model. When applied to visual pre-training on interleaved image-text data, visual latents are extracted through visual encoding network (such as VIT [20]), and then fed together with the text into a causal model. The optimization objective can be derived and decomposed into two parts: 1) contrastive learning between visual latent representation and their previous context to enhance semantic consistency, and 2) auto-regressive prediction to learn the predictability of visual representation for subsequent text. These two training objectives complement each other. For images, the learned latent representation retain information that can be predicted from previous contexts and information needed for predicting subsequent contexts, thus providing effective visual pre-training.
In the experiments, various interleaved and paired pre-training methods are evaluated. The evaluation conducted through transfer learning on multiple tasks, including image classification, image-text retrieval, image captioning, and visual dialogue. The pre-training datasets include the widely used image-text paired LATON-400M [57] and the image-text interleaved MMC4 [88] and Obelies [36]. In addition, we also re-organize interleaved version of LAION- and a paired version of MMr4-Pair, to facilitate the comparison between interleaved and paired pre-training methods under the same data source. Experiment results show that our LCL pre-training can achieve the same

---

performance as CLIP on paired pre- -retraining data and can better utilize interleaved pre- vre-training data. Our results also demonstrate the effectiveness of using interleaved image-text data to learn robust visual representation from scratch, and the potential of compression learning for visual pre-training.
2
Related Work
Vision- Pre-training Methods. Supervised Pre-training on large-scale annotated datasets [24, 62, 48, 82] has remained the mainstream method for long time, and has been favored by various vision tasks [9. 24, 79, 7] demonstrating strong performance.
Supervised Pre-training has gained significant popularity due to its advantage of utilizing unlabeled data. BEWT [6] follows the methodology of BERT [19] by randomly masking image tokens and reconstructing them as targets. masked pixels as reconstruction targets, making the pre- training process more straightforward and efficient. Saakly-Supereieed Pre-training leverages image- hashtag [50, 76, 63] and image-text datasets [71, 61, 8, 57], which rely on noisy text supervision datasets, related works [50, 63] have shown comparatively good performance across various transfer learning settings. In the case of image-text datasets, early efforts [3, 42, 49, 64, 66, 67, 70, 13, 39] focused on learning general visual-linguistic representation. Recently, exemplified by CLIP [55]. methods [55, 32] have been developed that involve pre- pre-training through aligned visual- iisuallinguuisic representation, achieving outstanding results in image classification tasks. Other works like M31 Pre-training [65] propose unified framework that integrates multiple pre-training strategies with data from various modalities and sources. Currently, weakly supervised pre-training from web-scale text supervision has become the core of multi-modal understanding, but existing methods have not yet to leverage the most widespread interleaved image-text data for training visual representation from scratch.
MAE [27] and SimMIM [80] directly
from the internet. For
Interlcaved Image- Text Incremental Pre-training. Training using Interleaved Image-Texi Data (IITD) has recently garnered significant attention due the vast amount of such data available online. Recent (8rks[87, 68, 69, 23, 4] such as Flamingo [2] and KOSMOS-1 [29] perform incremental learning on non-public IITD based on previously pre-trained vision and language model parameters as initialization, training text generation models with modti understanding capabilities. With the continuous advancement the field and the proliferation of public IITD (e.g., MMC4 [88] and OBELICS [36]), numerous models [83, 85, 14] capable of modal understanding have emerged. However, these efforts only perform incremental pre- trainiiiiig on IITD and only analyze the usage of IITD on multi-modal dialogue models. Whether IITD contributes to learning robust visual representation from scratch remains unknown.
Compression Learning in NLP. The perspective [59, 60] that compression is closely connected to intelligence has long history. A common compression method is arithmetic coding [56, 52], which is a practical approach that uses probability models for optimal data encoding. Some studies [28, 51, 38, 31, 37] argue that compression and intelligence are fundamentally equivalent. With the popularity of large language models, the equivalence of language modeling and compression has once again drawn widespread attention, prompting numerous explorations. Recently, [18] demonstrated through examples that language models serve as universal compressors. [30] posited that language modeling is equivalent to compressing a dataset into model parameters and proposed that compression efficiency is linearly related to model capabilities. Although compression learning has been proven effective the field of NLP, it is not clear whether this approach can be extended to other fields.
Text supervision pre-trained vision models [55, 32, 40] are widely utilized and have exhibited superior performance in tasks ranging from image retrieval and image classification to captioning. Currently, the most popular and closely watched application area is multi- modal dialogue. Multi-modal dialogue models [2. 41, 86, 17, 25, 21, 53] primarily rely on a powerful pre-trained vision encoder [55, 22] and text decoder [16, 74, 75]. The usage of pre-trained vision encoder can generally be divided into two categories: the majority [2, 721, represented by LLaVA [46], employ strategy where the pre-trained vision encoder is frozen, and only the subsequent adapters and language models are trained. A minority, exemplified by Qwen-VL [5, 53], utilize high-quality image-text dialogue data to continue fine-tuning the vision model. These two training strategies correspond to the two evaluation methods used in this paper to assess the performance of vision models.
Multi-modal Large Models.
3

---

3 Method
3.1 Latent Compression Learning
Auto-regressive Language Modeling as Compression Learning. Recent works [18, 30] have shown that auto-regressive language modeling is equivalent to compression learning. Suppose Given input text sequence I = (
, I1, .2,,...,,..,,.......... NN). where 
 is a special token indicating the beginning of text, the model outputs y 90 (2) (y1, V2... . yN) predicting the next token based on preced- ing context, L.e, 2g = ys = Io (z) The approximate probability of æ estimated by 9a is - - ( (x))). The model is optimized with NLL-loss, which equals to mini- mizing the the cross-entropy between the data distribution and model distribution
96 is language model (LM) with learnable parameters
N
(p. g) = Ezup
log (INLUK go(())))
Notice that (p, 9) is actually the optimal expected code length encoding p by 9, minimizing H (p.g) just means compressing the data into the model parameters.
Latent Compression for Interleaved Image-Text Data. We believe that the compression principle could apply to multi-modal domain, specifically, to train vision-language models by compressing interleaved image-text data. However, instead of directly dealing with pixel values, turn to compress high-level image representation for the following reasons: 1) high-level representation can extract useful information from raw pixels while discarding those unpredictable image details. 2) the learned visual representation will align with text semantics, making it possible to perform effective visual pre-tra pre-training with interleaved image-text data.
Specifically, let x = (
, F1, 22...., .NN) be an interleaved image-text sequence. To simplify the expression without loss of generality,
assume that there is only one image in the sequence. Sub-sequence T are M + 1 image patch tokens of the input image (e.g., non-overlapping patches in ViTs) and others image patches, and T = (1...., i 1, + M + 1 1...,,. N) denotes the indices of text tokens. As shown in Fig. to construct the sequence of latent representation image patches, we use a parametric vision encoder fa (e.g.. ViTs) to map the data sequence Fai4 their
text tokens. 1 = {i, i 1, 1,...,
M) denotes the indices of the
(
, Z1, 22, Æ2,.... ZN), for
into latent variable 24s44MM M. For text tokens, we directly use one-hot vectors corresponding vocabulary ids as the latent codes. Then, the latent representation z are fed into a causal attention model 9o for latent compression by minimizing
N H(p,)) - p(=) = EaND Σ
/
p(zz (r)) log (ZulUk = 96 o fo(z)a)
(2)
where the k-th element of the output y/s = 9o fe(x) predicts the next input latent zд. is identity for text tokens for simplicity of annotation. When the compression only applied on text tokens, it degenerates to auto-regressive language modeling on interleaved image-text data used by previous methods (e.g., Kosmos and Flamingo).
However, direct optimizing Eq. (2) for learning informative latent representation is non-trivial, since Eq. (2) suffers from a naturally trivial solution of visual representation collapse, i.e. the image latent representation Zi:++M M may be learned to be data-independent, In fact, showed in Sec. 4.2, we have observed such visual representation collapse, when training from scratch on MMC4 dataset with Eq. (2) applied to text tokens only (i.e., auto-regressive language modeling).
Maximizing Mutual Information for Latent Compression Learning. Optimizing directly for latent compression in Eq. (2) may cause the visual representation collapse. A natural constraint is to maximize the representation entropy to prevent collapse. We find that combining latent compression and maximum entropy constraint is exactly equivalent maximizing the mutual information between the model inputs and outputs.
Prior work [65] have shown the relationship between cross-entropy and mutual information in other pre-training tasks. Here, we derive this relationship in the latent compression task: maximizing the mutual information between the output y and the input latent of the causal attention model go is

---

equivalent to compressing by minimizing H 9) in Eq. (2) meanwhile maximizing the entropy of each element 2i: in 2:
N
/
log p(zk (bx)
[1y; 2) = Exzup
p( p((x))
N
max Exwu
p(eala) log (zulun - 96 o fo(z)))
Σ
/
p(zx) log log p(z))
--min min Hp, () + Σ H(x),
9
where we use p(ye =x|2) = plyx (x) (x)p(z) (rr in the first step since 25 and y/k can independently computed given input z, and p( y1) is estimated by an approximate parameterized distribution ((2x (k). For the derivation
the formula, please refer to [65].
Therefore, using I [1r 2) as the optimization objective can achieve latent compression while avoiding representation collapse of 2 the maximum entropy constraint. The compression of imposes the model to extract useful information and discard unpredictable information of the image. Meanwhile, maximizing (y; 2) requires that each y/ could obtain enough information from previous latent Eek to predict Z₁- Each Σς should carry predictable information. These guarantee that the image representation encode rich semantic information aligned with text. We suppose that the above properties learned by the image representation are desired for vision-language pre-training, thus we use Eq. (3) as our pre-training objective. Parameters e and are be jointly optimized under this objective. Intuitively, the vision encoder fo learns to represent images by high-level abstract, and the causal attention model Or learns to compress this high-level abstract the dataset.
3.2 Training Loss
In this sub-section, we demonstrate how Eq. (3) is decomposed into training tasks and losses. Firstly, (3y 2) can be decomposed as a cross-entropy term and an entropy term in the following two symmetric ways (see Appendix B for detailed derivation):
2)
min Ex [H (6 [2x fo(x)))) (EKlyk 90 fo(x)))) H(zx);
(4)
21
1(y;2)= =
min Exep H [yk- = go o fe(x)x) 92 (HulzN = fo(x)))) (yx).
(5)
92
Since given the input æ, latent 2ạ and yı are independent and deterministic (i.e. determined by fo and ga). yielding p(yk, (z) 5 [ fo()))) 5 [yx Is fo(x)) 8[-] is delta distribution. In Eq. (4), p( 2k lyk) is estimated by a parameterized distribution qı (2x (M:). which approximates the distribution of z given the model's prediction yk- Similarly, in Eq. (5), p(3k|Zz) estimated by Qa (x), the predicted distribution of given Therefore, maximizing mutual information can decomposed
follows: 1) The causal attention model 9o learns predict the next latent zy from the output Vk- 2) The learnable latent representation z learns to predict yk which is the representation of its previous context. The maximum entropy regularization avoids the collapse of 2g and yk-
In the following, we show that the cross-entropy terms in T(y; 2) can be achieved by common training tasks and loss functions, while the entropy constraints are implicitly satisfied.
Contrastive Learning between Image Representation and Preceding Context. For image la- tent 2i and the corresponding y/s representing the semantics of its preceding context, the objec- tive defines bidirectional prediction. We choose g as Boltzmann distribution, i.e., (xllv)) x exp(2, W, W2t4/r) and q (34/2*) & exp(v, W. W1*t/T), where T is the temperature, W1 and W2 are learnable linear projections. Consequently, the objective becomes the contrastive loss two directions between ž₁ and yk., when setting 2µ and yı from other images negative samples:
exp(y W. W12+/T)
exp(2, T W.T W2yk/T)
Leon
Σ log Σe exp(y W. W1²N/T) log En exp(2 2 WT W2yk/T) kel
(6)
5

---

Contraddive
Contraddive
Image gichal ripresenation
Aron
Lyon
Preceding context embedding
Image beleniiegresettaaoonn
Mi
oooling
the stret <bdl>>
women riding
bike <Bat>
There +
Aulo-regresive ledt Generation Lost
Vision ncoder
Language da
the
street
<ttol>
<Eot>
woman riding
bike
-dtoi>
Figure 2: Overview of our proposed Latent Compression Learning for vision model pre-training. Image latent representation is extracted via vision encoder and subsequently input into language model alongside textual embedding. Two complementary losses are utilized to learn robust visual representation from scratch on interleaved image-text data: a contrastive loss ensures consistency between the visual latent representation and its preceding context, while an auto-regressive loss enhances the predictability of visual representation for subsequent text.
Meanwhile, the contrastive loss also prevents 2n and yıs from being trivial representation by pulling them away from negative samples, implicitly appending the entropy regularization.
Auto-regressive Text Generation. For text token, its latent code 2 is one-hot vector and is not learnable, so the objective only imposes yı to predict 25 as in Eq. (4). We choose q (3k (bk) as softmax over the output logits on the text vocabulary, i.e, 7 (#x yv ) = 2 softmax (V yk). where V is the projection head of the language model. The objective corresponding to the text tokens is simply the objective of standard next token prediction with cross-entropy loss:
T L.gen
Σ KET
log श softmax (Vyk)
The total training loss defined as L XLrcom I gen where is balancing weight.
Relation to Previous Pre-training Tasks. In our proposed pre-training framework, the contrastive task and generation task align the image representation with preceding context and subsequent context, respectively. Hence, combining these two pre- pre-training tasks can fully leverage the semantic information contained in interleaved image-text data to supervise the learning of image representation. Manguage-Imase Pre-training (CLIP) [55] has
For previous pre- pre-training tasks, 1) Contrastive similar objective of maximizing the mutual information between corresponding image and text [65], but it can only be applied to paired image-text data. 2) Contrastive Captioner (CoCa) [81] combines a captioning (text generation) loss with the CLIP loss, but it cannot be applied to interleaved data, either. CLIP loss requires image-text pairs, while the captioner can only be conditioned on a single image can rather than flexible interleaved image-text contents. 3) Auto-regressive Text Generation task only leverages the semantic information in subsequent context to supervise the learning of image representation, while the preceding context is missing, and representation collapse cannot be avoided. Moreover, interleaved image-text data usually has information redundancy. i.e., the image and its corresponding text may contain similar information. So models may rely on information from the text rather than the image for prediction. This is particularly true when the vision encoder is trained from scratch, resulting that the image representation are never focused and optimized. Our experiments in Sec. 4.2 confirms this analyze.
3.3 Architecture
The overview of model architecture when adopting our LCL is shown Fig. 2. The interleaved image-text input sequence may contain multiple images. We adopt Vision Transformer (ViT) [20] as the vision encoder, which encodes each image into a sequence of visual embeddings as its latent representation. The visual embeddings of each image inserted at corresponding positions in the

---

interleaved sequence, and we introduce special tokens <BoI> and <EoI> to indicate the beginning and the ending positions, respectively. The combined visual and text embeddings are fed into a causal language model. The text generation loss is the standard cross-entropy loss over the output logits of the language model defined in Eq. (7). In the contrastive learning task, calculating the loss for each image token is extremely computationally expensive. To alleviate this problem, we consider utilizing one global representation per image instead of all latent representation for contrastive learning. Specifically, the global representation of each image is extracted from its latent representation R M through an attention pooling, which a multi-head attention layer with learnable query token. The output of the causal transformer model at <BoI> token, just before the image, is processed by a LayerNorm layer and a linear projection into t,. The contrastive loss in Eq. (6) is calculated between 2 and &z-
4 Experiment
4.1 Experiment Settings
Pre-train Data. The datasets utilized in our pre-training encompass the image-text pair dataset LAION-400M [57], as well as the image-text interleaved datasets MMC4 [88] and OBELICS [36]. We also re-organize two datasets for fair comparison with CLIP. 1) dataset. Images from LAION-400M are randomly placed before or after their paired caption to form image-text sequences. 2) MMC4-Pair paired dataset. For images in MMC4, we select matching text from the bipartite graph matching results derived from CLIP similarity to generate pseudo image-text paired data.
as interleaved
Implementation Details. We adopt the same image transform in OpenCLIP [15] for pre-training and set the image size as 224 x 224 for all experiments. We employ ViT [20] as our vision encoder. ViT-L/14 used for the main results, and ViT-B/16 used for ablation studies. The language model follows the same architecture contrastive loss balancing weight is set at = 0.1. AdamW optimizer with 31 = 0.9, 82 = 0.95 and weight decay of 0. are used. We employ cosine learning rate schedule with linear warmup and set the peak learning rate at 5c-4 for the LAION data and 3e-4 for others. The model for the main results is trained for 200k iterations with 64k images per batch on average. In the ablation study, the models are trained for 250k iterations with 8k images per batch.
OPT-125M [84] but is randomly initialized. By default, the
Vision Encoder Evaluation. CLIP pre-trained models typically uses zero-shot retrieval performance for evaluation. However such zero-shot inference on retrieval tasks [55] is not suitable for other generative pre-training methods. To address this discrepancy, we choose to evaluate pre-trained vision models by transfer learning on various downstream tasks under two configurations: "frozen transfer" and "full transfer". In the frozen transfer setting, only the parameters outside of the vision model are trained. In the full transfer setting, all parameters are trained.
The evaluation is conducted on image classification, image-text retrieval, image caption, and multi- modal dialogue. Please see Appendix A for more data, implementation, and evaluation details.
4.2 Comparison with Vision Pre-training Methods
In this section, we show the superiority of our LCL when using interleaved image-text data for vision pre-training. However, existing vision pre-training methods only support paired data, so we choose MMC4 dataset with a paired set (MMC4-Pair) available for those methods. In addition, to confirm that it is is not trivial to use interlcaved data for vision pre- pee-training, we also consider existing methods to use interleaved data. Those methods are originally proposed MLLM training methods, but here we test them for vision pre-training. Besides, on paired image-text data, we also compare our LCL with existing vision pre-training methods and show they are comparable. See Appendix A.3 and A.2 for detailed pre-training and evaluation settings.
We aggregate all the pre-training methods divide them into the following tasks: (1) Image-text contrastive (Con.) [55, 15, 32], (2) Image-text contrastive + image captioning (Con. Cap.) [81], (3) Image-text contrastive image captioning + image-text matching (Con. + Cap. Mat.) [40, 41, 14], (4) Auto-regressive text generation (Gen.) [2, 29, 44], (5) Auto-regressive text generation + image regression (Gen. + Reg.) [69, 68, 72]. (6) Mask data modeling (Mask.) [78]. We select one
7

---

Table Frozen transfer evaluations of vision models pre-trained on the MMC4 dataset. Vision models are pre-trained from scratch for all methods. "IN-Ik" denotes image classification on ImageNet [33]. "ret." and "cap." denote image-text retrieval and image captioning, respectively. * The method names refer to implementing those methods with our experiment setting but not their trained checkpoints. For pre-trainig tasks, Con. image-text contrastive; Cap. image captioning, Mat. image-text matching; Mask. mask data modeling; Gen. auto-regressive text generatiton: Reg. image (feature) regression. The names in parentheses refer to the pre-training task but not their trained checkpoints. Note that CoCa and BLIP2 need to pass each sample through the language model 2 and 3 times, respectively, to perform multi-task learning.
Pre-training method (task)
Pre-training IN-Ik data
COCO ret.
Flickr30k ret.
COCO cap.
NoCaps cap.
acc-1
TR @
IR@)
TR @1
IR @
B@4
C
B@4
C
CLIP (Con.)
MMC4-Pair
74.8
46.4
32.5
76.2
60.0
23.9
82.9
29.6
78.4
CoCa (Con. + Cap.)
MMC4-Pair
75.4
48.6
34.3
76.5
61.9
23.7
84.8
30.0
80.5
'BIIP (Con. Cap. Mat.)
MMC4-Pair
74.5
46.5
31.3
74.9
57.8
23.7
82.9
29.4
78.1
BHTT3 (Mask.)
MMC4
73.3
45.1
30.6
73.2
57.1
23.3
81.4
29.5
76.7
Flamingo (Gen.)
MMC4
24.0
10.6
5,6
17.7
10.8
8.7
18.1
15.0
18.5
Emu (Gen. Reg.)
MMC4
5.7
2.3
1.4
4.8
2.7
0.3
4.4
0.6
4.6
LCL (Ours)
MMC4
75.2
48.5
34.5
76.3
60.4
24.4
87.5
31.0
82.5
Table 2: Frozen transfer evaluations of vision models pre-trained on LAION dataset. Vision models are pre-trained from scratch for all methods. 'hee method names refer to implementing those methods with our experiment setting but not their trained checkpoints. f Note that CoCa and BLIP2 need to pass each sample through the language model 2 and 3 times, respectively, to perform multi-task learning.
* Pre-training method (task)
Pre-training data
IN-Ik
COCO
Flickr30k
COCO cap.
NoCaps cap.
acc-1
TR@1
IR@I
TR@1
IR
B@4
C
B@4
C
CLIP (Con.)
LAION-400M
75.0
47.2
34.2
76.5
59.8
24.1
84.1
30.0
78.8
CCoC (Con. + Cap.)
LAION-400M
75.2
48.6
34.8
76.9
61.4
24.6
88.2
30.4
82.9
'BLIP2 (Con, Cap. Mat.)
LAION-40MM
74.0
47.4
31.5
75.9
57.9
23.6
85.0
30,0
77.8
I.C.. (Ours)
LAION-Random
75.1
48.3
34.3
76.8
59.6
24.4
88.1
31.3
84.2
representative method from each task. For fair comparison, we implement these methods on the same model and training data with available open-source codes our reproduction.
Pre-training Interleaved Image- Text Data. We conduct experiments on the MMC4 dataset [88] to demonstrate the effectiveness of our LCL on interleaved image-text data. For pre-training methods that only support paired data, MMC4-Pair used.
The results are shown in Tab. 1. Our LCL pre-training method significantly outperforms all other methods in the caption tasks, indicating that we effectively utilize the rich text context information in MMC4 interleaved data. On the other hand, our method is on par with the best paired pre-training methods classification and retrieval tasks. Since the paired pre-training methods directly optimized for retrieval, our comparable performance shows that the visual features are learned to be general interleaved data, where no paired
highly distinguishable. It is worth mentioning that for versions exist, these paired pre-training methods cannot be applied.
In addition, we observed that the two methods using auto-regressive text generation do not achieve good performance and feature collapse occurs. However, their text prediction training loss is actually close to ours. This suggests that these methods tend to rely redundant text information rather than image information for subsequent text prediction. As discussed in Sec. 3.1, our approach can avoid such collapse.
Pre-training on Paired Image-Text Data. show that LCL pre-training also performs well on paired image-text data without specific modification. Tab shows that our method is comparable to paired pre-training methods on various tasks, indicating that all information in paired data is fully exploited.
We conduct experiments on the LAION-400M dataset to
8

---

Table Transfer evaluation results of pre-trained ViT/L-14 on classification, retrieval and captioning
tasks.
Pre-training data
Pre-training epoch
IN-IK
COCO
Flickr30k
COCO cap.
NoCaps cap.
Model
acc-1
TR@1
IR@1
TR@1
I IR@1
B@4
C
B@4
C
frozen transfer
OpenAl CLIP
WIT-400M
32
83.7
61.7
48.2
89.0
75.8
116.0
35.5
108.9
OpenCIIP
LAION-400M
32
82.1
59.5
46.0
86.9
31.0
111.5
34.8
106.0
LCL (Ours)
LAION-400M
32
82.2
59.6
46.2
86.7
74.0
31.4
112.3
35,0
106.7
LCL (Ours)
LAION- MMC4
16
82.0
60.0
46.0
87.6
74.6
32.0
113.7
35.1
107.1
full transfer
OpenAl CLIP
WIT-400M
32
87.4
62.1
49.6
90.3
77.9
39.5
132.7
41.4
116.9
OpenCLiP
LAION-400M
32
86.2
61.7
47.5
87.7
76.3
38.6
128.9
39,9
LCL (Ours)
LAION-400M
32
61.9
47.5
87.6
76.1
129.7
40.2
113.5
LCL (Ours)
LAION-4000 MMC4
16
86.1
62.2
47.6
88.1
76.2
39.5
130,9
40.6
113.8
Table 4: Transfer evaluation results of pre-trained ViTYL-14 multi-modal benchmarks. The transfer
adopts the downstream model and training pipeline LLa [46].
Model
Pre-training data
Pre-training epoch
VQAv2
GQA
VisWiz SQA
POPE
MME
MMB
SEED,
frozen transfer
OpenAl CLIP
WIT-400M
77.1
61.7
44.4
71.1
84.6
1486.9
65.1
64.6
OpenCLIP
LAION-400M
68.7
$7.0
39.5
69.0
81.8
1266.2
54.2
55.8
LCL (Ours)
LAION-400M MMC4
70.7
57.4
41.4
69.7
81.7
1291.7
55.3
56.5
full transfer
OpenAl CLIP
WIT-400M
32
79.0
62.8
46.8
71.8
85.7
1576.8
68.9
67.9
OpenCLIP
LAION-400M
32
71.5
58.6
42.2
70.4
82.5
1345.4
58.5
59.5
LCL (Ours)
LAION-400M MMC4
73.4
58.8
44.2
71.0
82.5
1382.3
59.5
60.3
4.3
Comparison with Pre-trained Checkpoints
To further confirm the effectiveness of proposed Latent Compression Learning (LCL), compare our pre-trained model with existing checkpoints of pre-trained vision encoders. We use LCL to pre-train ViT-L/14 with mixed data from the LAION-400X and MMC4. We compare it to the ViT-L/14 pre-trained by OpenCLIP [15] using the public LAION-400 -400M dataset, and the ViT-L/14 pre-trained by OpenAl CL.IP [55] with private data is listed reference. The total number of images seen during pre-training is 13B for all models. We evaluate the pre-trained vision encoders by transferring downstream tasks, i.e., integrate the vision encoders into downstream task models and compare the fine-tuning results. More training details are in Appendix A. and evaluation details are in Appendix A.2.
Tab. and Tab. show the results of transfer evaluations. When both use LAION-400M as pre- training data, with the previous experimental conclusions, LCL has similar performance CLIP. When combined with MMC4, our method achieves better performance, especially on caption and multi-modal dialogue tasks.
There exist approaches achieving better results on some henchmarks. However, they either use larger vision encoders, more training data, or private data. We list those results reference in Appendix C.
4.4 Ablation Study
Latent Compression Learning on Different Datasets. We apply LCL pre-training to more datasets to confirm its generalizability. As shown in Tab. 5, our method also achieves reasonable performance on interleaved dataset OBELICS. is worth noting that the models trained on MMC4 and OBELICS have achieved similar performance to that on LAION, indicating that it is completely feasible to pre-train visual models only from interleaved data. Furthermore, using both LAION and MMC4 data during pre-training improves performance, suggesting that further improvements can be obtained by incorporating more image-text data. In this case, supporting interleaved data is key advantage of approach, enabling the use of more diverse image-text data for pre-training.

---

Table Frozen transfer evaluations LCL pre-training on different datasets. We ensured that all entries had seen the same number of images during pre-training to ensure faimesss.
Pre-training method
Pre-tree trainini
IN-1k
COCO ret.
Flickr30k ret.
COCO cap.
NoCaps cap.
data
acc-1
@
@
TR@1
IR@1
B@4
C
B@4
LAION
75.1
48.3
34.3
76.8
59.6
24.4
88.1
31.3
84.2
MMC4
75.2
48.5
34.5
76.3
60.4
24.4
87.5
31.0
82.5
LCL (Ours)
Obelies
73.9
47.0
33.2
75.1
58.7
23.3
84.8
29.9
77.6
LAION + MMC4
75.8
49.4
35.3
77.7
61.1
25.1
88.9
31.4
84.9
Table 6: Ablations of the training loss and loss balancing weight in LCL. Models are evaluated under frozen transfer setting.
(a) Training loss ablation.
(b) Loss balancing weight ablation.
COCO ret.
COCO cap.
Con. weight
COCO Ret.
COCO Cap.
Training loss
X
@
IR@1
B@4
CIDEr
TR@1
B@4
CIDEr
Con. only
46.4
32.7
23.8
82.7
0.05
48.3
33.7
24,3
87.0
Gen. only
10.3
5.4
8.5
17.8
0.1
48.5
34.5
24.4
87.5
L.C.. 48.5
34.5
87.5
0.2
47.6
33.1
24.2
86.3
0.5
37.1
23.2
20.4
66.7
Loss Balance in Latent Compression Learning, Table 6a ablates the contrastive loss and generation loss used LCL. Consistent with the previous analyses, LCL can achieve the best performance. Table 6b studies the appropriate loss balancing weights (multiplied by the contrastive loss). It turns that À 0.1 will produce the best results. The performance drops significantly for larger values, indicating that the optimization directions of the two losses are not completely consistent.
5
Conclusion
No existing work has explored vision model pre-training with interleaved image-text data, To this end, propose Latent Compression Learning (LCL) framework that compresses interleaved image-text latent for vision pre-training. We theoretically show that latent compression is equivalent to maximiz- ing the mutual information between the input and output of causal model and further decompose this objective into two basic training tasks. Experiments demonstrate that our method comparable to CLIP on paired pre-training datasets, and it effectively learns robust visual representations utilizing interleaved image-text data. Our work showcases the effectiveness of using interleaved image-text data to learn robust visual representation from scratch, and confirms the potential of compression learning for visual pre-training.
Limitations. Our experiments are constrained to limited size of dataset and vision encoder, and the scaling property our proposed method remains unexplored.
Acknowledgments.
This work is supported by the National Key R&D Program of China (NO. 20222D0161300),, by the National Natural Science Foundation of China (62376134).
10

---

References
[1] H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and Anderson. Nocaps: Novel object captioning at scalc. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948 0951-8957777 2019.
[2] J.BB. Alayrac, Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, Lenc, A. Mensch, Millican. M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information pmoessing systems, 1--23766-,3736, 2022.
[3] C. Alberti, Ling, M. Collins, and D. Reitter. Fusion of detected objects in text for visual question answering, arXiv preprint arXiv: 05054, 2019.
[4] A. Awadalla, L. Gao. J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint
2...0038..1199 2023.
[5] J. Bai, S. Bai, S. Yang. S. Wang. S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-languas model for understanding, localization, text reading, and beyond. 2023.
[6] H. Bao, L. Dong, S. Piao, and F, Wei. Beit: Bert pre-training of image transformers. arXiv preprint miXiv:2106.08254 2021.
[7] N. Carion, F. Massa, G. Synnaeve. N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229, Springer, 2020.
[8] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail tail visual concepts. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3558 3568, 2021.
[9] L.-C. Chen, G. Papandreou, L Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected erfs. arXiv preprint arXiv:1412.7063 2014.
[10] L.-C Chen, G. Papandreou. I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4))834-88)) 848, 2017.
[11] X. Chen, H. Fang, T.-Y. Lin, Vedantam. Gupta, Dollár, and C. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 504.00325, 2015.
[12] X. Chen, M. Ding, X. Wang, Xin, S. Mo. Y. Wang. S. Han, P. Luo, Zeng, and Wang. Context autoencoder for self-supervised representation learning, International Journal of Computer Vision, 132(1)) 208-223 2024.
[13] Y.C. Chen, L. L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer, 2020.
[14] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic lingu-iin tasks. arXiv preprint arXiv:2312.144 14238, 2023.
[15] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco. C. Gordon, C. Schuhmann. L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18-899,,, 2023.
[16] H. W. Chung. L. Hou. S. Longpre, B. Zoph. Y. Tay, W. Fedus, Y. Li, X. Wang. M. Dehghani, S. Brahma. et al. Scaling instruction finetuned language models. Journal of Machine Learning Research, 25(70):1-3,, 2024.
instraction-finetueed
[17] Dai, J. Li, Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, N. Fung, and S. Hoi. Instructblip: general-purpose
Towards general- purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[18] G. Delétang, A. Ruoss, P.A. Duquenne, E. Catt, T. Genewein, C. Mattem, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309, 10668, 2023.
[19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint 1rxiv:1810.04805. 2018.

---

[20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010.
2020.
[21] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong. T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint vrxiv:2303.03378 2023.
[22] Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang. X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceddiggs of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[23] Y. Ge, S. Zhao, Z. Zeng, Y. Ge. C. Li, X. Wang, and Y. Shan. Making llama see and draw with seed tokenizer. arXiv preprint 11010221,, 2023.
[24] R. Girshick, J. Donahue, T. Darrell, and Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.
[25] T. Gong, C. Lyu. S. Zhang, Y. Wang. M. Zheng, Q. Zhao, K. Liu, W. Zhang, P. Luo, and K. Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint 2rXiv:2305.04790 2023.
[26] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 1904-6913 2017.
[27] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[28] J. Hernández-Oralln and N. Minaya- Hinaya-Collado, A formal definition of intelligence based on an intensional variant of algorithmic complexity. In Proceedings of International Symposiun of Engineering of Intelligent Systems (EIS98), pages 146-163, 1998.
[29] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36, 2024.
[30] Y. Huang. J. Zhang. Z. Shan, and J. He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937, 2024.
[31] M. Hutter. The human knowledge compression prize. URL http://prize. hutterl. net, 2006.
[32] C. Jia, Y. Yang. Y. Xia, Y--T. Chen, Z. Parekh, H. Pham, Q. Le, Y.HH. Sung. Li, and T. Ducrig, Sealing up visual and vision-language representation learning with noisy text supervision. International conference on machine learning, pages 490449966. PMLR, 2021.
[33] A. Krizhevsky, Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems, 25, 2012.
[34] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[35] G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.0700 076488 2016.
[36] H. Laurençon, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang. S. Karamcheti, A. Rush, D. Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
[37] S. Legg and M. Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines, 17:391-444, 2007.
[38] S. Legg. M. Hutter, et al. A universal measure of intelligence for artificial agents. In International Joint Conference on Artificial Intelligence, volume 19, page 1509. LAWRENCE ERLBAUM ASSOCIATES LTD,, 2005.
[39] G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang. Unicoder-vl: A universal encoder for vision and language by modal pre-training. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11336 11 1344, 2020.
12

---

[40] Li, D. Li. C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In International conference machine learning, pages 12888- 12900. PMLR, 2022.
[41] Li, D. S. Savarese, and S. Hoi. Blip-2: Bootstrapping angguage-imgge pre-training with frozen image enenders and large language models. International conference on machine learning, pages 19730-19742. PMI.R, 2023.
[42] L. H. Li, M. Yatskar, D. Yin, C.-. Hsieh, and K.W. Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 03557, 2019.
[43] Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-iman pre-training via masking. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390- 23400, 2023.
[44] Lin, H. Yin, Ping, Y. Lu, P. Molchanov, Tao, Mao, Kautz, M. Shocybi, and S. Han. Vila: On pre- pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2023.
[45] Liu, C. Li, Li, and J. Lec. Improved baselines with visual instruction tuning. arXiv preprint axivv23300003744 2023.
[46] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[47] Y. Liu, H. Duan. Y. Zhang, B. Li, S. Zhang. W. Zhao, Y. Yuan, J. Wang, C. He. Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 2rXtv:2307.06281, 2023.
[48] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2015.
[49] J. Lu, D. Batra, D. Parikh, and Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for iision-and-languaal tasks. Advances neural information processing systems, 32, 2019.
[50] D. Mahajan, R. Girshick, V. Ramanathan, He, M. Paluri, Y. Li, Bharambe, and L. Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018.
[51] M. V. Mahoney.
Text compression as test for artificial intelligence. AAAI/AAAI, 970, 1999.
[52] R. C. Pasco. Source coding algorithms for fast data compression, PhD thesis, Stanford University CA,
1976.
[53] Z. Peng. W. Wang. L. Dong, Y. Hao, S. Huang, S. Ma. and F. Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306.1488 14824, 2023.
[54] B. A. Plummer, L. Wang. C. M. Cervantes, C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region- to- oo-phaase correspondences for richer image- sentence models. Proceedings of the IEEE international conference on computer vision, pages 2641 211122649,0 2015.
[55] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages $748-8763. PMLR, 2021.
[56] J. J. Rissanen. Generalized kraft inequality and arithmetic coding. IBM Journal of research and develop- ment, 20(3):198 203, 1976.
[57] C. Schuhmann, R. Veneu, R. Beaumont, R. Kaczmarezyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
[58] C. Schuhmann, A. Köpf, R. Vencu, Coombes, and R. Beaumont. Laion coco: 600m synthetic captions from laion2h-en. URL https://aaani. ai/hlog/laion-ooo,, 2022.
[59] C. E. Shannon. A mathematical theory of communication The Bell system technical journal, 27(3): 379-423, 1948.
[60] C. E. Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1):50 64. 1951.
13

---

[61] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed. image alt- text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 2556-2565, 2018
[62] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[63] M. Singh, L. Gustafson, A. Adcock, V. de Freitas Reis, B. Gedik, R. P. Kosaraju. D. Mahajan, R. Girshick, P. Dollár, and L. Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 804-814, 84, 2022.
[64] Su, X. Zhu, Y. Cao, B. L., Lu, F. Wei, and Dai. VI-bert: Pre-training of generic visual-linguisee representations. arXiv preprint (rXXiv:908,0883300 2019.
[65] W. Su. X. Zhu, C. Tao, L. Lu, B. Li, G. Huang, Y. Qiao, X. Wang. J. Zhou, and J. Dai. Towards all-in-one pre- pre-training via maximizing multi-modal mutual information. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15888-15899 2023.
[66] C. Sun, F. Baradel, K. Murphy, and C. Schmid. Learning video representations using contrastive bidirec- tional transformer. arXiv preprint arXiv: / 1a0../9963..33 2019.
[67] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473, 2019.
[68] Q. Sun, Y. Cui, X. Zhang. F. Zhang. Q. Yu, Z. Luo, Y. Wang. Y. Rao, J. Liu, T. Huang, et al. Generative multimodal models are in-context learners. arXiv preprint 1rXiv:2312.1326,, 2023.
[69] Q Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang. Y. Wang. H. Gao, J. Liu, T. Huang, and X. Wang. Emu: Gener- ative pretraining in multimodality. In The Twelfth International Conference on Learning Representations. 2023.
[70] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint orXiv:9908 07490, 2019.
[71] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73 73, 2016.
[72] C. Tian, X. Zhu, Y. Xiong, W. Wang, Z. Chen, W. Wang, Y. Chen, L. Lu, T. Lu, J. Zhou, et al. Mm- interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401. 2024.
[73] H. Touvron, M. Cord, A. Sablayrolles, G. Synnacve, and H. Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 32 42, 2021.
[74] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23071, 2023.
[75] H. Touvron, I.. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint a7X 2023.
[76] A. Veit, M. Nickel, S. Belongie, and L. Van Der Maaten. Separating self- extresxprssionn and visual content in hashtag supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1919-5927 2018
177] F. Y. Vicente, L. Hou, C.-P. Yu, M. Hoai, and D. Samaras. Large-scale training of shadow detectors with noisily- shadow examples. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 816-832, Springer, 2016.
[78] W. Wang. H. Bao, L. Dong. J. Bjorck, Z, Peng. Q. Liu, K. Aggarwal, O. K. Mohammed. S. Singhal, S. Som, et al. Image a foreiga language: Beit pretraining for all vision and lision-language tasks. urXiv preprint arXiv:2208. 100.22442,. 2022.
[79] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418 434, 2018
14

---

[80] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao. Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In Proceedings the IEEE/CVF conference computer vision and puttern recognition, pages 9653-9663, 2022.
[81] J. Yu. Z. Wang. V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, Coca: Contrastive captioners image-text foundation models. arXiv preprint (rXiv:2005001917.22 2022.
[82] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104 12113,, 2022.
[83] P. Zhang. X. D. B. Wang, Y. Cao. C. Xu, L. Ouyang, Z. Zhao. S. Ding, S. Zhang, H. Duan, H. Yan. et al. intepon-xcomppser: vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 151 2023.
[84] S. Zhang, S. Roller, N. Goyal, M. Artetxe, Chen, Chen, Dewan, M. Diab, Li. Lin, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0 010068, 2022.
[85] Н. Zhao, Cai, Si, Ma, K. An, L. Chen, Liu, Wang. Han, and B. Chang. Mmicl: Empowering vision-language model with multi-modal in-context leamning, arXiv preprint arXiv:2309. 00.019999.7........ 2023.
[86] Zhu, Chen. X. Shen, with advanced large language models. arXiv preprint urXiv:2344.10520 2023.
and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding
[87] Zhu, X. Ding, Ge, Ge, Zhao, H. Zhao, X. Wang, and Shan. Vl-gpt: generative pre-trained transformer for vision and language understanding and generation. arXiv preprint arXiv:2312.09251, 2023.
[88] W. Zhu, J. Hessel, A. Awadalla, S. Y. Gadre, J. Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang, and Y. Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
15

---

A Experimental Details
A.1 Pre-training
Data. For the data in MMC4, we select images with a CLIP similarity the matching text of 0.24 or higher. From documents containing at least one such image, we randomly choose up to 6 images to form interleaved image-text sequence, utilizing all text from that document. If the sequence length exceeds 2048 tokens, the surplus is truncated while ensuring the integrity of both images and individual text segments, and then padded the designated length. For OBELICS, we similarly restrict the number of images per document to between I and 6. We then sequentially extract 2048 tokens from the concatenated documents. If image tokens are truncated, the entire image is moved to the next sample sequence.
To construct interleaved image-text samples from the MMC4 dataset, we randomly place images either before or after their corresponding sentences, adhering to 50% probability, thus generating document-wing interleaved sequence of images and text. For the OBELICS corpus, individual documents are concatenated, and a sliding window strategy is employed to select each image-text sequence, maintaining a total length of 2048 tokens.
Syper--arammeers. Our pre-training configuration is shown in Tab. The Adam optimizer was employed for model training with the learning rate set to 3e-4 and the weight decay set 0.1. Mixed numerical precision training with 6 is also employed to stabilize the optimization process. Furthermore, set drop-path [35] rate linearly increasing 0.2, and use layer-scale [73] for stable training.
Table 7: Hyper-parameters in pre-training.
optimizer
AdamW
learning rate
3e-4
weight decay
0.1
optimizer momentum Ir schedule
$1, = 0.9, 0.95
cosine decay
warmup
2k
numerical precision
bfloat16
train steps batch size (in images)
20k
64k
drop path
0.2
A.2 Evaluation
Transfer Tasks. We conduct our performance evaluation of pre-trained on image classification, image-text retrieval, text generation tasks with multimodal inputs (i.e., image captioning and multi- modal dialogue). Their model architecture in transfer learning are illustrated in Fig. 3.
For closed-set image classification, a lightweight classifier with a randomly initialized attention pooling layer, followed by layer normalization layer and linear layer, is appended to the top of the pre-trained vision model. the "frozen transfer" scenario, only the parameters of the added classifier are trainable, similar to the linear probing strategy. Conversely, in the "full transfer" approach, all parameters, including those of the pre-trained vision encoder, are adjustable.
Regarding the image-text retrieval task, we discard the pre-trained text encoder and apply contrastive learning to the pretrained vision encoder with a newly introduced through the vision encoder and a randomly initialized attention pooling layer to generate global image embedding. Textual captions are processed through the text encoder, utilizing the feature the final token as the text embedding representing the input caption. An additional linear layer facilitates the dimensional alignment between the image and text embeddings, enabling their use contrastive learning. During "frozen transfer", the attention pooling layer on the vision transformer and the text encoder are trainable; similarly, "full transfer". all parameters, including the vision encoder, can be optimized.
encoder. Images are processed
16

---

Class Logit
Similarity
Linear
Text
Attn Probe
Attn Probe
Linear
MLP
Vision Encoder
Vision Encoder
Text Encoder
Vision Encoder
Language Model
Image
Image
Text
Image
Text
(a) Image Classification
(b) Vision-Language Retricval
(c) Text Generation
Figure 3: Illustration of "frozen transfer" evaluation. The vision encoder is frozen during transfer
tuning. (a) Image classification: an attention probe and a linear classifier are built upon the vision encoder. (b) Image-text retrieval: an attention probe used extract global visual feature, which is trained to align with the text feature from the text encoder. (c) Text generation: an MLP is utilized to align the visual feature with the text embedding space, and the multi-modal embedding is fed into the language model for auto-regressive text generation.
For text generation tasks with multimodal inputs, we adopt image captioning and multi-modal dialogue benchmarks and employ prevalent architectures like those in [46]. Specifically, a pretrained LLM for text generation is integrated on top of the pre-trained vision model, incorporating an MLP layer to adjust the dimensions of the visual embeddings. During the "frozen transfer" evaluation, the parameters of the vision model remain fixed, while in the 'unfreezing' phase, these parameters are permitted to undergo training.
Implementation Details. Vision encoder evaluation with "frozen transfer" and "full transfer" con- figurations includes fine-tuning training and benchmark evaluation. Implementation details of each transfer task are list below, and the pyper-parameters involved are listed in Tab. 8.
Image classification. Model is trained on the ImageNet- K [33] train split and evaluated on val split. We follow the attention probe setting introduced by [12] for "frozen transfer". and the full fine-tune setting in [43] for "full transfer".
Image-text retrieval. Model is trained on a combination dataset comprised of CC12M [61], CC3M [61], and SBU [77], and is tested on the MSCOCO [11] karpathy-lest split and Flickr30k [54] test split. The model is trained with Adamw optimizer for 5000 iterations. The learning rate is set at 1e-3 and le-5 for parameters without initialization and with initialization, respectively.
Image captioning. Model is trained on a subset the LAION-COCO [58] dataset, which includes 10 million samples, and evaluation is performed on the MSCOCO [11] karpathy-test split and NoCaps [1] val split. Here, the model is trained for 20,000 iterations with a learning rate of le-4, Additionally, droppath technique is employed with ratio of 0.2 in the vision model to mitigate overfitting.
Multi-modal dialogue. We follow a two-stage training process similar to that used in LLAVA- 1.5 [45]. Initially, paired data with 558K samples is used train an MLP projector to align the Vision Transformer (ViT) with the pretrained LLM. Subsequently, the model undergoes instruction tuning on multimodal dialogue datasets with 665k samples. Both the alignment training and instruction tuning phases are conducted over a single epoch, with learning rates set at le-3 and 2e-5, respectively. Evaluations then performed on multimodal dialogue benchmark, e.g., MMBench [47] and VQAv2 [26].
A.3 Ablation Experiments
The effectiveness of our LCL are validated by conducting ablation experiments mainly on two corpora: LAION and MMC4. The experimental hyper--prameee involved are shown in Tab. We found that the optimal learning rate for the LAION dataset is 5c-4, while for the MMC4 dataset, slightly lower
17

---

Table Hyper-parameler in transfer evaluation.
Multi-modal dialogue
paramerr
Classification
Retrieval
Image Captioning
stagel
stage2
train dataset
IN-IK train
CC12M, CC3M, SBU
LAION-COCO
LCS-558K
LLaVA-SFT
test dataset
IN-IK val
COco.Flickr30k
COCO.Nocapp
MMbench, VQAv2, GQA...
optimizer
AdamW
learning rate
le.4
le-3
le-4
le-3
2e-5
weight decay
le-4
optimizer momentum
81 82
0.9, 0.95
learning rate schedule
cosine decay
warmup
1500
500
1000
30
156
train steps
14k (90ep)
5k
20k
1091
5198
batch size (in images)
8192
16k
512
512
128
rate of 3e-4 proves most effective. We speculate that this because MMC4 corpus contains relatively higher noise. Most of the original settings in the large-scale pre-training are retained in the ablation, with the exception of reducing the batch size by factor to decrease the computational overhead.
Table 9: Hyper-parameters ablation study.
Hyper-parameter
LAION MMC4
optimizer
AdamW
learning rate
5e
3e-4
weight decay
0.1
optimizer momentum
B1, 32 = 0.9, 0.95
learning rate schedule
cosine decay
warmup
2k steps linear
numerical precision
b
train steps
25k
batch size (in images)
8k
drop path
0.2
A.4 Experiments Compute Resources
Pre-training used 512 A800 GPUs and took 5 days.
B Theoretical derivation details
Using the notation defined in Sec. 3, the mutual information of the output of the language model (1)) and the latent representation (z) can be described as follows:
N
2
[yy: 2x)
1=1
N
pluns Rx)
pluu: =x) log
dya dzz
(8)
pluk )(()))
k=1
Ezap [ plue x))p(z 1x) log
P(Uk, 26)
dyk dzy
(9)
p(y))p(2x)
We can derive Eq. 9 from Eq. 8 because once the interleaved image-text input sequence & is given, output yh and the latent representation 2μ can be computed independently:
p(yk, 2k
x))
p(ux x)p(2t x))
18

---

From Eq. 9, we can further decompose the mutual information into cross- erosopy component and an entropy component two symmetric ways. One approach involves the output token yv predicting the next latent representation along with the entropy of my:
Σ Buuu |f plun x)p(zk x)) log
p(zk Uk) p(zx)
T(y;
dyt des
(10)
k=1
Σ Ezup 1Ō(2z = fo(x))) log P(zk 90 o fo(x)))] / logp(ze)
(11)
The other approach considers how the latent representation z approximates the previous context and includes entropy of the output yı:
N
|f plue x)p(zz æ) log
p(uK 2x) dyn dizk plyx)
Ewwp
(12)
k=
N
(8(Uk 96 fo(())) log P(ys | =s so(x)a)) - -f plun) logp(u)) (13)
Exp
k=1
Note that the reason for transitioning from Eq. 10 to Eq. 11 and from Eq. 12 to Eq. 13 is because, given the input sequence x, the outputs yı and 2g are determined as follows:
plyk (x) 5 [yk 94 (o(x)))
p(zz x) = 5 [a [=()))
C Supplementary Benchmark Results
We present supplementary results on classification, retrieval and captioning tasks (Tab, 10) and multi-modal benchmarks (Tab. 11).
D Broader Impacts
This work may share the common negative impacts large-scale vision training. The data used in pre- training may contain dataset bias, and raise ethical concerns. It may also require large computational resources, which consume lots of electricity and result increased carbon emissions.
19

---

Table 10: Supplementary transfer fer ovaluation results on classification retrieval and captioning tasks. Reproducing using open source code. "Repro. CoCa"; reproduced CoCa. Results with larger vision encoders, more training data, or private data are grayed out as reference.
finetune CIDEr
COCO cap.
129.7
130.9
128.9
132.7
143.6
COCO ret.
frozen IR @R@
34.3
34.2
34.8
34.5
30.6
5.6
1.4
46.2
46.0
46.0
48.2
COCO ret. frezen finctume frozen TR
48.3
47.2
48.6
48.5
45.1
10.6
2.3
59.6
60.0
59.5
61.7
IN-1k IN-
86.1
86.
86.2
87.4
91.0
75.0
75.0
75.2
75.2
73.3
24.0
5.7
82.2
82.0
82.1
83.7
90.6
Data
public
Comparison with MILMM training methous (used for vision training) interlaved data
Seen samples
2B
2B
2B
2B
2B
2B
2B
13B
13B
13B
13B
33B
LAION LATON-40OM 400M + MMC4
Pre-truining data
JFT-3B + ALIGN
Comparison with existing vision pre- pre-traiin traininiiggggg methods on puired data
LAION 4004
LAION MION-400M 400M
LAION-400M 400M
Comparison with existing pre trained checkpoints OF reported results VIT-L (304M)
LAION 400M
LATON-400M
WIT-400M
MMC4
MMC4
MMC4
MMC4
Vision encoder
ViT-B (86M)
ViT-B (86M)
ViT-B (86M)
ViT-B (86M)
ViT-B (86M)
VIT-B (86M)
VIT-B (86M)
ViT-L (304M)
VIT-L (304M)
VIT-L (304M)
VIT-G (1B)
training method interleuve data
Support
V
"OpenFlamingo
Vision pwe-
*Repro. CoCa
OpenAl CLIP
LCL (ours)
*OpenCLIP
LCL (ours)
LCL (ours)
LCL (ours)
OpenCLIP
"BETT3
*Emu
CoCa
20

---

Table 11: Supple eelentenrry results of multi multi-modal modda benehmarks i Training data of the teacher model is included. Approaches are grayed out as reference if they use stronger vision encoders or LLM, more training data, or private data.
MME MMB SEED)
56.5
55.8
64.6
65.4
70.2
75.9
76.0
55.3
54.2
65.1
64.3
67.4
79.6
77.6
82.2
1291.7
1266.2
1486.9
1510.7
1519
1712
1487.5
1525.1
1637
POPE
81.7
81.8
84.6
85.9
86.5
87.9
86.4
88.3
VizWiz SQA
69.7
69.0
71.1
66.8
70.1
91.2
94.0
41.4
39.5
44.4
50.0
57.6
38.9
38.3
54.9
52.5
63.5
VQAv2 GQA
57.4
57.0
61.7
62.0
64.2
57.5
46.0
65.1
62.9
65.7
70.7
68.7
77.1
78.5
81.8
78.2
62.0
84.9
82.3
79.3
MLLM data public
MLLM training samples
1.2M
1.2M
1.2M
1.2M
1.3M
~00
1.5B
83M
160M
11.5
1.0B
~55
LLM
size
7B
7B
7B
7B
7B
78
7B
13B
33B
7B
7B
20B
MLLM method
LLAVA-NEXT
InterLLM-XC2
Qwen- VL-Chat
CogVLM-Chai Chat
InterVLL 1.5
LLaVA-1.5
LLaVA-13
LLaVA-155
LLaVA-155
Emu2-Chi Chat
InterVLL
Emn-1
Pre Pre-training Pre-truining data public
V
samples
13B
13B
13B
13B
138
34B
28B11
288
~2881
~99
~929
VIT-L-336 (304M)
VIT-L-336 (304M)
Vision encoder
VIT-L (304M)
VIT-L (304M)
VIT-L (304M)
Advanced MLLMs as reference
VII-L (304M)
VIT-L (304M)
VIT-E (4.4B)
VIT-E (4.4B)
VIT-G (1B)
Our transfer evaluation
VIT-6B
VIT-6B
Vision pre- training method
OpenAl CLIP
OpenAl CLIP
OpenAl CLIP
OpenAI CLIP
LCL (ours)
OpenCLIP
OpenCLIP
EVA-CLIP
EVA-CLIP
EVA -LIP
InterViT
Inter
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Sec I.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Sec 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Please see Sec 3.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details of pre-training and evaluation be found in 4.1 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: The code has been released at https: : thub. cht.b..co//ppencWaaa/I.... Only pulic datasets have been used in our research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
ooiicyyyyy)).. for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips... public/guider
sssissii wwwssssoiccyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
Most experiments have stable results with little variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp:/:///uui...//eeeee wttppppp.ccppppeeateeeee httppp/lcc/hhhcccceeeeeeee..eeee ctw.idheellellleeccccccc ines?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Appendix D.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release a novel dataset or model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper each time asset appears.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "20U40JIM6z", "reviewer": "Reviewer_ktxP", "summary": "This paper aims to explore the use of weak supervision signals in multimodal interleaved image-text data to pretrain visual encoder, compressing the distribution of high-level features into the visual encoder. The paper employs contrastive loss and autoregressive loss to train the model. To prevent the collapse of visual representations, an entropy maximization constraint is applied. The paper derives the equivalence of maximizing the mutual information between the model's input and output as a latent compression and entropy constraint. The proposed pre-training method, called LCL, achieves performance comparable to CLIP on paired data while better utilizing the supervision information in interleaved data.", "questions": "see weaknesses.", "limitations": "The paper's discussion on data bias and energy consumption limitations is commendable; however, it could further explore issues related to data privacy.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper explores how to use weak supervision signals in more general interleaved image-text data to accomplish visual pre-training. Its advantages are as follows:\n1. Unlike previous approaches that fine-tune pre-trained visual models to align visual representations with the text space (Flamingo, LLaVA), this paper explores how to train visual models from scratch using interleaved image-text data. This is a meaningful exploration.\n2. To prevent the collapse of visual representations, where autoregressive generation relies solely on textual information, this paper imposes an entropy constraint and further derives it as optimizing mutual information. This approach aids in model training.\n3. Extensive quantitative experiments have validated the effectiveness of the visual models trained using this approach.", "weaknesses": "This paper has the following areas for improvement:\n1. In some cases, the textual context may have little relevance to the image. It is worth investigating whether such data could harm the model's performance.\n2. The paper lacks qualitative experiments to further demonstrate the effectiveness of the method. Designing reasonable visualization analyses would help to further elucidate the advantages of the approach.\n3. Similar to CLIP, further demonstrating the model's transfer learning performance through domain adaptation tests and few-shot metrics would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your good questions and constructive suggestions.\n___\n**Q1:** In some cases, the textual context may have little relevance to the image. It is worth investigating whether such data could harm the model's performance.\n\n**A1:** This is a very good question.  It is inevitable that image-text data obtained from the internet may have weak correlations. For fair experiments, we currently follow the settings in previous works when using datasets. \nPrevious works have used filtering methods to improve the quality of data. For example, the LAION-400M dataset used CLIP scores to filter image-text pairs, and OpenFlamingo used CLIP scores to filter out uncorrelated images in the MMC4 dataset. We now follow their settings.\n\nWe note that recent work, OmniCorpus[a], analyzed model pre-training with differently filtered data in Section 5.2 (Table 3). The results show that appropriate filtering can improve the correlation of images and texts, thereby enhancing model performance. However, excessive filtering may lead to insufficient data diversity, thus hurting performance. We will analyze the impact of data quality on our model in future work.\n\n[a] OmniCorpus: An Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text. arXiv:2406.08418.\n___\n**Q2:** Qualitative experiments to further demonstrate the effectiveness of the method. Designing reasonable visualization analyses to further elucidate the advantages of the approach.\n\n**A2:** We use t-SNE to visualize the learned features using images from some classes in ImageNet-1K val set. See Fig. R1 in the rebuttal pdf. For both CLIP and our LCL, visual features of different classes are generally distinguishable, and the distance between classes is related to semantic similarity. Compared to CLIP, visualized points of LCL are slightly tighter for semantically similar classes and slightly further away for classes with low semantic relationships. \n___\n**Q3:** Similar to CLIP, further demonstrating the model's transfer learning performance through domain adaptation tests and few-shot metrics.\n\n**A3:** Thanks for your suggestion. We adopt zero-shot domain adaptation tests in CLIP. Pre-training ViT-B/16 on LAION-400M, OpenCLIP achieves 67.1 on zero-shot ImageNet-1k, while ours is 60.1. However, domain adaptation via zero-shot retrieval is designed for contrastive learning such as CLIP. It may not be suitable for other types of pre-training methods (e.g., MIM, generative tasks). We have discussed this in Section 4.1.\n\nMoreover, similar to CLIP, we have used probing (fine-tuning a task head with backbone frozen) to evaluate transfer learning capabilities. See Tab.5, IN-1k frozen transfer results. Our LCL is comparable with OpenCLIP, demonstrating the strong transferability of our pre-trained vision model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Okk5ri69uJ", "reviewer": "Reviewer_A7ig", "summary": "The paper pre-trains models with a combination of a contrastive image-text objective and a generative language objective. The authors provide many results on image classification and vision-language tasks suggesting the competitiveness of the method in controlled settings.", "questions": "Please see the weaknesses section for specific questions and topics to address.", "limitations": "The authors address limitations in a dedicated paragraph.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. The paper is well framed and motivates nicely the need to pre-training on interleaved data.\n\nS2. The paper gives good intuition about what the various equations mean, making the manuscript more accessible.\n\nS3. Consideration of many pre-training datasets including LAION-400M, MMC4, and OBELICS.\n\nS4. Extensive objective ablations spanning both contrastive and generative losses.", "weaknesses": "W1. [MAJOR] The paper presents the objective as novel (L44-54); however, it seems similar to CoCa (Yu et al., 2022.), which also employs a contrastive loss and a next token prediction loss. Can the authors clarify the differences and why the formulation is novel?\n\nW2. It seems equation 3 appears in prior work; however, when it is first presented it seems to be presented as a novel insight. I recommend making the attribution to prior work more clear before introducing the equation.\n\nW3. In the relation to previous pre-training tasks, it is important to also relate to CoCa. It seems the objective is pretty much the same suggesting that the objective is not actually a contribution of the work. Is there any reason CoCa is not mentioned here given the similarities?\n\nW4. Make sure it is clear that you train on sequences with more than one image per sample (I am assuming this is true because you train on MMC4, but when explaining the objectives you include only one sequence for simplicity). 3.3 is a nice place to add this information. Also any special tricks to get multi-image to work? If so, it could also be nice to mention this.\n\nW5. Why are the numbers for Flamingo in Tab 1 for IN-1k so low? Flamingo uses a pre-trained vision backbone, so I expect numbers to be good here.\n\nW6. Is the COCO CIDEr evaluation protocol zero-shot? If so the number in table 4(a) of 87.5 looks extremely high relative to open flamingo and Idefics. Please double check this number and if few-shot prompting is used here, please make this clear. Also why is Gen. only worse than Con. only for captioning. How is contrastive learner able to do captioning?\n\nW7. In the frozen transfer setting in Tab. 6 are all models fine-tuned on the same data? If so, what data? The specifics of the experiment are not clear to me, making it hard to interpret the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.\n___\n**Q1: (a)** The proposed objective seems similar to CoCa, which also employs a contrastive loss and a next token prediction loss. Clarify the differences and why the formulation is novel. **(b)** In \"relation to previous pre-training tasks\", why CoCa is not mentioned given the similarities.\n\n**A1 (a):** CoCa cannot be extended to interleaved data since its learning objective relies on paired data. Applicable to interleaved data is a contribution of our proposed learning objective. There are significant differences between ours and CoCa in how the loss is applied:\n1. CoCa's contrastive loss relies on the positive pairs from paired data, which cannot be directly obtained in general interleaved data.\n2. CoCa's generation can only be conditioned on a single image, while our generation is conditioned on preceding context containing flexible interleaved image-text contents.\n\n**A1 (b):** In \"relation to previous pre-training tasks,\" we aim to claim that existing pre-training methods cannot train visual features from scratch using interleaved data. CLIP is the most representative method for paired data, and CoCa is also specialized for paired data. Therefore, we did not specifically discuss CoCa in this context.\nGiven the similarity in the loss form, we will include further discussions about CoCa in the revision and clarify our contribution as mentioned above.\n___\n**Q2:** Equation 3 seems to appear in prior work. Make the attribution to prior work more clear before introducing the equation.\n\n**A2:** Thanks for your suggestion. Prior work (i.e., M3I mentioned in the paper) showed the relationship between cross-entropy and mutual information in other pre-training tasks, e.g., image self-supervised learning and contrastive learning. While Equation 3 specifically demonstrates this relationship when using an autoregressive model to compress interleaved image-text data. Our novel insight is that, in this scenario, the compression (cross-entropy) objective and the maximum mutual information objective are equivalent. In the revision, we will make the attribution to prior work and clarify our novel insight clearer.\n___\n**Q3:** Make sure that it is trained on sequences with more than one image per sample. Any special tricks to get multi-image to work.\n\n**A3:** Yes, we support interleaved data containing multiple images. As described in Appendix A.1, each MMC4 sequence may contain 1 to 6 images. We did not use any special implementation tricks. See Fig. R2 in the rebuttal pdf for illustration. The visual features of multiple images are inserted at their positions in the sequence, combined with text embeddings, and then processed through a causal language model to compute the generation loss. For each image, its global feature and the language model’s output feature at its preceding <boi> token form a positive pair in contrastive loss. We will add this information to Section 3.3.\n___\n**Q4:** Why are the numbers for Flamingo in Tab 1 for IN-1k so low?\n\n**A4:** The number of Flamingo comes from training from scratch with the Flamingo's pre-training task rather than its released checkpoint. Tab. 1 compares pre-training tasks, so the model and data for all methods should be fair. To avoid misunderstanding, we will clarify in the caption of Tab. 1 that the method names in parentheses refer to the pre-training task but not their trained checkpoints.\n___\n**Q5: (a)** Is the COCO CIDEr evaluation protocol zero-shot? The number in Tab.4(a) of 87.5 looks extremely high. **(b)** Why is Gen. only worse than Con. only for captioning? How is contrastive learner able to do captioning?\n\n**A5 (a):** The results in Tab. 4 are evaluated under frozen transfer setting, which we will clarify in the table caption. See Appendix A.2  for frozen transfer details. Frozen transfer refers to fine-tuning downstream task models while the parameters of the pre-trained visual encoder are frozen. We use LAION-COCO for captioning fine-tuning, so the COCO CIDEr is reasonable.\n\n**A5 (b):** The vision encoder pre-trained by \"Con. only\"  also performs captioning by frozen transfer (see Appendix A.2). \"Gen. only\" is worse than \"Con. only\" for captioning, because visual features pre-trained by \"Gen. only\" experienced a collapse. We have discussed this feature collapse in Section 3.2, \"Relation to Previous Pre-training Tasks.\"\n___\n**Q6:** In the frozen transfer setting in Tab. 6, are all models fine-tuned on the same data? If so, what data?\n\n**A6:** Yes. The frozen transfer in Tab. 6 evaluates the pre-trained vision encoders on multimodal tasks following LLaVa-1.5. All models use the same data and training settings as LLaVa-1.5. See Appendix A.2 \"Multi-modal dialogue\" for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eAijpzsyCu", "reviewer": "Reviewer_KZWZ", "summary": "The paper tackles the problem of vision model pre-training. More exactly, it aims to exploit the interleaved image-text data that is very prevalent on the Internet. It proposes Latent Compression Learning that maximises the mutual information between the inputs and outputs of a causal attention model. When visual pre-training is applied to interleaved image-text data, visual latents are extracted using a visual encoding network and then combined with the text and fed to the causal model.", "questions": "Why is OpenAI CLIP greyed out?\n\nWhy are the numbers reported for BEiT3 different than what they report in the paper? For example for Flicker 3K, R@1, it's reported 73.2 while the paper reports 88.2\n\nWhy, for example the comparison with BEIT3 is only shown in Tab.1 when they report results on VQAv2? The same question for CoCA?", "limitations": "Limitations are barely discussed at the end of the conclusions. Some limitations can be inferred from the rest of the paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important task and proposes an interesting method that may be of interest to the research community.", "weaknesses": "While the method seems interesting, my main concern is related to the experimental part that I find confusing. For example for BEiT3 the numbers reported are different from the ones reported in the paper.\n\nAlso, I think that for Tab 6, more multi-modal LLMs need to be included. While there can be a debate on fair vs unfair comparison, I think that you present results on a dataset these need to be complete. So, they can be greyed out, put in a different section, etc and explained why the comparison is not fair, but I don't think it's suitable for models that perform better to not be included at all. So, missing comparisons:\n\nFang, Yuxin, et al. \"Eva: Exploring the limits of masked visual representation learning at scale.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\nZou, Xueyan, et al. \"Generalized decoding for pixel, image, and language.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nor even some very recent ones for the sake of completeness:\nSun, Quan, et al. \"Generative multimodal models are in-context learners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\nLiu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the reviews and questions. But there is a misunderstanding here.\n\nFirst, as discussed in Fig. 1, we would like to clarify that our proposed LCL aims to pre-train a vision encoder from scratch using interleaved image-text data, rather than incrementally training a multi-modal model based on a pre-trained vision encoder. Their differences are illustrated in Fig. 1(b) and 1(c) and also acknowledged by Reviewer ktxP.\n\nAccordingly, our experiment is to evaluate the quality of pre-trained vision encoders rather than to compare the performances of different MLLMs. Therefore, when comparing with other pre-training methods, instead of using their released MLLMs checkpoints, we use their released codes or reproduce their methods to pre-train the visual encoders from scratch in a fair setting. We ensure a fair comparison in the same settings, including the model structure, training data, and evaluation settings.\n___\n**Q1: (a)** For BEiT3, the numbers reported are different from the ones reported in the paper. **(b)** Why the comparison with BEIT3 is only shown in Tab.1 when they report results on VQAv2? The same for CoCA?\n\n**A1 (a):** The number of BEiT3 comes from training from scratch with BEiT3's pre-training task rather than the reported ones of the trained checkpoint. Tab. 1 compares pre-training methods and pre-training tasks, so the model and data for all methods should be fair. To avoid misunderstanding, we will clarify in the caption of Tab. 1 that the method names in parentheses refer to the pre-training tasks but not their trained checkpoints.\n\n**A1 (b):** Table 6 evaluates pre-trained vision encoders on multimodal dialogue tasks by integrating them with the same pre-trained LLM for fair comparison. Therefore, the reported results of BEiT3 and CoCa's released MLLM checkpoints are not included. In Table 1, BEiT3 and CoCa are compared as pre-training methods under fair settings.\n___\n**Q2:** In Tab 6, more multi-modal LLMs need to be included.\n\n**A2:** As mentioned above, our proposed LCL pre-trains vision encoder from scratch, which is orthogonal to the training of MLLMs. Accordingly, our experiment is to evaluate the quality of pre-trained visual features rather than to compare the performances of different MLLMs.\n\nTab. 6 is to fairly compare different pre-trained vision encoders in multimodal tasks, so the results of other MLLM's trained checkpoints, such as EVA[a], X-Decoder[b], and Emu[c], should not be incorporated. We have used LLaVA-1.5[d] for this evaluation (see Appendix A.2, \"Multi-modal dialogue\"). There are better MLLMs available than LLaVA-1.5, but training better MLLMs is beyond the scope of this paper, and we do not need to make such comparisons.\n\nIn addition, we have compared the text generation + image feature regression task used by Emu in Tab. 1. However, EVA requires pre-trained CLIP features and X-Decoder uses segmentation data, so their training tasks are not comparable to those methods pre-training from scratch only using image-text data.\n\n[a] Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023\n\n[b] Generalized decoding for pixel, image, and language. In CVPR, 2023.\n\n[c] Generative multimodal models are in-context learners. In CVPR, 2024.\n\n[d] Improved baselines with visual instruction tuning. In CVPR, 2024.\n___\n**Q3:** Why is OpenAI CLIP greyed out?\n\n**A3:** We aim to compare vision pre-training methods rather than pre-trained checkpoints. Both OpenCLIP and we use the LAION-400M dataset for training, allowing for a fair comparison. OpenAI CLIP, which uses private data, is included only as a reference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RzoB0WhCLO", "reviewer": "Reviewer_vQis", "summary": "This paper introduces a vision backbone pre-training method named Latent Compression Learning (LCL) to utilize interleaved image-text data. The proposed LCL approach maximizes mutual information between the inputs and outputs of a GPT-like model in autoregressive manner. The proposed method integrate both discriminative and generative objectives by contrasting preceding context and generate subsequent text based on visual representation. The extensive experiments demonstrate that LCL not only matches the performance of existing models like CLIP on paired datasets (e.g., LAION) but also effectively leverages interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch.", "questions": "1. The scaling behavior is not demonstrated. While the author has shown the effectiveness of training on MMC4 and Laion-400M, it remains unclear how the model performs and correlates across different dataset scales. Understanding this could provide valuable insights into the feasibility and performance of scaling the proposed method to larger datasets, such as DataComp 12.8B and Laion 2B.\n\n2. Can you show the seen samples of each model? It would be helpful for readers understanding the scale of model's training.", "limitations": "The authors have addressed the limitation in their manuscript.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n\n2. The paper introduces a new pre-training method, Latent Compression Learning (LCL), which utilizes interleaved image-text data for visual backbone pre-training for the first time. And this can effectively leveraging large-scale web-crawled data, which is easier to crawl compared to the image-text pairs.\n\n3. Extensive experiments are conducted, demonstrating the effectiveness of the proposed method on both paired datasets (e.g., LAION) and interleaved datasets (e.g., MMC4).", "weaknesses": "1. From Table 5, it appears that solely leveraging image-text pairs with LCL does not provide benefits over the CLIP baseline. However, when using the MMC4 dataset, which is manually composed of interleaved text, there is significant performance improvement on downstream tasks. I am curious whether this performance gain results from the increased number of training samples (i.e., the total number of images used during training).\n\n2. According to Table 3, utilizing original interleaved datasets such as Obelics does not yield any performance gain. In comparison, the MMC4 dataset requires more computation for data filtering with the CLIP score and the use of image-text pairs to create interleaved data. It is unclear how to efficiently utilize the original interleaved data directly crawled from the web. Do you have any insights on the differences between these two types of interleaved datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your good questions and constructive suggestions.\n___\n**Q1:** In Tab. 5, LCL is on par with CLIP baseline solely with image-text pairs but is significantly better when using the MMC4 dataset. Whether this performance gain is from the increased number of training samples.\n\n**A1:** It is reasonable that our LCL is comparable to CLIP on image-text pairs. Our LCL uses the simply constructed LAION-Random dataset (see Section 4.1), degrading to using half of the data for contrastive learning and the other half for generation. Compared to CLIP which uses all data for contrastive learning, there is no significant advantage or disadvantage.\n\nWhen incorporating MMC4 data, the significant improvement comes from increased data diversity. As mentioned in Section 4.3, all models in Tab. 5 are exposed to the same total number of training images. The number of images per epoch doubles when using LAION-400M + MMC4. Tab. 3 also shows that using MMC4 and LAION-400M alone are on par. This suggests that data diversity is the primary factor.\n\nTherefore, the advantage of our LCL is its ability to use larger-scale and more diverse interleaved image-text data, while not suffering performance losses when using paired data.\n___\n**Q2:** In Tab. 3, using original interleaved datasets such as Obelics does not yield performance gain. MMC4 uses CLIP score to filter image-text pairs and create interleaved data. Any insights on the differences between these two types of interleaved datasets?\n\n**A2:** This is a very good question, and we suppose that different types of interleaved data may be suitable for different training tasks. Our results suggest that using filtered interleaved data like MMC4 may be better for vision model pre-training. We note that recent work OmniCorpus[a] shows that using original interleaved data like Obelics may be better for fine-tuning MLLM (see their Tab. 4).\n\n[a] OmniCorpus: An Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text. arXiv:2406.08418.\n___\n**Q3:** The scaling behavior is not demonstrated.\n\n**A3:** We are also very interested in the scaling behavior of our method. We anticipate using datasets such as Laion-2B and OmniCorpus[a] (a large-scale interleaved dataset) to train our vision model to achieve SoTA performance. However, using a total of 34B samples like OpenCLIP, training a ViT-L/14 model would require at least 8000 GPU days, which is more than we can afford now.\n___\n**Q4:** Show the seen samples of each model.\n\n**A4:** The number of seen images is 13B for all models in Tab. 5 and 6, as mentioned in Section 4.3. For ablation studies, all models use 2B training images, as stated in Section 4.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the careful reviews and constructive suggestions. We respond to your questions respectively. The PDF contains supplementary figures for rebuttal.

Meta-review: The paper received originally mixed comments. Although all reviewers agreed that the direction is useful, and the approach is technically sound, there are two main issues raised: the similarty of the training objective proposed to CoCa, and the apparent discrepancy of reported SoA results to published ones. Both issues were discussed in length between the authors and reviewers, and the reviewers offered convincing explanations and new information for both.

I lean towards accepting this paper, assuming that the new information and clarifications explosed during the review process will be properly reflected in the revised version.
Decision: Accept (poster)