Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

Authors: chenyu  yang
xizhou  zhu
jinguo  zhu
weijie  su
junjie  wang
xuan  dong
wenhai  wang
lewei  lu
bin  li
jie  zhou
yu  qiao
jifeng  dai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 13:53:24

Abstract: Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent on the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of a causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets (e.g., LAION), but can also leverage interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch, showcasing the potential of vision model pre-training with interleaved image-text data.

Main paper: 
 introduction: 
 Over the past decade, ImageNet [34] pre-trained vision models have significantly advanced computer vision, continuously achieving breakthroughs in various vision tasks [7, 24, 10] . The success of ImageNet has inspired further exploration of better methods for pre-training vision models from scratch. Recently, the focus of pre-training has shifted from manually annotated data to large-scale, web-crawled image-text data. A key milestone in this shift is CLIP [55] , which utilizes image-text pair data hundreds of times larger than ImageNet, delivering superior performance across various tasks and progressively becoming the mainstream method for vision model pre-training. Building on this trend, there is increasing interest in exploring interleaved image-text data, which is more prevalent on the Internet. Unlike the structured image-text pairs used in CLIP, this interleaved data is free-format and non-paired, larger in scale, and richer in textual information. Fully exploiting these interleaved image-text data is necessary for further improving vision model pre-training at scale. Currently, no pre-training method can effectively utilize interleaved image-text data to pre-train vision models from scratch. In preliminary attempts [2, 29, 44] of using interleaved data, vision models were already pre-trained by CLIP on paired image-text data. Subsequent training on interleaved data primarily serves to align the pre-trained vision models with language models, thereby enhancing the multi-modal capabilities of the entire vision-language network. Therefore, it remains an important and open problem of how to effectively learn robust visual representation from scratch on interleaved image-text data.A recent study [30] in Natural Language Processing (NLP) suggests that the success of modern language models originates from the compression of training datasets into model parameters. We believe that such compression learning is also applicable to the multi-modal field, except that the data to be compressed expands from structured plain texts to interleaved image-text data, where the images are of raw pixels and unstructured. Such compression learning should be revised to accommodate to the image data. Raw pixels are unstructured and often contain unnecessary and unpredictable details. Such details are irrelevant to the high-level semantic tasks and should be discarded in compression learning. Thus, we argue that compression learning on interleaved text-image data should be applied to latent image representation to better extract semantic abstracts.In this paper, we propose a novel visual pre-training framework, named Latent Compression Learning. We first theoretically demonstrate that effective latent compression learning can be performed by maximizing the mutual information between outputs and inputs of a causal attention model. When applied to visual pre-training on interleaved image-text data, visual latents are extracted through a visual encoding network (such as ViT [20] ), and then fed together with the text into a causal model. The optimization objective can be derived and decomposed into two parts: 1) contrastive learning between visual latent representation and their previous context to enhance semantic consistency, and 2) auto-regressive prediction to learn the predictability of visual representation for subsequent text. These two training objectives complement each other. For images, the learned latent representation retain information that can be predicted from previous contexts and information needed for predicting subsequent contexts, thus providing effective visual pre-training.In the experiments, various interleaved and paired pre-training methods are evaluated. The evaluation is conducted through transfer learning on multiple tasks, including image classification, image-text retrieval, image captioning, and visual dialogue. The pre-training datasets include the widely used image-text paired LAION-400M [57] and the image-text interleaved MMC4 [88] and Obelics [36] .In addition, we also re-organize an interleaved version of LAION-Random and a paired version of MMC4-Pair, to facilitate the comparison between interleaved and paired pre-training methods under the same data source. Experiment results show that our LCL pre-training can achieve the same performance as CLIP on paired pre-training data and can better utilize interleaved pre-training data.Our results also demonstrate the effectiveness of using interleaved image-text data to learn robust visual representation from scratch, and the potential of compression learning for visual pre-training.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please see Sec 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Vision-centric Pre-training Methods. Supervised Pre-training on large-scale annotated datasets [24, 62, 48, 82] has remained the mainstream method for a long time, and has been favored by various vision tasks [9, 24, 79, 7] demonstrating strong performance. Self-Supervised Pre-training has gained significant popularity due to its advantage of utilizing unlabeled data. BEiT [6] follows the methodology of BERT [19] by randomly masking image tokens and reconstructing them as targets. MAE [27] and SimMIM [80] directly use masked pixels as reconstruction targets, making the pretraining process more straightforward and efficient. Weakly-Supervised Pre-training leverages imagehashtag [50, 76, 63] and image-text datasets [71, 61, 8, 57] , which rely on noisy text supervision from the internet. For image-hashtag datasets, related works [50, 63] have shown comparatively good performance across various transfer learning settings. In the case of image-text datasets, early efforts [3, 42, 49, 64, 66, 67, 70, 13, 39] focused on learning general visual-linguistic representation.Recently, exemplified by CLIP [55] , methods [55, 32] Recent works [87, 68, 69, 23, 4] such as Flamingo [2] and KOSMOS-1 [29] perform incremental learning on non-public IITD based on previously pre-trained vision and language model parameters as initialization, training text generation models with multi-modal understanding capabilities. With the continuous advancement in the field and the proliferation of public IITD (e.g., MMC4 [88] and OBELICS [36] ), numerous models [83, 85, 14] capable of multi-modal understanding have emerged. However, these efforts only perform incremental pre-training on IITD and only analyze the usage of IITD on multi-modal dialogue models. Whether IITD contributes to learning robust visual representation from scratch remains unknown.Compression Learning in NLP. The perspective [59, 60] that compression is closely connected to intelligence has a long history. A common compression method is arithmetic coding [56, 52] , which is a practical approach that uses probability models for optimal data encoding. Some studies [28, 51, 38, 31, 37] argue that compression and intelligence are fundamentally equivalent. With the popularity of large language models, the equivalence of language modeling and compression has once again drawn widespread attention, prompting numerous explorations. Recently, [18] demonstrated through examples that language models serve as universal compressors. [30] posited that language modeling is equivalent to compressing a dataset into model parameters and proposed that compression efficiency is linearly related to model capabilities. Although compression learning has been proven effective in the field of NLP, it is not clear whether this approach can be extended to other fields.Multi-modal Large Models. Text supervision pre-trained vision models [55, 32, 40] are widely utilized and have exhibited superior performance in tasks ranging from image retrieval and image classification to captioning. Currently, the most popular and closely watched application area is multimodal dialogue. Multi-modal dialogue models [2, 41, 86, 17, 25, 21, 53] primarily rely on a powerful pre-trained vision encoder [55, 22] and text decoder [16, 74, 75] . The usage of pre-trained vision encoder can generally be divided into two categories: the majority [2, 72] , represented by LLaVA [46] , employ a strategy where the pre-trained vision encoder is frozen, and only the subsequent adapters and language models are trained. A minority, exemplified by Qwen-VL [5, 53] , utilize high-quality image-text dialogue data to continue fine-tuning the vision model. These two training strategies correspond to the two evaluation methods used in this paper to assess the performance of vision models. Image latent representation is extracted via a vision encoder and subsequently input into a language model alongside textual embedding. Two complementary losses are utilized to learn robust visual representation from scratch on interleaved image-text data: a contrastive loss ensures consistency between the visual latent representation and its preceding context, while an auto-regressive loss enhances the predictability of visual representation for subsequent text.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Sec 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
latent compression learning: 
 Auto-regressive Language Modeling as Compression Learning. Recent works [18, 30] have shown that auto-regressive language modeling is equivalent to compression learning. Suppose g ϕ is a language model (LM) with learnable parameters ϕ. Given an input text sequence x = (
, x 1 , x 2 , . . . , x N ), where 
 is a special token indicating the beginning of text, the model outputs y = g ϕ (x) = (y 1 , y 2 , . . . , y N ) predicting the next token based on preceding context, i.e., xk = y k = g ϕ (x) k . The approximate probability of x estimated by g ϕ is q(x) = N k=1 q (x k |y k = g ϕ (x) k ). The model is optimized with NLL-loss, which equals to minimizing the the cross-entropy between the data distribution p and model distribution q:EQUATIONNotice that H(p, q) is actually the optimal expected code length encoding p by q, minimizing H(p, q) just means compressing the data into the model parameters.Latent Compression for Interleaved Image-Text Data. We believe that the compression principle could apply to multi-modal domain, specifically, to train vision-language models by compressing interleaved image-text data. However, instead of directly dealing with pixel values, we turn to compress high-level image representation for the following reasons: 1) high-level representation can extract useful information from raw pixels while discarding those unpredictable image details. 2) the learned visual representation will align with text semantics, making it possible to perform effective visual pre-training with interleaved image-text data.Specifically, let x = (
, x 1 , x 2 , . . . , x N ) be an interleaved image-text sequence. To simplify the expression without loss of generality, we assume that there is only one image in the sequence. Sub-sequence x i:i+M are M + 1 image patch tokens of the input image (e.g., non-overlapping patches in ViTs) and others are text tokens. I = {i, i + 1, . . . , i + M } denotes the indices of the image patches, and T = {1, . . . , i -1, i + M + 1 . . . , N } denotes the indices of text tokens. As shown in Fig. 2 , to construct the sequence of latent representation z = (
, z 1 , z 2 , . . . , z N ), for image patches, we use a parametric vision encoder f θ (e.g., ViTs) to map the data sequence x i:i+M into latent variable z i:i+M . For text tokens, we directly use one-hot vectors corresponding to their vocabulary ids as the latent codes. Then, the latent representation z are fed into a causal attention model g ϕ for latent compression by minimizingEQUATIONwhere the k-th element of the output y k = g ϕ • f θ (x) k predicts the next input latent z k , f θ is identity for text tokens for simplicity of annotation. When the compression only applied on text tokens, it degenerates to auto-regressive language modeling on interleaved image-text data used by previous methods (e.g., Kosmos and Flamingo).However, direct optimizing Eq. ( 2) for learning informative latent representation is non-trivial, since Eq. ( 2) suffers from a naturally trivial solution of visual representation collapse, i.e. the image latent representation z i:i+M may be learned to be data-independent. In fact, as showed in Sec. 4.2, we have observed such visual representation collapse, when training from scratch on MMC4 dataset with Eq. ( 2) applied to text tokens only (i.e., auto-regressive language modeling).Maximizing Mutual Information for Latent Compression Learning. Optimizing directly for latent compression in Eq. ( 2) may cause the visual representation collapse. A natural constraint is to maximize the representation entropy to prevent collapse. We find that combining latent compression and maximum entropy constraint is exactly equivalent to maximizing the mutual information between the model inputs and outputs.Prior work [65] have shown the relationship between cross-entropy and mutual information in other pre-training tasks. Here, we derive this relationship in the latent compression task: maximizing the mutual information between the output y and the input latent z of the causal attention model g ϕ is equivalent to compressing z by minimizing H(p, q) in Eq. ( 2) meanwhile maximizing the entropy of each element z k in z:EQUATIONwhere we use p(y k , z k |x) = p(y k |x)p(z k |x) in the first step since z k and y k can be independently computed given input x, and p(z k |y k ) is estimated by an approximate parameterized distribution q(z k |y k ). For the derivation of the formula, please refer to [65] .Therefore, using I(y; z) as the optimization objective can achieve latent compression while avoiding representation collapse of z via the maximum entropy constraint. The compression of z imposes the model to extract useful information and discard unpredictable information of the image. Meanwhile, maximizing I(y; z) requires that each y k could obtain enough information from previous latent z <k to predict z k . Each z k should carry predictable information. These guarantee that the image representation encode rich semantic information aligned with text. We suppose that the above properties learned by the image representation are desired for vision-language pre-training , thus we use Eq. ( 3) as our pre-training objective. Parameters ϕ and θ are be jointly optimized under this objective. Intuitively, the vision encoder f θ learns to represent images by high-level abstract, and the causal attention model g ϕ learns to compress this high-level abstract of the dataset.In this sub-section, we demonstrate how Eq. ( 3) is decomposed into training tasks and losses. Firstly, I(y; z) can be decomposed as a cross-entropy term and an entropy term in the following two symmetric ways (see Appendix B for detailed derivation):EQUATIONEQUATIONSince given the input x, latent z k and y k are independent and deterministic (i.e., determined by f θ and g ϕ ), yielding 4), p(z k |y k ) is estimated by a parameterized distribution q 1 (z k |y k ), which approximates the distribution of z k given the model's prediction y k . Similarly, in Eq. ( 5), p(y k |z k ) is estimated by q 2 (y k |z k ), the predicted distribution of y k given z k . Therefore, maximizing mutual information can be decomposed as follows: 1) The causal attention model g ϕ learns to predict the next latent z k from the output y k .p(y k , z k |x) = δ [z k = f θ (x) k ] • δ [y k = g ϕ • f θ (x) k ]. δ[•] is delta distribution. In Eq. (2) The learnable latent representation z k learns to predict y k , which is the representation of its previous context.3) The maximum entropy regularization avoids the collapse of z k and y k .In the following, we show that the cross-entropy terms in I(y; z) can be achieved by two common training tasks and loss functions, while the entropy constraints are implicitly satisfied.Contrastive Learning between Image Representation and Preceding Context. For image latent z k and the corresponding y k representing the semantics of its preceding context, the objective defines a bidirectional prediction. We choose q as Boltzmann distribution, i.e., q(z k |y k ) ∝ exp(z ⊤ k W ⊤ 1 W 2 y k /τ ) and q (y k |z k ) ∝ exp(y ⊤ k W ⊤ 2 W 1 z k /τ ),where τ is the temperature, W 1 and W 2 are learnable linear projections. Consequently, the objective becomes the contrastive loss in two directions between z k and y k , when setting z k ′ and y k ′ from other images as negative samples:EQUATIONMeanwhile, the contrastive loss also prevents z k and y k from being trivial representation by pulling them away from negative samples, implicitly appending the entropy regularization.Auto-regressive Text Generation. For a text token, its latent code z k is a one-hot vector and is not learnable, so the objective only imposes y k to predict z k as in Eq. ( 4). We choose q (z k |y k ) as softmax over the output logits on the text vocabulary, i.e, q (z k |y k ) = z ⊤ k softmax(V y k ), where V is the projection head of the language model. The objective corresponding to the text tokens is simply the objective of standard next token prediction with cross-entropy loss:EQUATIONThe total training loss is defined as L = λL con + L gen , where λ is balancing weight.Relation to Previous Pre-training Tasks. In our proposed pre-training framework, the contrastive task and generation task align the image representation with preceding context and subsequent context, respectively. Hence, combining these two pre-training tasks can fully leverage the semantic information contained in interleaved image-text data to supervise the learning of image representation. For previous pre-training tasks, 1) Contrastive Language-Image Pre-training (CLIP) [55] has a similar objective of maximizing the mutual information between corresponding image and text [65] , but it can only be applied to paired image-text data. 2) Contrastive Captioner (CoCa) [81] combines a captioning (text generation) loss with the CLIP loss, but it cannot be applied to interleaved data, either. CLIP loss requires image-text pairs, while the captioner can only be conditioned on a single image can rather than flexible interleaved image-text contents. 3) Auto-regressive Text Generation task only leverages the semantic information in subsequent context to supervise the learning of image representation, while the preceding context is missing, and representation collapse cannot be avoided. Moreover, interleaved image-text data usually has information redundancy, i.e., the image and its corresponding text may contain similar information. So models may rely on information from the text rather than the image for prediction. This is particularly true when the vision encoder is trained from scratch, resulting that the image representation are never focused and optimized. Our experiments in Sec. 4.2 confirms this analyse.The overview of model architecture when adopting our LCL is shown in Fig. 2 . The interleaved image-text input sequence may contain multiple images. We adopt a Vision Transformer (ViT) [20] as the vision encoder, which encodes each image into a sequence of visual embeddings as its latent representation. The visual embeddings of each image are inserted at corresponding positions in the interleaved sequence, and we introduce special tokens <BoI> and <EoI> to indicate the beginning and the ending positions, respectively. The combined visual and text embeddings are fed into a causal language model. The text generation loss is the standard cross-entropy loss over the output logits of the language model defined in Eq. ( 7). In the contrastive learning task, calculating the loss for each image token is extremely computationally expensive. To alleviate this problem, we consider utilizing one global representation per image instead of all latent representation for contrastive learning. Specifically, the global representation v i of each image is extracted from its latent representation z i:i+M through an attention pooling, which is a multi-head attention layer with a learnable query token. The output of the causal transformer model at <BoI> token, just before the image, is processed by a LayerNorm layer and a linear projection into t i . The contrastive loss in Eq. ( 6) is calculated between v i and t i .4 ExperimentQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment settings: 
 Pre-train Data. The datasets utilized in our pre-training encompass the image-text pair dataset LAION-400M [57] , as well as the image-text interleaved datasets MMC4 [88] and OBELICS [36] .We also re-organize two datasets for fair comparison with CLIP. 1) LAION-Random as a interleaved dataset. Images from LAION-400M are randomly placed before or after their paired caption to form image-text sequences. 2) MMC4-Pair as a paired dataset. For images in MMC4, we select matching text from the bipartite graph matching results derived from CLIP similarity to generate pseudo image-text paired data.Implementation Details. We adopt the same image transform in OpenCLIP [15] for pre-training and set the image size as 224 × 224 for all experiments. We employ ViT [20] as our vision encoder.ViT-L/14 is used for the main results, and ViT-B/16 is used for ablation studies. The language model follows the same architecture as OPT-125M [84] but is randomly initialized. By default, the contrastive loss balancing weight is set at λ = 0.1. AdamW optimizer with β 1 = 0.9, β 2 = 0.95 and a weight decay of 0.1 are used. We employ a cosine learning rate schedule with linear warmup and set the peak learning rate at 5e-4 for the LAION data and 3e-4 for others. The model for the main results is trained for 200k iterations with 64k images per batch on average. In the ablation study, the models are trained for 250k iterations with 8k images per batch.Vision Encoder Evaluation. CLIP pre-trained models typically uses zero-shot retrieval performance for evaluation. However such zero-shot inference on retrieval tasks [55] is not suitable for other generative pre-training methods. To address this discrepancy, we choose to evaluate pre-trained vision models by transfer learning on various downstream tasks under two configurations: "frozen transfer" and "full transfer". In the frozen transfer setting, only the parameters outside of the vision model are trained. In the full transfer setting, all parameters are trained.The evaluation is conducted on image classification, image-text retrieval, image caption, and multimodal dialogue. Please see Appendix A for more data, implementation, and evaluation details.In this section, we show the superiority of our LCL when using interleaved image-text data for vision pre-training. However, existing vision pre-training methods only support paired data, so we choose MMC4 dataset with a paired set (MMC4-Pair) available for those methods. In addition, to confirm that it is is not trivial to use interleaved data for vision pre-training, we also consider existing methods to use interleaved data. We aggregate all the pre-training methods divide them into the following tasks: (1) Image-text contrastive (Con.) [55, 15, 32] , (2) Image-text contrastive + image captioning (Con. + Cap.) [81] , (3) Image-text contrastive + image captioning + image-text matching (Con. + Cap. + Mat.) [40, 41, 14] , (4) Auto-regressive text generation (Gen.) [2, 29, 44] , (5) Auto-regressive text generation + image regression (Gen. + Reg.) [69, 68, 72] , (6) Mask data modeling (Mask.) [78] . We select one representative method from each task. For fair comparison, we implement these methods on the same model and training data with available open-source codes or our reproduction.Pre-training on Interleaved Image-Text Data. We conduct experiments on the MMC4 dataset [88] to demonstrate the effectiveness of our LCL on interleaved image-text data. For pre-training methods that only support paired data, MMC4-Pair is used.The results are shown in Tab. 1. Our LCL pre-training method significantly outperforms all other methods in the caption tasks, indicating that we can effectively utilize the rich text context information in MMC4 interleaved data. On the other hand, our method is on par with the best paired pre-training methods on classification and retrieval tasks. Since the paired pre-training methods are directly optimized for retrieval, our comparable performance shows that the visual features are learned to be highly distinguishable. It is worth mentioning that for more general interleaved data, where no paired versions exist, these paired pre-training methods cannot be applied.In addition, we observed that the two methods using auto-regressive text generation do not achieve good performance and feature collapse occurs. However, their text prediction training loss is actually close to ours. This suggests that these methods tend to rely on redundant text information rather than image information for subsequent text prediction. As discussed in Sec. 3.1, our approach can avoid such collapse. Pre-training on Paired Image-Text Data. We conduct experiments on the LAION-400M dataset to show that LCL pre-training also performs well on paired image-text data without specific modification. Tab 2 shows that our method is comparable to paired pre-training methods on various tasks, indicating that all information in paired data is fully exploited.To further confirm the effectiveness of our proposed Latent Compression Learning (LCL), we compare our pre-trained model with existing checkpoints of pre-trained vision encoders. We use LCL to pre-train a ViT-L/14 with mixed data from the LAION-400M and MMC4. We compare it to the ViT-L/14 pre-trained by OpenCLIP [15] using the public LAION-400M dataset, and the ViT-L/14 pre-trained by OpenAI CLIP [55] with private data is listed as reference. The total number of images seen during pre-training is 13B for all models. We evaluate the pre-trained vision encoders by transferring to downstream tasks, i.e., integrate the vision encoders into downstream task models and compare the fine-tuning results. More training details are in Appendix A.1 and evaluation details are in Appendix A.2.Tab. 3 and Tab. 4 show the results of transfer evaluations. When both use LAION-400M as pretraining data, as with the previous experimental conclusions, LCL has similar performance to CLIP. When combined with MMC4, our method achieves better performance, especially on caption and multi-modal dialogue tasks.There exist approaches achieving better results on some benchmarks. However, they either use larger vision encoders, more training data, or private data. We list those results as reference in Appendix C.Latent Compression Learning on Different Datasets. We apply LCL pre-training to more datasets to confirm its generalizability. As shown in Tab. 5, our method also achieves reasonable performance on interleaved dataset OBELICS. It is worth noting that the models trained on MMC4 and OBELICS have achieved similar performance to that on LAION, indicating that it is completely feasible to pre-train visual models only from interleaved data. Furthermore, using both LAION and MMC4 data during pre-training improves performance, suggesting that further improvements can be obtained by incorporating more image-text data. In this case, supporting interleaved data is a key advantage of our approach, enabling the use of more diverse image-text data for pre-training.Loss Balance in Latent Compression Learning. Table 6a ablates the contrastive loss and generation loss used in LCL . Consistent with the previous analyses, LCL can achieve the best performance. Table 6b studies the appropriate loss balancing weights (multiplied by the contrastive loss). It turns out that λ = 0.1 will produce the best results. The performance drops significantly for larger λ values, indicating that the optimization directions of the two losses are not completely consistent. 
conclusion: 
 No existing work has explored vision model pre-training with interleaved image-text data. To this end, we propose Latent Compression Learning (LCL) framework that compresses interleaved image-text latent for vision pre-training. We theoretically show that latent compression is equivalent to maximizing the mutual information between the input and output of a causal model and further decompose this objective into two basic training tasks. Experiments demonstrate that our method is comparable to CLIP on paired pre-training datasets, and it effectively learns robust visual representations utilizing interleaved image-text data. Our work showcases the effectiveness of using interleaved image-text data to learn robust visual representation from scratch, and confirms the potential of compression learning for visual pre-training.Limitations. Our experiments are constrained to a limited size of dataset and vision encoder, and the scaling property of our proposed method remains unexplored. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Please see Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Most experiments have stable results with little variance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Please see Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Please see Appendix D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release a novel dataset or model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper each time a new asset appears.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Pre-training Data. For the data in MMC4, we select images with a CLIP similarity to the matching text of 0.24 or higher. From documents containing at least one such image, we randomly choose up to 6 images to form an interleaved image-text sequence, utilizing all text from that document. If the sequence length exceeds 2048 tokens, the surplus is truncated while ensuring the integrity of both images and individual text segments, and then padded to the designated length. For OBELICS, we similarly restrict the number of images per document to between 1 and 6. We then sequentially extract 2048 tokens from the concatenated documents. If image tokens are truncated, the entire image is moved to the next sample sequence.To construct interleaved image-text samples from the MMC4 dataset, we randomly place images either before or after their corresponding sentences, adhering to a 50% probability, thus generating a document-wise interleaved sequence of images and text. For the OBELICS corpus, individual documents are concatenated, and a sliding window strategy is employed to select each image-text sequence, maintaining a total length of 2048 tokens.Hyper-parameters. Our pre-training configuration is shown in Tab. 7. The AdamW optimizer was employed for model training with the learning rate set to 3e-4 and the weight decay set to 0.1. Mixed numerical precision training with bfloat16 is also employed to stabilize the optimization process. Furthermore, we set a drop-path [35] rate linearly increasing to 0.2, and use layer-scale [73] for stable training. Transfer Tasks. We conduct our performance evaluation of pre-trained on image classification, image-text retrieval, text generation tasks with multimodal inputs (i.e., image captioning and multimodal dialogue). Their model architecture in transfer learning are illustrated in Fig. 3 .For closed-set image classification, a lightweight classifier with a randomly initialized attention pooling layer, followed by a layer normalization layer and a linear layer, is appended to the top of the pre-trained vision model. In the "frozen transfer" scenario, only the parameters of the added classifier are trainable, similar to the linear probing strategy. Conversely, in the "full transfer" approach, all parameters, including those of the pre-trained vision encoder, are adjustable.Regarding the image-text retrieval task, we discard the pre-trained text encoder and apply contrastive learning to the pretrained vision encoder with a newly introduced text encoder. Images are processed through the vision encoder and a randomly initialized attention pooling layer to generate a global image embedding. Textual captions are processed through the text encoder, utilizing the feature of the final token as the text embedding representing the input caption. An additional linear layer facilitates the dimensional alignment between the image and text embeddings, enabling their use in contrastive learning. During "frozen transfer", the attention pooling layer on the vision transformer and the text encoder are trainable; similarly, in "full transfer", all parameters, including the vision encoder, can be optimized. For text generation tasks with multimodal inputs, we adopt image captioning and multi-modal dialogue benchmarks and employ prevalent architectures like those in [46] . Specifically, a pretrained LLM for text generation is integrated on top of the pre-trained vision model, incorporating an MLP layer to adjust the dimensions of the visual embeddings. During the "frozen transfer" evaluation, the parameters of the vision model remain fixed, while in the 'unfreezing' phase, these parameters are permitted to undergo training.Implementation Details. Vision encoder evaluation with "frozen transfer" and "full transfer" configurations includes fine-tuning training and benchmark evaluation. Implementation details of each transfer task are list below, and the hyper-parameters involved are listed in Tab. 8.• Image classification. Model is trained on the ImageNet-1K [33] train split and evaluated on val split. We follow the attention probe setting introduced by [12] for "frozen transfer", and the full fine-tune setting in [43] for "full transfer".• Image-text retrieval. Model is trained on a combination dataset comprised of CC12M [61] , CC3M [61] , and SBU [77] , and is tested on the MSCOCO [11] karpathy-test split and Flickr30k [54] test split. The model is trained with Adamw optimizer for 5000 iterations. The learning rate is set at 1e-3 and 1e-5 for parameters without initialization and with initialization, respectively.• Image captioning. Model is trained on a subset of the LAION-COCO [58] dataset, which includes 10 million samples, and evaluation is performed on the MSCOCO [11] karpathy-test split and NoCaps [1] val split. Here, the model is trained for 20,000 iterations with a learning rate of 1e-4. Additionally, a droppath technique is employed with a ratio of 0.2 in the vision model to mitigate overfitting.• Multi-modal dialogue. We follow a two-stage training process similar to that used in LLAVA-1.5 [45] . Initially, paired data with 558K samples is used to train an MLP projector to align the Vision Transformer (ViT) with the pretrained LLM. Subsequently, the model undergoes instruction tuning on multimodal dialogue datasets with 665k samples. Both the alignment training and instruction tuning phases are conducted over a single epoch, with learning rates set at 1e-3 and 2e-5, respectively. Evaluations are then performed on multimodal dialogue benchmark, e.g., MMBench [47] and VQAv2 [26] .The effectiveness of our LCL are validated by conducting ablation experiments mainly on two corpora: LAION and MMC4. The experimental hyper-parameters involved are shown in Tab. 9. We found that the optimal learning rate for the LAION dataset is 5e-4, while for the MMC4 dataset, a slightly lower rate of 3e-4 proves most effective. We speculate that this is because MMC4 corpus contains relatively higher noise. Most of the original settings in the large-scale pre-training are retained in the ablation, with the exception of reducing the batch size by a factor of 8 to decrease the computational overhead. Using the notation defined in Sec. 3, the mutual information of the output of the language model (y) and the latent representation (z) can be described as follows:EQUATIONWe can derive Eq. 9 from Eq. 8 because once the interleaved image-text input sequence x is given, the output y k and the latent representation z k can be computed independently:p(y k , z k | x) = p(y k | x)p(z k | x)From Eq. 9, we can further decompose the mutual information into a cross-entropy component and an entropy component in two symmetric ways. One approach involves the output token y k predicting the next latent representation z k , along with the entropy of z k :EQUATIONThe other approach considers how the latent representation z k approximates the previous context y k and includes the entropy of the output y k :I(y; z) = N k=1 E x∼p p(y k | x)p(z k | x) log p(y k | z k ) p(y k ) dy k dz k (12) = N k=1 E x∼p [δ(y k = g ϕ • f θ (x) k ) log P (y k | z k = f θ (x) k )] -p(y k ) log p(y k ) (13)Note that the reason for transitioning from Eq. 10 to Eq. 11 and from Eq. 12 to Eq. 13 is because, given the input sequence x, the outputs y k and z k are determined as follows:p(y k | x) = δ [y k = g ϕ • f θ (x) k ] p(z k | x) = δ [z k = f θ (x) k ]We present supplementary results on classification, retrieval and captioning tasks (Tab. 10) and multi-modal benchmarks (Tab. 11).This work may share the common negative impacts of large-scale vision training. The data used in pretraining may contain dataset bias, and raise ethical concerns. It may also require large computational resources, which consume lots of electricity and result in increased carbon emissions.NeurIPS Paper ChecklistJustification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper received originally mixed comments. Although all reviewers agreed that the direction is useful, and the approach is technically sound, there are two main issues raised: the similarty of the training objective proposed to CoCa, and the apparent discrepancy of reported SoA results to published ones. Both issues were discussed in length between the authors and reviewers, and the reviewers offered convincing explanations and new information for both.

I lean towards accepting this paper, assuming that the new information and clarifications explosed during the review process will be properly reflected in the revised version.

Reviews: review: paper_summary: paper_summary: This paper aims to explore the use of weak supervision signals in multimodal interleaved image-text data to pretrain visual encoder, compressing the distribution of high-level features into the visual encoder. The paper employs contrastive loss and autoregressive loss to train the model. To prevent the collapse of visual representations, an entropy maximization constraint is applied. The paper derives the equivalence of maximizing the mutual information between the model's input and output as a latent compression and entropy constraint. The proposed pre-training method, called LCL, achieves performance comparable to CLIP on paired data while better utilizing the supervision information in interleaved data.
strength_weakness: strength_weakness: This paper has the following areas for improvement:
1. In some cases, the textual context may have little relevance to the image. It is worth investigating whether such data could harm the model's performance.
2. The paper lacks qualitative experiments to further demonstrate the effectiveness of the method. Designing reasonable visualization analyses would help to further elucidate the advantages of the approach.
3. Similar to CLIP, further demonstrating the model's transfer learning performance through domain adaptation tests and few-shot metrics would be beneficial.
questions: questions: see weaknesses.
limitations: limitations: The paper's discussion on data bias and energy consumption limitations is commendable; however, it could further explore issues related to data privacy.
paper_summary: paper_summary: The paper pre-trains models with a combination of a contrastive image-text objective and a generative language objective. The authors provide many results on image classification and vision-language tasks suggesting the competitiveness of the method in controlled settings.
strength_weakness: strength_weakness: W1. [MAJOR] The paper presents the objective as novel (L44-54); however, it seems similar to CoCa (Yu et al., 2022.), which also employs a contrastive loss and a next token prediction loss. Can the authors clarify the differences and why the formulation is novel?

W2. It seems equation 3 appears in prior work; however, when it is first presented it seems to be presented as a novel insight. I recommend making the attribution to prior work more clear before introducing the equation.

W3. In the relation to previous pre-training tasks, it is important to also relate to CoCa. It seems the objective is pretty much the same suggesting that the objective is not actually a contribution of the work. Is there any reason CoCa is not mentioned here given the similarities?

W4. Make sure it is clear that you train on sequences with more than one image per sample (I am assuming this is true because you train on MMC4, but when explaining the objectives you include only one sequence for simplicity). 3.3 is a nice place to add this information. Also any special tricks to get multi-image to work? If so, it could also be nice to mention this.

W5. Why are the numbers for Flamingo in Tab 1 for IN-1k so low? Flamingo uses a pre-trained vision backbone, so I expect numbers to be good here.

W6. Is the COCO CIDEr evaluation protocol zero-shot? If so the number in table 4(a) of 87.5 looks extremely high relative to open flamingo and Idefics. Please double check this number and if few-shot prompting is used here, please make this clear. Also why is Gen. only worse than Con. only for captioning. How is contrastive learner able to do captioning?

W7. In the frozen transfer setting in Tab. 6 are all models fine-tuned on the same data? If so, what data? The specifics of the experiment are not clear to me, making it hard to interpret the results.
questions: questions: Please see the weaknesses section for specific questions and topics to address.
limitations: limitations: The authors address limitations in a dedicated paragraph.
paper_summary: paper_summary: The paper tackles the problem of vision model pre-training. More exactly, it aims to exploit the interleaved image-text data that is very prevalent on the Internet. It proposes Latent Compression Learning that maximises the mutual information between the inputs and outputs of a causal attention model. When visual pre-training is applied to interleaved image-text data, visual latents are extracted using a visual encoding network and then combined with the text and fed to the causal model.
strength_weakness: strength_weakness: While the method seems interesting, my main concern is related to the experimental part that I find confusing. For example for BEiT3 the numbers reported are different from the ones reported in the paper.

Also, I think that for Tab 6, more multi-modal LLMs need to be included. While there can be a debate on fair vs unfair comparison, I think that you present results on a dataset these need to be complete. So, they can be greyed out, put in a different section, etc and explained why the comparison is not fair, but I don't think it's suitable for models that perform better to not be included at all. So, missing comparisons:

Fang, Yuxin, et al. "Eva: Exploring the limits of masked visual representation learning at scale." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Zou, Xueyan, et al. "Generalized decoding for pixel, image, and language." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

or even some very recent ones for the sake of completeness:
Sun, Quan, et al. "Generative multimodal models are in-context learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Liu, Haotian, et al. "Improved baselines with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: Why is OpenAI CLIP greyed out?

Why are the numbers reported for BEiT3 different than what they report in the paper? For example for Flicker 3K, R@1, it's reported 73.2 while the paper reports 88.2

Why, for example the comparison with BEIT3 is only shown in Tab.1 when they report results on VQAv2? The same question for CoCA?
limitations: limitations: Limitations are barely discussed at the end of the conclusions. Some limitations can be inferred from the rest of the paper.
paper_summary: paper_summary: This paper introduces a vision backbone pre-training method named Latent Compression Learning (LCL) to utilize interleaved image-text data. The proposed LCL approach maximizes mutual information between the inputs and outputs of a GPT-like model in autoregressive manner. The proposed method integrate both discriminative and generative objectives by contrasting preceding context and generate subsequent text based on visual representation. The extensive experiments demonstrate that LCL not only matches the performance of existing models like CLIP on paired datasets (e.g., LAION) but also effectively leverages interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch.
strength_weakness: strength_weakness: 1. From Table 5, it appears that solely leveraging image-text pairs with LCL does not provide benefits over the CLIP baseline. However, when using the MMC4 dataset, which is manually composed of interleaved text, there is significant performance improvement on downstream tasks. I am curious whether this performance gain results from the increased number of training samples (i.e., the total number of images used during training).

2. According to Table 3, utilizing original interleaved datasets such as Obelics does not yield any performance gain. In comparison, the MMC4 dataset requires more computation for data filtering with the CLIP score and the use of image-text pairs to create interleaved data. It is unclear how to efficiently utilize the original interleaved data directly crawled from the web. Do you have any insights on the differences between these two types of interleaved datasets?
questions: questions: 1. The scaling behavior is not demonstrated. While the author has shown the effectiveness of training on MMC4 and Laion-400M, it remains unclear how the model performs and correlates across different dataset scales. Understanding this could provide valuable insights into the feasibility and performance of scaling the proposed method to larger datasets, such as DataComp 12.8B and Laion 2B.

2. Can you show the seen samples of each model? It would be helpful for readers understanding the scale of model's training.
limitations: limitations: The authors have addressed the limitation in their manuscript.

Comments: title: General Response cont. 2
General Response cont.
General Response
Rebuttal answer
comment: We sincerely thank you for your insightful comments and suggestions. Please see the general response. We revise our table to include more benchmark results with further discussion for a better understanding.
We sincerely thank you for your thorough and constructive review. In the revision, we will explicitly compare the differences between our objective and the CoCa's in the figure and text. Please see the general response for further discussion about the comparison with state-of-the-art performance.
We sincerely thank you for your thorough review and constructive comments. We respond to your concerns in the general response.
We are glad that your major concerns have been addressed. Thanks for your thorough and constructive review.
| Vision pre-training method | Vision encoder | Pre-training samples | Pre-training data public | MLLM method | LLM size | MLLM training samples | MLLM data public | VQAv2 | GQA | VizWiz | SQA | POPE | MME | MMB | SEED image |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Our transfer evaluation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LCL (ours) | ViT-L (304M) | 13B | ✅ | LLaVA-1.5 | 7B | 1.2M | ✅ | 70.7 | 57.4 | 41.4 | 69.7 | 81.7 | 1291.7 | 55.3 | 56.5 |
| OpenCLIP | ViT-L (304M) | 13B | ✅ | LLaVA-1.5 | 7B | 1.2M | ✅ | 68.7 | 57.0 | 39.5 | 69.0 | 81.8 | 1266.2 | 54.2 | 55.8 |
| OpenAI CLIP | ViT-L (304M) | 13B | ❌ | LLaVA-1.5 | 7B | 1.2M | ✅ | 77.1 | 61.7 | 44.4 | 71.1 | 84.6 | 1486.9 | 65.1 | 64.6 |
| Advanced MLLMs as reference |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OpenAI CLIP | ViT-L-336 (304M) | 13B | ❌ | LLaVA-1.5 | 7B | 1.2M | ✅ | 78.5 | 62.0 | 50.0 | 66.8 | 85.9 | 1510.7 | 64.3 | 65.4 |
| OpenAI CLIP | ViT-L-336 (304M) | 13B | ❌ | LLaVA-NEXT | 7B | 1.3M | ❌ | 81.8 | 64.2 | 57.6 | 70.1 | 86.5 | 1519 | 67.4 | 70.2 |
| OpenAI CLIP | ViT-L (304M) | 13B | ❌ | InternLM-XC2 | 7B | ~20M | ❌ |  |  |  |  |  | 1712 | 79.6 | 75.9 |
| OpenCLIP | ViT-L (304M) | 34B | ✅ | Qwen-VL-Chat | 7B | ~1.5B | ❌ | 78.2 | 57.5 | 38.9 |  |  | 1487.5 |  |  |
| EVA-CLIP | ViT-G (1B) | ～28B | ❌ | *Emu-I | 13B | ~83M | ❌ | 62.0 | 46.0 | 38.3 |  |  |  |  |  |
| EVA-CLIP | ViT-E (4.4B) | ～28B | ❌ | *Emu2-Chat | 33B | ~160M | ❌ | 84.9 | 65.1 | 54.9 |  |  |  |  |  |
| EVA-CLIP | ViT-E (4.4B) | ～28B | ❌ | CogVLM-Chat | 7B | ~1.5B | ❌ | 82.3 |  |  | 91.2 | 87.9 |  | 77.6 |  |
| InternViT | ViT-6B | ～29B | ❌ | InternVL | 7B | ~1.0B | ❌ | 79.3 | 62.9 | 52.5 |  | 86.4 | 1525.1 |  |  |
| InternViT | ViT-6B | ～29B | ❌ | InternVL 1.5 | 20B | ~25M | ❌ | - | 65.7 | 63.5 | 94.0 | 88.3 | 1637 | 82.2 | 76.0 |

Tab. R2: Results of multimodal tasks evaluation. Revised from Tab. 6 in the paper.
| Vision pre-training method | Support interleave data | Vision encoder | Pre-training data | Seen samples | Data public | IN-1k frozen | IN-1k finetune | COCO ret. frozen TR@1 | COCO ret. frozen IR@1 | COCO cap. finetune CIDEr |
|---|---|---|---|---|---|---|---|---|---|---|
| Existing vision pre-training methods only support paired data |  |  |  |  |  |  |  |  |  |  |
| LCL (ours) | ✅ | ViT-B (86M) | LAION-400M | 2B | ✅ | 75.0 |  | 48.3 | 34.3 |  |
| *OpenCLIP | ❌ | ViT-B (86M) | LAION-400M | 2B | ✅ | 75.0 |  | 47.2 | 34.2 |  |
| *Reproduced CoCa | ❌ | ViT-B (86M) | LAION-400M | 2B | ✅ | 75.2 |  | 48.6 | 34.8 |  |
| Some MLLM training methods support interleaved data but do not work for vision pre-training |  |  |  |  |  |  |  |  |  |  |
| LCL (ours) | ✅ | ViT-B (86M) | MMC4 | 2B | ✅ | 75.2 |  | 48.5 | 34.5 |  |
| *BEiT3 | ✅ | ViT-B (86M) | MMC4 | 2B | ✅ | 73.3 |  | 45.1 | 30.6 |  |
| *OpenFlamingo | ✅ | ViT-B (86M) | MMC4 | 2B | ✅ | 24.0 |  | 10.6 | 5.6 |  |
| *Emu | ✅ | ViT-B (86M) | MMC4 | 2B | ✅ | 5.7 |  | 2.3 | 1.4 |  |
| Compare with existing checkpoints or reported results |  |  |  |  |  |  |  |  |  |  |
| LCL (ours) | ✅ | ViT-L (304M) | LAION-400M | 13B | ✅ | 82.2 | 86.1 | 59.6 | 46.2 | 129.7 |
| LCL (ours) | ✅ | ViT-L (304M) | LAION-400M + MMC4 | 13B | ✅ | 82.0 | 86.1 | 60.0 | 46.0 | 130.9 |
| OpenCLIP | ❌ | ViT-L (304M) | LAION-400M | 13B | ✅ | 82.1 | 86.2 | 59.5 | 46.0 | 128.9 |
| OpenAI CLIP | ❌ | ViT-L (304M) | WIT-400M | 13B | ❌ | 83.7 | 87.4 | 61.7 | 48.2 | 132.7 |
| EVA-02-CLIP | ❌ | ViT-L (304M) | $\dagger$ WIT-400M, LAION2B, etc | ~23B | ❌ |  | 90.0 |  |  |  |
| CoCa | ❌ | ViT-G (1B) | JFT-3B + ALIGN | 33B | ❌ | 90.6 | 91.0 |  |  | 143.6 |

Tab. R1: Comparison of different vision pre-training methods and models. Revised from Tab. 1, 2, 5 in the paper. * means reproducing or using open source code. $\dagger$ Training data of the teacher model is also considered.
We respond to the common concerns of the reviewers here.

First, we claim that our proposed method is the first to use interleaved image-text data to pre-train vision encoder from scratch. Existing vision pre-training methods do not support interleaved data. Simple ways to use interleaved data, e.g., autoregressive text generation, do not perform well for vision pre-training due to the vision feature collapse. We address this by our novel training objective that leverages both contrastive and generation loss.
___
**Q1:** Comparison with CoCa's objective.

**A1:** The major difference between our approach and CoCa's objective is that CoCa does not support interleaved data. For a detailed discussion, please refer to the response to Reviewer A7ig. In the revision, we will further elaborate on the differences between our method and CoCa, both in the figures and the textual descriptions.
___
**Q2:** Comparison with state-of-the-art methods.

**A2:** We would like to emphasize that we propose a vision pre-training method with image-text data. Accordingly, it is appropriate to benchmark it against SoTA vision pre-training methods that use image-text data. We suppose that achieving SoTA performance at the system level of MLLM is not a requisite for a vision pre-training method.
Below are two tables revised from the originally tables in our paper, which we will use to further explain our comparison with SoTA performance.

1. Existing vision pre-training methods only support paired data, and we compare with their SoTA performance on paired data. Due to the unavailability of some data, checkpoints, and source code, we have had to either reproduce existing methods or rely on available open-source implementations for a fair comparison (marked by * in Tab R1). The results in Tab. R1 show that our method is comparable to existing methods on paired data.

2. To show that using interleaved image-text data for vision pre-training is not trivial, we also consider existing ways to use interleaved data. Those methods are proposed as MLLM training methods, but we test them for vision pre-training. Tab R1 shows those methods do not work well for vision pre-training.

3. To further confirm our proposed method, we need to compare our pre-trained model with existing checkpoints. Tab. R1 shows that our pre-trained ViT-L/14 is on par or better than the OpenCLIP's checkpoint in a fair comparison. Other checkpoints listed in Tab. R1 could be the reference. They either use a larger vision encoder, more training data, or private data.

4. For multimodal benchmarks, as explained in the response to reviewer KZWZ, our goal is to fairly evaluate pre-trained vision encoders, not to compare different MLLMs at the system level. It is reasonable for us to adopt the open-source LLaVA-1.5. We also list some advanced MLLMs for reference in Tab. R2. Those MLLMs may outperform ours due to their use of stronger vision encoders, larger LLMs, or private data. We anticipate constructing a SoTA MLLM system in future work, but in the current stage, we focus on vision model pre-training due to resource constraints.
Thank the authors for their response. And most of my concerns have been well-addressed. I'd like to maintain the rating in the current stage.
Thank you for your discussion on data quality and the analysis of LCL generalization performance. I believe that the contextual relevance of interleaved data (i.e., data quality) has a far greater impact on model performance than the quality of paired data. Furthermore, in real-world scenarios, the generalization performance of the model is a key metric for evaluating pre-trained representations, so it would be meaningful to analyze it.

However, after reading the comments from other reviewers, I am concerned that the paper does not thoroughly discuss its relationship with other methods, especially regarding one of the core contributions of this paper—the pre-training objective. Additionally, as other reviewers have mentioned, further listing some state-of-the-art (SOTA) methods in the evaluation and clarifying how to achieve a fair comparison would further reduce misunderstandings for readers. Considering the above factors, I will somewhat lower my rating. Further discussion has the potential to affect my ratings.
Thanks for clarifying with respect to CoCa. I think given the similarities (e.g., Figure 1 of the manuscript and CoCa look very similar), it is important to make this comparison explicit. Also after reviewing other reviewers' comments, I am a bit concerned that state-of-the-art performance was not reported to contextualize the results. Additionally, many of the requested writing changes may require significant re-writing. Given this, I am electing to keep my score. Thanks to the authors for all of their effort in putting together the paper and the rebuttal.
Hi. Thank you for writing the rebuttal! I confirm I have read the rebuttal. I understand that the goal is to pre-train a vision encoder from scratch and the rebuttal sort of adds more clarity, but I think more details should be added in the paper to make things clearer. The authors have promised to add more details to the captions, so I don't have much to add here.

However, when reporting results on a well defined benchmark, I personally believe all types of methods should be included, especially if they have a better performance. I understand that the space is limited and I do not propose to have a list of extensive non-related paper, but I consider including at least 1,2 especially can be very important for readers of the paper in order to make them aware of different methods and possibly increased performance on the benchmark. As I said in the original review, what is fair or not can be shortly explained and the methods clearly separated in the table.