Title: Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: vision pre-training

compression learning

interleaved image-text data
Abstract: Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent on the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of a causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets (e.g., LAION), but can also leverage interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch, showcasing the potential of vision model pre-training with interleaved image-text data.

Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

Abstract

Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent on the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of a causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets (e.g., LAION), but can also leverage interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch, showcasing the potential of vision model pre-training with interleaved image-text data. Code is released at https://github.com/OpenGVLab/LCL .

1 Introduction

Over the past decade, ImageNet [34] pre-trained vision models have significantly advanced computer vision, continuously achieving breakthroughs in various vision tasks [7, 24, 10]. The success of ImageNet has inspired further exploration of better methods for pre-training vision models from scratch. Recently, the focus of pre-training has shifted from manually annotated data to large-scale, web-crawled image-text data. A key milestone in this shift is CLIP [55], which utilizes image-text pair data hundreds of times larger than ImageNet, delivering superior performance across various tasks and progressively becoming the mainstream method for vision model pre-training. Building on this trend, there is increasing interest in exploring interleaved image-text data, which is more prevalent on the Internet. Unlike the structured image-text pairs used in CLIP, this interleaved data is free-format and non-paired, larger in scale, and richer in textual information. Fully exploiting these interleaved image-text data is necessary for further improving vision model pre-training at scale.

Vision

Encoder

Text

Encoder

Image

Text

(a)

CLIPPre-training

Interleaved Image-Text

(c)

Multi-modal Incremental Training

Pre-trained

Vision

Encoder

Language

Model

Text Generative Loss

Interleaved Image-Text

(b)

Our Latent Compression Learning

Vision

Encoder

Language

Model

Text Generative Loss

Contrastive Loss

Vision-language Alignment

learning vision models from scratch

training on interleaved data

learning vision models from scratch

training on interleaved data

training on interleaved data

learning vision models from scratch

Discardable after Training

Frozen or Fine-tuned

Visual Embedding

Text Embedding

Currently, no pre-training method can effectively utilize interleaved image-text data to pre-train vision models from scratch. In preliminary attempts [2, 29, 44] of using interleaved data, vision models were already pre-trained by CLIP on paired image-text data. Subsequent training on interleaved data primarily serves to align the pre-trained vision models with language models, thereby enhancing the multi-modal capabilities of the entire vision-language network. Therefore, it remains an important and open problem of how to effectively learn robust visual representation from scratch on interleaved image-text data.

A recent study [30] in Natural Language Processing (NLP) suggests that the success of modern language models originates from the compression of training datasets into model parameters. We believe that such compression learning is also applicable to the multi-modal field, except that the data to be compressed expands from structured plain texts to interleaved image-text data, where the images are of raw pixels and unstructured. Such compression learning should be revised to accommodate to the image data. Raw pixels are unstructured and often contain unnecessary and unpredictable details. Such details are irrelevant to the high-level semantic tasks and should be discarded in compression learning. Thus, we argue that compression learning on interleaved text-image data should be applied to latent image representation to better extract semantic abstracts.

In this paper, we propose a novel visual pre-training framework, named Latent Compression Learning. We first theoretically demonstrate that effective latent compression learning can be performed by maximizing the mutual information between outputs and inputs of a causal attention model. When applied to visual pre-training on interleaved image-text data, visual latents are extracted through a visual encoding network (such as ViT [20]), and then fed together with the text into a causal model. The optimization objective can be derived and decomposed into two parts: 1) contrastive learning between visual latent representation and their previous context to enhance semantic consistency, and 2) auto-regressive prediction to learn the predictability of visual representation for subsequent text. These two training objectives complement each other. For images, the learned latent representation retain information that can be predicted from previous contexts and information needed for predicting subsequent contexts, thus providing effective visual pre-training.

In the experiments, various interleaved and paired pre-training methods are evaluated. The evaluation is conducted through transfer learning on multiple tasks, including image classification, image-text retrieval, image captioning, and visual dialogue. The pre-training datasets include the widely used image-text paired LAION-400M [57] and the image-text interleaved MMC4 [88] and Obelics [36]. In addition, we also re-organize an interleaved version of LAION-Random and a paired version of MMC4-Pair, to facilitate the comparison between interleaved and paired pre-training methods under the same data source. Experiment results show that our LCL pre-training can achieve the same

performance as CLIP on paired pre-training data and can better utilize interleaved pre-training data. Our results also demonstrate the effectiveness of using interleaved image-text data to learn robust visual representation from scratch, and the potential of compression learning for visual pre-training.

2 Related Work

Vision-centric Pre-training Methods. Supervised Pre-training on large-scale annotated datasets [24, 62, 48, 82] has remained the mainstream method for a long time, and has been favored by various vision tasks [9, 24, 79, 7] demonstrating strong performance. Self-Supervised Pre-training has gained significant popularity due to its advantage of utilizing unlabeled data. BEiT [6] follows the methodology of BERT [19] by randomly masking image tokens and reconstructing them as targets. MAE [27] and SimMIM [80] directly use masked pixels as reconstruction targets, making the pretraining process more straightforward and efficient. Weakly-Supervised Pre-training leverages imagehashtag [50, 76, 63] and image-text datasets [71, 61, 8, 57], which rely on noisy text supervision from the internet. For image-hashtag datasets, related works [50, 63] have shown comparatively good performance across various transfer learning settings. In the case of image-text datasets, early efforts [3, 42, 49, 64, 66, 67, 70, 13, 39] focused on learning general visual-linguistic representation. Recently, exemplified by CLIP [55], methods [55, 32] have been developed that involve pre-training through aligned visual-linguistic representation, achieving outstanding results in image classification tasks. Other works like M3I Pre-training [65] propose a unified framework that integrates multiple pre-training strategies with data from various modalities and sources. Currently, weakly supervised pre-training from web-scale text supervision has become the core of multi-modal understanding, but existing methods have not yet to leverage the most widespread interleaved image-text data for training visual representation from scratch.

Interleaved Image-Text Incremental Pre-training. Training using Interleaved Image-Text Data (IITD) has recently garnered significant attention due to the vast amount of such data available online. Recent works[87, 68, 69, 23, 4] such as Flamingo [2] and KOSMOS-1 [29] perform incremental learning on non-public IITD based on previously pre-trained vision and language model parameters as initialization, training text generation models with multi-modal understanding capabilities. With the continuous advancement in the field and the proliferation of public IITD ( e.g. , MMC4 [88] and OBELICS [36]), numerous models [83, 85, 14] capable of multi-modal understanding have emerged. However, these efforts only perform incremental pre-training on IITD and only analyze the usage of IITD on multi-modal dialogue models. Whether IITD contributes to learning robust visual representation from scratch remains unknown.

Compression Learning in NLP. The perspective [59, 60] that compression is closely connected to intelligence has a long history. A common compression method is arithmetic coding [56, 52], which is a practical approach that uses probability models for optimal data encoding. Some studies [28, 51, 38, 31, 37] argue that compression and intelligence are fundamentally equivalent. With the popularity of large language models, the equivalence of language modeling and compression has once again drawn widespread attention, prompting numerous explorations. Recently, [18] demonstrated through examples that language models serve as universal compressors. [30] posited that language modeling is equivalent to compressing a dataset into model parameters and proposed that compression efficiency is linearly related to model capabilities. Although compression learning has been proven effective in the field of NLP, it is not clear whether this approach can be extended to other fields.

Multi-modal Large Models. Text supervision pre-trained vision models [55, 32, 40] are widely utilized and have exhibited superior performance in tasks ranging from image retrieval and image classification to captioning. Currently, the most popular and closely watched application area is multimodal dialogue. Multi-modal dialogue models [2, 41, 86, 17, 25, 21, 53] primarily rely on a powerful pre-trained vision encoder [55, 22] and text decoder [16, 74, 75]. The usage of pre-trained vision encoder can generally be divided into two categories: the majority [2, 72], represented by LLaVA [46], employ a strategy where the pre-trained vision encoder is frozen, and only the subsequent adapters and language models are trained. A minority, exemplified by Qwen-VL [5, 53], utilize high-quality image-text dialogue data to continue fine-tuning the vision model. These two training strategies correspond to the two evaluation methods used in this paper to assess the performance of vision models.

3 Method

3.1 Latent Compression Learning

Auto-regressive Language Modeling as Compression Learning. Recent works [18, 30] have shown that auto-regressive language modeling is equivalent to compression learning. Suppose g ϕ is a language model (LM) with learnable parameters ϕ . Given an input text sequence x = ( 
 , x 1 , x 2 , . . . , x N ) , where 
 is a special token indicating the beginning of text, the model outputs y = g ϕ ( x ) = ( y 1 , y 2 , . . . , y N ) predicting the next token based on preceding context, i.e. , ˆ x k = y k = g ϕ ( x ) k . The approximate probability of x estimated by g ϕ is q ( x ) = ∏ N k =1 q ( x k | y k = g ϕ ( x ) k ) . The model is optimized with NLL-loss, which equals to minimizing the the cross-entropy between the data distribution p and model distribution q :

Notice that H ( p, q ) is actually the optimal expected code length encoding p by q , minimizing H ( p, q ) just means compressing the data into the model parameters.

Latent Compression for Interleaved Image-Text Data. We believe that the compression principle could apply to multi-modal domain, specifically, to train vision-language models by compressing interleaved image-text data. However, instead of directly dealing with pixel values, we turn to compress high-level image representation for the following reasons: 1) high-level representation can extract useful information from raw pixels while discarding those unpredictable image details. 2) the learned visual representation will align with text semantics, making it possible to perform effective visual pre-training with interleaved image-text data.

Specifically, let x = ( 
 , x 1 , x 2 , . . . , x N ) be an interleaved image-text sequence. To simplify the expression without loss of generality, we assume that there is only one image in the sequence. Sub-sequence x i : i + M are M + 1 image patch tokens of the input image (e.g., non-overlapping patches in ViTs) and others are text tokens. I = { i, i +1 , . . . , i + M } denotes the indices of the image patches, and T = { 1 , . . . , i -1 , i + M +1 . . . , N } denotes the indices of text tokens. As shown in Fig. 2, to construct the sequence of latent representation z = ( 
 , z 1 , z 2 , . . . , z N ) , for image patches, we use a parametric vision encoder f θ (e.g., ViTs) to map the data sequence x i : i + M into latent variable z i : i + M . For text tokens, we directly use one-hot vectors corresponding to their vocabulary ids as the latent codes. Then, the latent representation z are fed into a causal attention model g ϕ for latent compression by minimizing

where the k -th element of the output y k = g ϕ ◦ f θ ( x ) k predicts the next input latent z k , f θ is identity for text tokens for simplicity of annotation. When the compression only applied on text tokens, it degenerates to auto-regressive language modeling on interleaved image-text data used by previous methods (e.g., Kosmos and Flamingo).

However, direct optimizing Eq. (2) for learning informative latent representation is non-trivial, since Eq. (2) suffers from a naturally trivial solution of visual representation collapse, i.e. the image latent representation z i : i + M may be learned to be data-independent. In fact, as showed in Sec. 4.2, we have observed such visual representation collapse, when training from scratch on MMC4 dataset with Eq. (2) applied to text tokens only (i.e., auto-regressive language modeling).

Maximizing Mutual Information for Latent Compression Learning. Optimizing directly for latent compression in Eq. (2) may cause the visual representation collapse. A natural constraint is to maximize the representation entropy to prevent collapse. We find that combining latent compression and maximum entropy constraint is exactly equivalent to maximizing the mutual information between the model inputs and outputs.

Prior work [65] have shown the relationship between cross-entropy and mutual information in other pre-training tasks. Here, we derive this relationship in the latent compression task: maximizing the mutual information between the output y and the input latent z of the causal attention model g ϕ is

equivalent to compressing z by minimizing H ( p, q ) in Eq. (2) meanwhile maximizing the entropy of each element z k in z :

where we use p ( y k , z k | x ) = p ( y k | x ) p ( z k | x ) in the first step since z k and y k can be independently computed given input x , and p ( z k | y k ) is estimated by an approximate parameterized distribution q ( z k | y k ) . For the derivation of the formula, please refer to [65].

Therefore, using I ( y ; z ) as the optimization objective can achieve latent compression while avoiding representation collapse of z via the maximum entropy constraint. The compression of z imposes the model to extract useful information and discard unpredictable information of the image. Meanwhile, maximizing I ( y ; z ) requires that each y k could obtain enough information from previous latent z <k to predict z k . Each z k should carry predictable information. These guarantee that the image representation encode rich semantic information aligned with text. We suppose that the above properties learned by the image representation are desired for vision-language pre-training , thus we use Eq. (3) as our pre-training objective. Parameters ϕ and θ are be jointly optimized under this objective. Intuitively, the vision encoder f θ learns to represent images by high-level abstract, and the causal attention model g ϕ learns to compress this high-level abstract of the dataset.

3.2 Training Loss

In this sub-section, we demonstrate how Eq. (3) is decomposed into training tasks and losses. Firstly, I ( y ; z ) can be decomposed as a cross-entropy term and an entropy term in the following two symmetric ways (see Appendix B for detailed derivation):

Since given the input x , latent z k and y k are independent and deterministic ( i.e. , determined by f θ and g ϕ ), yielding p ( y k , z k | x ) = δ [ z k = f θ ( x ) k ] · δ [ y k = g ϕ ◦ f θ ( x ) k ] . δ [ · ] is delta distribution. In Eq. (4), p ( z k | y k ) is estimated by a parameterized distribution q 1 ( z k | y k ) , which approximates the distribution of z k given the model's prediction y k . Similarly, in Eq. (5), p ( y k | z k ) is estimated by q 2 ( y k | z k ) , the predicted distribution of y k given z k . Therefore, maximizing mutual information can be decomposed as follows: 1) The causal attention model g ϕ learns to predict the next latent z k from the output y k . 2) The learnable latent representation z k learns to predict y k , which is the representation of its previous context. 3) The maximum entropy regularization avoids the collapse of z k and y k .

In the following, we show that the cross-entropy terms in I ( y ; z ) can be achieved by two common training tasks and loss functions, while the entropy constraints are implicitly satisfied.

Contrastive Learning between Image Representation and Preceding Context. For image latent z k and the corresponding y k representing the semantics of its preceding context, the objective defines a bidirectional prediction. We choose q as Boltzmann distribution, i.e. , q ( z k | y k ) ∝ exp( z ⊤ k W ⊤ 1 W 2 y k /τ ) and q ( y k | z k ) ∝ exp( y ⊤ k W ⊤ 2 W 1 z k /τ ) , where τ is the temperature, W 1 and W 2 are learnable linear projections. Consequently, the objective becomes the contrastive loss in two directions between z k and y k , when setting z k ′ and y k ′ from other images as negative samples:

𝑔𝜙

Vision

Encoder

𝑓 𝜃

𝑣𝑖1

𝑡

𝑖1

Auto-regressive Text

Generation Loss

𝑳𝒈𝒆𝒏

Contrastive Loss

𝑳𝒄𝒐𝒏

𝑣𝑖

Image global representation

𝑡

𝑖

Preceding context embedding

A woman riding

a

bike

riding

a

bike

A

woman

<BoI>

<BoI>

<EoI>

…

…

street

<BoI>

𝑣𝑖2

Contrastive Loss

𝑳𝒄𝒐𝒏

𝑡

𝑖2

…

<EoI>

<BoI>

on

the

…

street

the

There

Image latent representation

𝑧𝑖:𝑖+𝑀

𝑧𝑖2:𝑖 2+𝑀

Meanwhile, the contrastive loss also prevents z k and y k from being trivial representation by pulling them away from negative samples, implicitly appending the entropy regularization.

Auto-regressive Text Generation. For a text token, its latent code z k is a one-hot vector and is not learnable, so the objective only imposes y k to predict z k as in Eq. (4). We choose q ( z k | y k ) as softmax over the output logits on the text vocabulary, i.e, q ( z k | y k ) = z ⊤ k softmax ( V y k ) , where V is the projection head of the language model. The objective corresponding to the text tokens is simply the objective of standard next token prediction with cross-entropy loss:

The total training loss is defined as L = λL con + L gen , where λ is balancing weight.

Relation to Previous Pre-training Tasks. In our proposed pre-training framework, the contrastive task and generation task align the image representation with preceding context and subsequent context, respectively. Hence, combining these two pre-training tasks can fully leverage the semantic information contained in interleaved image-text data to supervise the learning of image representation. For previous pre-training tasks, 1) Contrastive Language-Image Pre-training (CLIP) [55] has a similar objective of maximizing the mutual information between corresponding image and text [65], but it can only be applied to paired image-text data. 2) Contrastive Captioner (CoCa) [81] combines a captioning (text generation) loss with the CLIP loss, but it cannot be applied to interleaved data, either. CLIP loss requires image-text pairs, while the captioner can only be conditioned on a single image can rather than flexible interleaved image-text contents. 3) Auto-regressive Text Generation task only leverages the semantic information in subsequent context to supervise the learning of image representation, while the preceding context is missing, and representation collapse cannot be avoided. Moreover, interleaved image-text data usually has information redundancy, i.e. , the image and its corresponding text may contain similar information. So models may rely on information from the text rather than the image for prediction. This is particularly true when the vision encoder is trained from scratch, resulting that the image representation are never focused and optimized. Our experiments in Sec. 4.2 confirms this analyze.

3.3 Architecture

The overview of model architecture when adopting our LCL is shown in Fig. 2. The interleaved image-text input sequence may contain multiple images. We adopt a Vision Transformer (ViT) [20] as the vision encoder, which encodes each image into a sequence of visual embeddings as its latent representation. The visual embeddings of each image are inserted at corresponding positions in the

interleaved sequence, and we introduce special tokens <BoI> and <EoI> to indicate the beginning and the ending positions, respectively. The combined visual and text embeddings are fed into a causal language model. The text generation loss is the standard cross-entropy loss over the output logits of the language model defined in Eq. (7). In the contrastive learning task, calculating the loss for each image token is extremely computationally expensive. To alleviate this problem, we consider utilizing one global representation per image instead of all latent representation for contrastive learning. Specifically, the global representation v i of each image is extracted from its latent representation z i : i + M through an attention pooling, which is a multi-head attention layer with a learnable query token. The output of the causal transformer model at <BoI> token, just before the image, is processed by a LayerNorm layer and a linear projection into t i . The contrastive loss in Eq. (6) is calculated between v i and t i .

4 Experiment

4.1 Experiment Settings

Pre-train Data. The datasets utilized in our pre-training encompass the image-text pair dataset LAION-400M [57], as well as the image-text interleaved datasets MMC4 [88] and OBELICS [36]. We also re-organize two datasets for fair comparison with CLIP. 1) LAION-Random as a interleaved dataset. Images from LAION-400M are randomly placed before or after their paired caption to form image-text sequences. 2) MMC4-Pair as a paired dataset. For images in MMC4, we select matching text from the bipartite graph matching results derived from CLIP similarity to generate pseudo image-text paired data.

Implementation Details. We adopt the same image transform in OpenCLIP [15] for pre-training and set the image size as 224 × 224 for all experiments. We employ ViT [20] as our vision encoder. ViT-L/14 is used for the main results, and ViT-B/16 is used for ablation studies. The language model follows the same architecture as OPT-125M [84] but is randomly initialized. By default, the contrastive loss balancing weight is set at λ = 0 . 1 . AdamW optimizer with β 1 = 0 . 9 , β 2 = 0 . 95 and a weight decay of 0.1 are used. We employ a cosine learning rate schedule with linear warmup and set the peak learning rate at 5e-4 for the LAION data and 3e-4 for others. The model for the main results is trained for 200k iterations with 64k images per batch on average. In the ablation study, the models are trained for 250k iterations with 8k images per batch.

Vision Encoder Evaluation. CLIP pre-trained models typically uses zero-shot retrieval performance for evaluation. However such zero-shot inference on retrieval tasks [55] is not suitable for other generative pre-training methods. To address this discrepancy, we choose to evaluate pre-trained vision models by transfer learning on various downstream tasks under two configurations: 'frozen transfer' and 'full transfer'. In the frozen transfer setting, only the parameters outside of the vision model are trained. In the full transfer setting, all parameters are trained.

The evaluation is conducted on image classification, image-text retrieval, image caption, and multimodal dialogue. Please see Appendix A for more data, implementation, and evaluation details.

4.2 Comparison with Vision Pre-training Methods

In this section, we show the superiority of our LCL when using interleaved image-text data for vision pre-training. However, existing vision pre-training methods only support paired data, so we choose MMC4 dataset with a paired set (MMC4-Pair) available for those methods. In addition, to confirm that it is is not trivial to use interleaved data for vision pre-training, we also consider existing methods to use interleaved data. Those methods are originally proposed as MLLM training methods, but here we test them for vision pre-training. Besides, on paired image-text data, we also compare our LCL with existing vision pre-training methods and show they are comparable. See Appendix A.3 and A.2 for detailed pre-training and evaluation settings.

We aggregate all the pre-training methods divide them into the following tasks: (1) Image-text contrastive (Con.) [55, 15, 32], (2) Image-text contrastive + image captioning (Con. + Cap.) [81], (3) Image-text contrastive + image captioning + image-text matching (Con. + Cap. + Mat.) [40, 41, 14], (4) Auto-regressive text generation (Gen.) [2, 29, 44], (5) Auto-regressive text generation + image regression (Gen. + Reg.) [69, 68, 72], (6) Mask data modeling (Mask.) [78]. We select one

representative method from each task. For fair comparison, we implement these methods on the same model and training data with available open-source codes or our reproduction.

Pre-training on Interleaved Image-Text Data. We conduct experiments on the MMC4 dataset [88] to demonstrate the effectiveness of our LCL on interleaved image-text data. For pre-training methods that only support paired data, MMC4-Pair is used.

The results are shown in Tab. 1. Our LCL pre-training method significantly outperforms all other methods in the caption tasks, indicating that we can effectively utilize the rich text context information in MMC4 interleaved data. On the other hand, our method is on par with the best paired pre-training methods on classification and retrieval tasks. Since the paired pre-training methods are directly optimized for retrieval, our comparable performance shows that the visual features are learned to be highly distinguishable. It is worth mentioning that for more general interleaved data, where no paired versions exist, these paired pre-training methods cannot be applied.

In addition, we observed that the two methods using auto-regressive text generation do not achieve good performance and feature collapse occurs. However, their text prediction training loss is actually close to ours. This suggests that these methods tend to rely on redundant text information rather than image information for subsequent text prediction. As discussed in Sec. 3.1, our approach can avoid such collapse.

Pre-training on Paired Image-Text Data. We conduct experiments on the LAION-400M dataset to show that LCL pre-training also performs well on paired image-text data without specific modification. Tab 2 shows that our method is comparable to paired pre-training methods on various tasks, indicating that all information in paired data is fully exploited.

4.3 Comparison with Pre-trained Checkpoints

To further confirm the effectiveness of our proposed Latent Compression Learning (LCL), we compare our pre-trained model with existing checkpoints of pre-trained vision encoders. We use LCL to pre-train a ViT-L/14 with mixed data from the LAION-400M and MMC4. We compare it to the ViT-L/14 pre-trained by OpenCLIP [15] using the public LAION-400M dataset, and the ViT-L/14 pre-trained by OpenAI CLIP [55] with private data is listed as reference. The total number of images seen during pre-training is 13B for all models. We evaluate the pre-trained vision encoders by transferring to downstream tasks, i.e. , integrate the vision encoders into downstream task models and compare the fine-tuning results. More training details are in Appendix A.1 and evaluation details are in Appendix A.2.

Tab. 3 and Tab. 4 show the results of transfer evaluations. When both use LAION-400M as pretraining data, as with the previous experimental conclusions, LCL has similar performance to CLIP. When combined with MMC4, our method achieves better performance, especially on caption and multi-modal dialogue tasks.

There exist approaches achieving better results on some benchmarks. However, they either use larger vision encoders, more training data, or private data. We list those results as reference in Appendix C.

4.4 Ablation Study

Latent Compression Learning on Different Datasets. We apply LCL pre-training to more datasets to confirm its generalizability. As shown in Tab. 5, our method also achieves reasonable performance on interleaved dataset OBELICS. It is worth noting that the models trained on MMC4 and OBELICS have achieved similar performance to that on LAION, indicating that it is completely feasible to pre-train visual models only from interleaved data. Furthermore, using both LAION and MMC4 data during pre-training improves performance, suggesting that further improvements can be obtained by incorporating more image-text data. In this case, supporting interleaved data is a key advantage of our approach, enabling the use of more diverse image-text data for pre-training.

(a) Training loss ablation.

(b)

Loss balancing weight ablation.

Loss Balance in Latent Compression Learning. Table 6a ablates the contrastive loss and generation loss used in LCL . Consistent with the previous analyses, LCL can achieve the best performance. Table 6b studies the appropriate loss balancing weights (multiplied by the contrastive loss). It turns out that λ = 0 . 1 will produce the best results. The performance drops significantly for larger λ values, indicating that the optimization directions of the two losses are not completely consistent.

5 Conclusion

No existing work has explored vision model pre-training with interleaved image-text data. To this end, we propose Latent Compression Learning (LCL) framework that compresses interleaved image-text latent for vision pre-training. We theoretically show that latent compression is equivalent to maximizing the mutual information between the input and output of a causal model and further decompose this objective into two basic training tasks. Experiments demonstrate that our method is comparable to CLIP on paired pre-training datasets, and it effectively learns robust visual representations utilizing interleaved image-text data. Our work showcases the effectiveness of using interleaved image-text data to learn robust visual representation from scratch, and confirms the potential of compression learning for visual pre-training.

Limitations . Our experiments are constrained to a limited size of dataset and vision encoder, and the scaling property of our proposed method remains unexplored.

Acknowledgments.

This work is supported by the National Key R&D Program of China (NO. 2022ZD0161300), by the National Natural Science Foundation of China (62376134).

References

H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8948-8957, 2019.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.

C. Alberti, J. Ling, M. Collins, and D. Reitter. Fusion of detected objects in text for visual question answering. arXiv preprint arXiv:1908.05054 , 2019.

A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023.

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.

H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pretraining to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3558-3568, 2021.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062 , 2014.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015.

X. Chen, M. Ding, X. Wang, Y. Xin, S. Mo, Y. Wang, S. Han, P. Luo, G. Zeng, and J. Wang. Context autoencoder for self-supervised representation learning. International Journal of Computer Vision , 132(1): 208-223, 2024.

Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104-120. Springer, 2020.

Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 , 2023.

M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2818-2829, 2023.

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research , 25(70):1-53, 2024.

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems , 36, 2024.

G. Delétang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10668 , 2023.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19358-19369, 2023.

Y. Ge, S. Zhao, Z. Zeng, Y. Ge, C. Li, X. Wang, and Y. Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218 , 2023.

R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 580-587, 2014.

T. Gong, C. Lyu, S. Zhang, Y. Wang, M. Zheng, Q. Zhao, K. Liu, W. Zhang, P. Luo, and K. Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790 , 2023.

Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

J. Hernández-Orallo and N. Minaya-Collado. A formal definition of intelligence based on an intensional variant of algorithmic complexity. In Proceedings of International Symposium of Engineering of Intelligent Systems (EIS98) , pages 146-163, 1998.

S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems , 36, 2024.

Y. Huang, J. Zhang, Z. Shan, and J. He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937 , 2024.

M. Hutter. The human knowledge compression prize. URL http://prize. hutter1. net , 2006.

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 , 2016.

H. Laurençon, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karamcheti, A. Rush, D. Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems , 36, 2024.

S. Legg and M. Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines , 17:391-444, 2007.

S. Legg, M. Hutter, et al. A universal measure of intelligence for artificial agents. In International Joint Conference on Artificial Intelligence , volume 19, page 1509. LAWRENCE ERLBAUM ASSOCIATES LTD, 2005.

G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11336-11344, 2020.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified visionlanguage understanding and generation. In International conference on machine learning , pages 1288812900. PMLR, 2022.

J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019.

Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2339023400, 2023.

J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533 , 2023.

H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 , 2023.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281 , 2023.

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019.

D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV) , pages 181-196, 2018.

M. V. Mahoney. Text compression as a test for artificial intelligence. AAAI/IAAI , 970, 1999.

R. C. Pasco. Source coding algorithms for fast data compression . PhD thesis, Stanford University CA, 1976.

Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 , 2023.

B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pages 2641-2649, 2015.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

J. J. Rissanen. Generalized kraft inequality and arithmetic coding. IBM Journal of research and development , 20(3):198-203, 1976.

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

C. Schuhmann, A. Köpf, R. Vencu, T. Coombes, and R. Beaumont. Laion coco: 600m synthetic captions from laion2b-en. URL https://laion. ai/blog/laion-coco , 2022.

C. E. Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3): 379-423, 1948.

C. E. Shannon. Prediction and entropy of printed english. Bell system technical journal , 30(1):50-64, 1951.

P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alttext dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems , 27, 2014.

M. Singh, L. Gustafson, A. Adcock, V. de Freitas Reis, B. Gedik, R. P. Kosaraju, D. Mahajan, R. Girshick, P. Dollár, and L. Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 804-814, 2022.

W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 , 2019.

W. Su, X. Zhu, C. Tao, L. Lu, B. Li, G. Huang, Y. Qiao, X. Wang, J. Zhou, and J. Dai. Towards all-in-one pre-training via maximizing multi-modal mutual information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15888-15899, 2023.

C. Sun, F. Baradel, K. Murphy, and C. Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.05743 , 2019.

C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7464-7473, 2019.

Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu, T. Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286 , 2023.

Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations , 2023.

H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 , 2019.

B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

C. Tian, X. Zhu, Y. Xiong, W. Wang, Z. Chen, W. Wang, Y. Chen, L. Lu, T. Lu, J. Zhou, et al. Mminterleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401.10208 , 2024.

H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 32-42, 2021.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

A. Veit, M. Nickel, S. Belongie, and L. Van Der Maaten. Separating self-expression and visual content in hashtag supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5919-5927, 2018.

T. F. Y. Vicente, L. Hou, C.-P. Yu, M. Hoai, and D. Samaras. Large-scale training of shadow detectors with noisily-annotated shadow examples. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14 , pages 816-832. Springer, 2016.

W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV) , pages 418-434, 2018.

Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9653-9663, 2022.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12104-12113, 2022.

P. Zhang, X. D. B. Wang, Y. Cao, C. Xu, L. Ouyang, Z. Zhao, S. Ding, S. Zhang, H. Duan, H. Yan, et al. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112 , 2023.

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

H. Zhao, Z. Cai, S. Si, X. Ma, K. An, L. Chen, Z. Liu, S. Wang, W. Han, and B. Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915 , 2023.

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

J. Zhu, X. Ding, Y. Ge, Y. Ge, S. Zhao, H. Zhao, X. Wang, and Y. Shan. Vl-gpt: A generative pre-trained transformer for vision and language understanding and generation. arXiv preprint arXiv:2312.09251 , 2023.

W. Zhu, J. Hessel, A. Awadalla, S. Y. Gadre, J. Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang, and Y. Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems , 36, 2024.

A Experimental Details

A.1 Pre-training

Data. For the data in MMC4, we select images with a CLIP similarity to the matching text of 0.24 or higher. From documents containing at least one such image, we randomly choose up to 6 images to form an interleaved image-text sequence, utilizing all text from that document. If the sequence length exceeds 2048 tokens, the surplus is truncated while ensuring the integrity of both images and individual text segments, and then padded to the designated length. For OBELICS, we similarly restrict the number of images per document to between 1 and 6. We then sequentially extract 2048 tokens from the concatenated documents. If image tokens are truncated, the entire image is moved to the next sample sequence.

To construct interleaved image-text samples from the MMC4 dataset, we randomly place images either before or after their corresponding sentences, adhering to a 50% probability, thus generating a document-wise interleaved sequence of images and text. For the OBELICS corpus, individual documents are concatenated, and a sliding window strategy is employed to select each image-text sequence, maintaining a total length of 2048 tokens.

Hyper-parameters. Our pre-training configuration is shown in Tab. 7. The AdamW optimizer was employed for model training with the learning rate set to 3e-4 and the weight decay set to 0.1. Mixed numerical precision training with bfloat16 is also employed to stabilize the optimization process. Furthermore, we set a drop-path [35] rate linearly increasing to 0.2, and use layer-scale [73] for stable training.

A.2 Evaluation

Transfer Tasks. We conduct our performance evaluation of pre-trained on image classification, image-text retrieval, text generation tasks with multimodal inputs ( i.e. , image captioning and multimodal dialogue). Their model architecture in transfer learning are illustrated in Fig. 3.

For closed-set image classification, a lightweight classifier with a randomly initialized attention pooling layer, followed by a layer normalization layer and a linear layer, is appended to the top of the pre-trained vision model. In the 'frozen transfer' scenario, only the parameters of the added classifier are trainable, similar to the linear probing strategy. Conversely, in the 'full transfer' approach, all parameters, including those of the pre-trained vision encoder, are adjustable.

Regarding the image-text retrieval task, we discard the pre-trained text encoder and apply contrastive learning to the pretrained vision encoder with a newly introduced text encoder. Images are processed through the vision encoder and a randomly initialized attention pooling layer to generate a global image embedding. Textual captions are processed through the text encoder, utilizing the feature of the final token as the text embedding representing the input caption. An additional linear layer facilitates the dimensional alignment between the image and text embeddings, enabling their use in contrastive learning. During 'frozen transfer', the attention pooling layer on the vision transformer and the text encoder are trainable; similarly, in 'full transfer', all parameters, including the vision encoder, can be optimized.

Vision

Encoder

Vision

Encoder

Text

Encoder

Linear

Vision

Encoder

Language

Model

MLP

Similarity

Image

Image

Text

Image

Text

(a) Image Classification

(b) Vision-Language Retrieval

(c) Text Generation

Text

Class Logit

Linear

For text generation tasks with multimodal inputs, we adopt image captioning and multi-modal dialogue benchmarks and employ prevalent architectures like those in [46]. Specifically, a pretrained LLM for text generation is integrated on top of the pre-trained vision model, incorporating an MLP layer to adjust the dimensions of the visual embeddings. During the 'frozen transfer' evaluation, the parameters of the vision model remain fixed, while in the 'unfreezing' phase, these parameters are permitted to undergo training.

Implementation Details. Vision encoder evaluation with 'frozen transfer' and 'full transfer' configurations includes fine-tuning training and benchmark evaluation. Implementation details of each transfer task are list below, and the hyper-parameters involved are listed in Tab. 8.

Image classification. Model is trained on the ImageNet-1K [33] train split and evaluated on val split. We follow the attention probe setting introduced by [12] for 'frozen transfer', and the full fine-tune setting in [43] for 'full transfer'.

Image-text retrieval. Model is trained on a combination dataset comprised of CC12M [61], CC3M [61], and SBU [77], and is tested on the MSCOCO [11] karpathy-test split and Flickr30k [54] test split. The model is trained with Adamw optimizer for 5000 iterations. The learning rate is set at 1e-3 and 1e-5 for parameters without initialization and with initialization, respectively.

Image captioning. Model is trained on a subset of the LAION-COCO [58] dataset, which includes 10 million samples, and evaluation is performed on the MSCOCO [11] karpathy-test split and NoCaps [1] val split. Here, the model is trained for 20,000 iterations with a learning rate of 1e-4. Additionally, a droppath technique is employed with a ratio of 0.2 in the vision model to mitigate overfitting.

Multi-modal dialogue. We follow a two-stage training process similar to that used in LLAVA1.5 [45]. Initially, paired data with 558K samples is used to train an MLP projector to align the Vision Transformer (ViT) with the pretrained LLM. Subsequently, the model undergoes instruction tuning on multimodal dialogue datasets with 665k samples. Both the alignment training and instruction tuning phases are conducted over a single epoch, with learning rates set at 1e-3 and 2e-5, respectively. Evaluations are then performed on multimodal dialogue benchmark, e.g. , MMBench [47] and VQAv2 [26].

A.3 Ablation Experiments

The effectiveness of our LCL are validated by conducting ablation experiments mainly on two corpora: LAION and MMC4. The experimental hyper-parameters involved are shown in Tab. 9. We found that the optimal learning rate for the LAION dataset is 5e-4, while for the MMC4 dataset, a slightly lower

rate of 3e-4 proves most effective. We speculate that this is because MMC4 corpus contains relatively higher noise. Most of the original settings in the large-scale pre-training are retained in the ablation, with the exception of reducing the batch size by a factor of 8 to decrease the computational overhead.

A.4 Experiments Compute Resources

Pre-training used 512 A800 GPUs and took 5 days.

B Theoretical derivation details

Using the notation defined in Sec. 3, the mutual information of the output of the language model ( y ) and the latent representation ( z ) can be described as follows:

We can derive Eq. 9 from Eq. 8 because once the interleaved image-text input sequence x is given, the output y k and the latent representation z k can be computed independently:

From Eq. 9, we can further decompose the mutual information into a cross-entropy component and an entropy component in two symmetric ways. One approach involves the output token y k predicting the next latent representation z k , along with the entropy of z k :

The other approach considers how the latent representation z k approximates the previous context y k and includes the entropy of the output y k :

Note that the reason for transitioning from Eq. 10 to Eq. 11 and from Eq. 12 to Eq. 13 is because, given the input sequence x , the outputs y k and z k are determined as follows:

C Supplementary Benchmark Results

We present supplementary results on classification, retrieval and captioning tasks (Tab. 10) and multi-modal benchmarks (Tab. 11).

D Broader Impacts

This work may share the common negative impacts of large-scale vision training. The data used in pretraining may contain dataset bias, and raise ethical concerns. It may also require large computational resources, which consume lots of electricity and result in increased carbon emissions.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please see Sec 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Sec 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please see Sec 3.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details of pre-training and evaluation can be found in 4.1 and Appendix A Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code has been released at https://github.com/OpenGVLab/LCL . Only pulic datasets have been used in our research.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Most experiments have stable results with little variance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please see Appendix D.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release a novel dataset or model.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original paper each time a new asset appears.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a vision backbone pre-training method named Latent Compression Learning (LCL) to utilize interleaved image-text data. The proposed LCL approach maximizes mutual information between the inputs and outputs of a GPT-like model in autoregressive manner. The proposed method integrate both discriminative and generative objectives by contrasting preceding context and generate subsequent text based on visual representation. The extensive experiments demonstrate that LCL not only matches the performance of existing models like CLIP on paired datasets (e.g., LAION) but also effectively leverages interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch.\n\nSTRENGTHS:\n1. The paper is well written and easy to follow.\n\n2. The paper introduces a new pre-training method, Latent Compression Learning (LCL), which utilizes interleaved image-text data for visual backbone pre-training for the first time. And this can effectively leveraging large-scale web-crawled data, which is easier to crawl compared to the image-text pairs.\n\n3. Extensive experiments are conducted, demonstrating the effectiveness of the proposed method on both paired datasets (e.g., LAION) and interleaved datasets (e.g., MMC4).\n\nWEAKNESSES:\n1. From Table 5, it appears that solely leveraging image-text pairs with LCL does not provide benefits over the CLIP baseline. However, when using the MMC4 dataset, which is manually composed of interleaved text, there is significant performance improvement on downstream tasks. I am curious whether this performance gain results from the increased number of training samples (i.e., the total number of images used during training).\n\n2. According to Table 3, utilizing original interleaved datasets such as Obelics does not yield any performance gain. In comparison, the MMC4 dataset requires more computation for data filtering with the CLIP score and the use of image-text pairs to create interleaved data. It is unclear how to efficiently utilize the original interleaved data directly crawled from the web. Do you have any insights on the differences between these two types of interleaved datasets?\n\nQUESTIONS:\n1. The scaling behavior is not demonstrated. While the author has shown the effectiveness of training on MMC4 and Laion-400M, it remains unclear how the model performs and correlates across different dataset scales. Understanding this could provide valuable insights into the feasibility and performance of scaling the proposed method to larger datasets, such as DataComp 12.8B and Laion 2B.\n\n2. Can you show the seen samples of each model? It would be helpful for readers understanding the scale of model's training.\n\nLIMITATIONS:\nThe authors have addressed the limitation in their manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of vision model pre-training. More exactly, it aims to exploit the interleaved image-text data that is very prevalent on the Internet. It proposes Latent Compression Learning that maximises the mutual information between the inputs and outputs of a causal attention model. When visual pre-training is applied to interleaved image-text data, visual latents are extracted using a visual encoding network and then combined with the text and fed to the causal model.\n\nSTRENGTHS:\nThe paper tackles an important task and proposes an interesting method that may be of interest to the research community.\n\nWEAKNESSES:\nWhile the method seems interesting, my main concern is related to the experimental part that I find confusing. For example for BEiT3 the numbers reported are different from the ones reported in the paper.\n\nAlso, I think that for Tab 6, more multi-modal LLMs need to be included. While there can be a debate on fair vs unfair comparison, I think that you present results on a dataset these need to be complete. So, they can be greyed out, put in a different section, etc and explained why the comparison is not fair, but I don't think it's suitable for models that perform better to not be included at all. So, missing comparisons:\n\nFang, Yuxin, et al. \"Eva: Exploring the limits of masked visual representation learning at scale.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\nZou, Xueyan, et al. \"Generalized decoding for pixel, image, and language.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nor even some very recent ones for the sake of completeness:\nSun, Quan, et al. \"Generative multimodal models are in-context learners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\nLiu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nWhy is OpenAI CLIP greyed out?\n\nWhy are the numbers reported for BEiT3 different than what they report in the paper? For example for Flicker 3K, R@1, it's reported 73.2 while the paper reports 88.2\n\nWhy, for example the comparison with BEIT3 is only shown in Tab.1 when they report results on VQAv2? The same question for CoCA?\n\nLIMITATIONS:\nLimitations are barely discussed at the end of the conclusions. Some limitations can be inferred from the rest of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper pre-trains models with a combination of a contrastive image-text objective and a generative language objective. The authors provide many results on image classification and vision-language tasks suggesting the competitiveness of the method in controlled settings.\n\nSTRENGTHS:\nS1. The paper is well framed and motivates nicely the need to pre-training on interleaved data.\n\nS2. The paper gives good intuition about what the various equations mean, making the manuscript more accessible.\n\nS3. Consideration of many pre-training datasets including LAION-400M, MMC4, and OBELICS.\n\nS4. Extensive objective ablations spanning both contrastive and generative losses.\n\nWEAKNESSES:\nW1. [MAJOR] The paper presents the objective as novel (L44-54); however, it seems similar to CoCa (Yu et al., 2022.), which also employs a contrastive loss and a next token prediction loss. Can the authors clarify the differences and why the formulation is novel?\n\nW2. It seems equation 3 appears in prior work; however, when it is first presented it seems to be presented as a novel insight. I recommend making the attribution to prior work more clear before introducing the equation.\n\nW3. In the relation to previous pre-training tasks, it is important to also relate to CoCa. It seems the objective is pretty much the same suggesting that the objective is not actually a contribution of the work. Is there any reason CoCa is not mentioned here given the similarities?\n\nW4. Make sure it is clear that you train on sequences with more than one image per sample (I am assuming this is true because you train on MMC4, but when explaining the objectives you include only one sequence for simplicity). 3.3 is a nice place to add this information. Also any special tricks to get multi-image to work? If so, it could also be nice to mention this.\n\nW5. Why are the numbers for Flamingo in Tab 1 for IN-1k so low? Flamingo uses a pre-trained vision backbone, so I expect numbers to be good here.\n\nW6. Is the COCO CIDEr evaluation protocol zero-shot? If so the number in table 4(a) of 87.5 looks extremely high relative to open flamingo and Idefics. Please double check this number and if few-shot prompting is used here, please make this clear. Also why is Gen. only worse than Con. only for captioning. How is contrastive learner able to do captioning?\n\nW7. In the frozen transfer setting in Tab. 6 are all models fine-tuned on the same data? If so, what data? The specifics of the experiment are not clear to me, making it hard to interpret the results.\n\nQUESTIONS:\nPlease see the weaknesses section for specific questions and topics to address.\n\nLIMITATIONS:\nThe authors address limitations in a dedicated paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to explore the use of weak supervision signals in multimodal interleaved image-text data to pretrain visual encoder, compressing the distribution of high-level features into the visual encoder. The paper employs contrastive loss and autoregressive loss to train the model. To prevent the collapse of visual representations, an entropy maximization constraint is applied. The paper derives the equivalence of maximizing the mutual information between the model's input and output as a latent compression and entropy constraint. The proposed pre-training method, called LCL, achieves performance comparable to CLIP on paired data while better utilizing the supervision information in interleaved data.\n\nSTRENGTHS:\nThis paper explores how to use weak supervision signals in more general interleaved image-text data to accomplish visual pre-training. Its advantages are as follows:\n1. Unlike previous approaches that fine-tune pre-trained visual models to align visual representations with the text space (Flamingo, LLaVA), this paper explores how to train visual models from scratch using interleaved image-text data. This is a meaningful exploration.\n2. To prevent the collapse of visual representations, where autoregressive generation relies solely on textual information, this paper imposes an entropy constraint and further derives it as optimizing mutual information. This approach aids in model training.\n3. Extensive quantitative experiments have validated the effectiveness of the visual models trained using this approach.\n\nWEAKNESSES:\nThis paper has the following areas for improvement:\n1. In some cases, the textual context may have little relevance to the image. It is worth investigating whether such data could harm the model's performance.\n2. The paper lacks qualitative experiments to further demonstrate the effectiveness of the method. Designing reasonable visualization analyses would help to further elucidate the advantages of the approach.\n3. Similar to CLIP, further demonstrating the model's transfer learning performance through domain adaptation tests and few-shot metrics would be beneficial.\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nThe paper's discussion on data bias and energy consumption limitations is commendable; however, it could further explore issues related to data privacy.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}