Vision Learners Meet Web Image-Text Pairs
Authors: Bingchen Zhao
Quan Cui
Hao Wu
Osamu Yoshie
Cheng Yang
Oisin Mac Aodha
Venue: Accepted by TMLR ()

Abstract: Most recent self-supervised learning methods are pre-trained on the well-curated ImageNet-1K dataset. 
In this work, given the excellent scalability of web data, we consider self-supervised pre-training on noisy web sourced image-text paired data.
First, we conduct a benchmark study of representative self-supervised pre-training methods on large-scale web data in a like-for-like setting. 
We compare a range of methods, including single-modal ones that use masked training objectives and multi-modal ones that use image-text constrastive training. 
We observe that existing multi-modal methods do not outperform their single-modal counterparts on vision transfer learning tasks. 
We derive an information-theoretical view to explain these benchmark results, which provides insight into how to design a novel vision learner. 
Inspired by this insight, we present a new visual representation pre-training method, MUlti-modal Generator~(MUG), that learns from scalable web sourced image-text data.
MUG achieves state-of-the-art transfer performance on a variety of tasks and demonstrates promising scaling properties.  
Pre-trained models and code will be made public upon acceptance.

Published in Transactions on Machine Learning Research [August/2224]
Vision Learners Meet Web Image-Text Pairs
Bingchen Zhao', Quan Cui', Hao Wu', Osamu Yoshie', Cheng Yang', Oisin Mac Aodha'
University of Edinburgh Bytedance
Waseda University
Reviewed on OpenReview: https: openreviee net/ forum? id- OCnTRfaTTb
Abstract
Many self-supervised learning methods are pre-trained on the well-curated ImageNet-Ik 1kk dataset. In this work, given the excellent scalability of web data, we consider self-supervised pre-training on noisy web sourced image-text paired data. First, we conduct a benchmark study of representative self-supervised pre-training methods on large-scale web data in a like-for-like setting. We compare range of methods, including single-modal ones that use masked training objectives and multi-modal ones that use image-text constrastive train- ing. We observe that existing multi-modal methods do not outperform their single-modal counterparts on vision transfer learning tasks. We derive an information-thhorrticcl view to explain these benchmark results, which provides insight into how to design a novel vi- sion learner. Inspired by this insight, we present new visual representation traitingn method, MUlti-modal Generator (MUG), that learns from scalable web sourced image-text data. MUG achieves state-of-the-art transfer performance on a variely of tasks and demon- strates promising scaling properties.
1 Introduction
Self-supervised representation learning (SSL) has attracted a considerable amount of attention recently as it offers the potential to reduce the reliance on abboriiteessive human collected annotations when training models. The design of SSL methods is mainly guided by the InfoMax principle (Hjelm et al., 2019), which shows that maximizing the mutual information T(X; Z) of the input data X and the learned representation Z can lead to better representations. In the case of visual data, discriminative contrastive learning (He et al., 2020; Chen et al., 2020d; 2021; 2020b;c; Shen et al., 2022; Bardes et al., 2022) and generative masked image modeling (Bao et al., 2022; He et al., 2022; Baevski et al., 2022; Wei et al., 2022; Peng et al., 2022) have been demonstrated to Iearn transferable representations from images by attempting to solve pre-defined pretexl objectives that aim to indirectly optimize T(X; Z); achieving stale-of-the-art results on popular computer vision benchmarks, Despite these remarkable advancements, most SSL methods are developed the ImageNet-1k (Deng et al., 2009) datasct, which is well-curated and thus not representative of many real world use cases. As a result, these works do not fully exploit the potential scalability of SSL to larger uncurated datasets, e.g., many existing methods are only evaluated on the ImageNet-1k 1k dataset (Xie et al., 2021a; He et al., 2020; 2022). There exists works on extending SSL to larger uncurated datasets (Wen et al., 2022; Xie et al., 2021a; Hénaff et al., 2021; Xie et al., 2021b; Wang et al., 2021), yet most of these works focus on how to obtain object centric representations (Wen et al., 2022), or target one specific rownstream task, such as object detection (Wang et al., 2021; Xic al., 2021b;a; Hénaff al., 2021).
In contrast to the previous single-modal SSL methods, pioneering multi-modal SSL methods (e.g.. CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021)) have shown impressive scalability due to their ability to train on casily collected large-scale web sourced image-text pairs (Sharma et al., 2018; Thomce et al., 2016; Schuhmann et al., 2021). Multiple works have shown that the lextual information present in web image-text pairs can greatly benefit various downstream tasks (Mu et al., 2022; Kim et al., 2021; Yao et al., 2022; Li et al., 2022b) and thus contribute to improved transfer lcarning performance. However, many of these existing methods differ not only in the training losses they use, but also in the terms of the model architectures and training data, which makes direct comparison difficult. To address this, in this work we

---

Published in Transactions on Machine Learning Research (August/2024)
diseruiinating
discriminating
image aneoder
imagee enender
image enooder
imaage decodar
image encoder
text anorder
image onealer
imaage decodar
text decoder
image
image
genenning
image
text
image
generating
text
(i) single-modal discriminative
(ii) single-modal generative
multi-modal discriminative (ii)
(iv) multi-modal generative
Figure 1: Comparison of different vision pre-training paradigms that use images or image-text pairs. Four paradigms are considered: (i) single-modal discriminative (e.g., SimCLR (Chen et al., 2020b)), (ii) single- modal generative (e.g., MAF (He et al., 2022)), (iii) multi-modal discriminative (e.g., CL.IP (Radford et al., 2021)), , and (iv) our proposed multi-modal generative approach named MUG. The multi-modal generative paradigm simultancously generates images and text using only image representations.
conduct a benchmark study of single-modal SSL methods (i... with only images) and multi-modal SST. methods (f.e., with image-text pairs) using the same web-sourced image-text datasets and training settings.
Based our benchmark study, we make two key observations. First, generative methods achieve the best transfer performance. Second, and more surprisingly, the representations learned from multi-modal SSL methods do not outperform single-modal SSL ones. To explain these observations, we explore an information-theoretiee. motivated perspective. Starting from the InfoMax principle (Hjelm et al., 2019) for learning representations, we use the informatioonbootleeeek theory to explain the transferability of deep representations (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Alemi et al., 2017; Cui al., 2022a). We show that generative methods approach better upper bound on the mutual information I(X; Z), and that introducing multiple generative objectives can further improve upon this bound.
In Fig. 1, we group previous representative methods into three familics: (i) single-modal discriminative (e.g., SimCLR (Chen et al., 2020b)), (ii) single-modal generative (e.g., MAE (He et al., 2022)), and (iii) multi- modal discriminative (e.g., CLIP (Radford et al., 2021)). Empirical observations and theoretical analyses jointly inspire us to design a fourth paradigm, multi-modal generative pre-training (see Fig. (iv)). Under this paradigm, we propose novel vision learner for web image-text data pre- pre-training, named MUlti-modal Generator (MUG). MUG is composed of purely multi-modal generative objectives that attempt to generate the multi-modal joint distribution from single-modal image input. From information thtiroticteoeeacall view, we show that MUG has a better upper bound for feature transfcrability, which helps the vision encoder learn more transferable representations when compared to alternative paradigms.
We present empirical results across a variety of downstream vision tasks to further validate the advantages of MUG. For example, MUG outperforms the previous best performing methods by 2.0% mIoU when transferred to the ADE20k benchmark and by 0.5% on ImageNet- Ik classification. Extensive ablations are conducted to help better illuminate the impact of critical model and design choices. Finally, visualizations are provided that illustrate that our learned representations are capable of capturing the joint image-text distribution.
2 Related Work
There is a large body of existing work on the topic self-supervised learning of representations from visual data. We focus on constrastive and masked-based learning objectives, and also discuss learning from paired image and text data.
2.1 Vision Learners with Contrastive Learning
Contrastive learning is commonly used paradigm for single-modal self-supervised representation learning. Here, the pretext lask is an instance discrimination one (Wu et al., 2018) which learns representation by pulling positive examples that are generated from augmentations negative examples (e.g., different images from the same mini-batch) apart in the embedding space. Building on the basic instance discrimination task, SimCLR (Chen al., 2020b;c) showed that larger mini- batches and songer augmentations can result in more effective representations. MoCo (He et al., 2020; Chen et al.,
the same image closer, while pushing
2

---

Published in Transactions on Machine Learning Research [August/2024]
2020d; 2021) showed that introducing more negative examples using momentum encoder and memory bank results in representations that are even more effective for downstream transfer. A variety of techniques for improving contrastive learning have been proposed, including hard example mining (Kalantidis et al., 2020; Zhao & Wen, 2020), image mixing (Shen et al., 2022; Zhu et al., 2021), and by injecting localized priors (Hénaff et al., 2021; Wen el al., 2022; Xie et al., 2021b; Wang et al., 2021).
The underlying principle of contrastive learning is the Info principle (Hjell et al., 2019), which posits that the mutual information I(X; Z) between the imput image X and the encoded representation Z should be maximized so that the representation of an image can uniquely identify it from the entire dataset. CTC (Cui et al., 2022a) showed that the mutual information I(X; Z) is the key to the transfcrability of one representation, and that preventing the decrease of [(;; 2) during training can improve the transferability of the learned representation. CTC (Cui et al., 2022a) also showed that contrastive learning objectives are in practice learning to optimize T(X; Z) = H()) - H ( by increasing (Z).
[(X:
( -
(1)
Representation Learning
Contrastive Learning
Generative Learning
2.2 Vision Learners with Masked Image Modeling
Inspired by the success of masked language modeling (MLM) using transformers (Vaswani et al., 2017) for natural language processing (Devlin et al., 2018; Brown et al., 2020), masked image modeling (Chen et al., 2020a; Bao et al., 2022; He et al., 2022; Xie et al., 2022) has been explored in the context of learn- ing representations using vision transformers (Dosovitskiy et al., 2021; Touvron et al., 2020). Specifically, iGPT (Chen et al., 2020a) learns a representation by predicting unknown pixels from a sequence of pixels. Instead of operating in pixel space, BETT (Bao et al., 2022) predicts a set of discrete tokens from the encoder of DALLE (Ramesh et al., 2022; Van Den Oord et al., 2017), where the tokens are encoded versions of image patches as in (Ramesh et al,, 2021). The training task was further simplified by MAE (He et al., 2022) and SimMIM (Xie et al., 2022), which only enforee the model to reconstruct the masked pixels of the input image using a lz loss and do not require the use of discrete token encoders.
The pretext task of masked image modeling can be viewed as optimizing - Iox where Z is the representation of the non-masked image X and p(8 (2) is the likelihood of generating the masked image parts X given the learned representation of the input tokens. This is equivalent minimizing (X) Given that ((;Z) = H(X)- (X[)),, we can see that masked image modeling still follows the InfoMax (Hjelm et al., 2019) principle, where the maximization of I(X; 2) is performed by optimizing H Z), instead of (Z) as in contrastive learning. MTM has been shown to be more effective at learning transferable representations compared to contrastive learning (He et al., 2022; Xie et al., 2022), indicating the effectiveness of generative pro-training.
2.3 Vision Learners with Web Image- Text Pairs
The vast majority of the previously discussed works have explored SSL in the context of single-modal well-curated image datasets such as ImageNet-lk (Deng et al., 2009). However, it is relatively easy to collect large-scale datasets containing image-text paired data from the web, e.g., images and associated text captions (Sharma et al., 2018; Schuhmann et al., 2021). Many approaches have recently began to study what impact this type of data has on learning visual representations, e.g., Radford et al. (2021); Mu et al. (2022); Kim et al. (2021); Yao et al. (2022); Li et al. (2022b); Yu et al. (2022); Desai & Johnson (2021). CLIP (Radford et al., 2021) explores this challenge by performing contrastive learning over paired image and text data. By leveraging separate visual and text encoders, their training objective aims to maximize the similarity between the learned embeddings of the paired image-text data from each training instance and minimize the similarity between unpaired images and text. SLIP (Mu et al., 2022) builds on this, by learning the representation by jointly performing contrastive learning on both imago-text paired datascts like CC3M (Sharma et al., 2018) and image-only datasets like ImageNet (Deng et al., 2009) using strong augmentations (Chen et al., 2020b). SLII showed that the added self-supervised signal image-only
3

---

Published in Transactions on Machine Learning Research (August/2024)
datasets can improve the performance for zero-shot image classification and transfer learning. However, as we will show later in our like-for-like experiments, the benefit of SLIP over the baseline CLIP model is limited, we conjecture this is due to the fact that the two objectives of SLIP are both contrastive learning losses which are discriminative. In Table 1, we show that discriminative objectives are not as effective as generalive ones.
CoCa (Yu et al., 2022) poses representation learning as a cross-modal contrastive learning and auto-regressive caption generation task. The contrastive loss of CoCa contrast the image embedding with its corresponding textual caption embedding. The caption generation loss is designed to first encode the text only features using a text encoder with causal attention, and then transformer decoder with cross-attention to image encoded features are used to decode the caption auto-regressively. These two objectives both enforce the model to learn a predictive image representation of the caption. CapPa (Tschannen et al., 2023) further explores visual representation learning using only an auto-regressive image captioning loss. High performance of zero-shot learning and few-shot linear classification is achieve in CapPa. In contrast, in our designed model in Section 4, we enforce the image representation to not only be predietive of the caption; but also to be predictive of the image itself. Additionally, this paper also provides an intuitive understanding of why and how such generative training objective results in strong representation learning ability from an information theoretical perspective.
There also exists other work that focuses on achieving higher zero-shot performance from a frozen visual representation (Zhai et al., 2022) by filtering the caption to achieve better performance (Li et al., 2022a), or adapting frozen visual and language models together to achieve in-context learning abilities on complex visual-language tasks (Alayrac et al., 2022; Li et al., 2023; Koh et al., 2023; Zong et al., 2024). In this work, we first comprehensively study existing paradigms for learning transferable representations from publicly available image-text paired datasets. Inspired by this, we propose a new method for learning representations and further compare our method to the previous state-of-the-ar works where we out perform them across a wide varicty of tasks.
3 Preliminaries
3.1
Benchmarking SSL Methods on Web Data
Given dataset of image-text pairs D = {(=)' xx))) sampled from the joint distribution of p(XV, x2),, where image is of size x.= E RC'HHW and piece of text is of size of R4, our goal is to learn visual feature encoder ʃ Hx R.P that maps input image into a compact feature embedding and enables effective transfer learning performance (He et al., 2019).
Two familics of self-supervised learning methods can be used to achieve this goal. This first is single-modal SSL, which uses only the images æY within the dataset D. Section 2.1 and Section 2.2 introduce and describe the current most representative methods for single-modal SSL. The second approach is multi-modal SSL, This aims to learn effective representation by aligning the representation of the input image with the corresponding paired input text. A brief introduction to molti SSL methods is provided in Section 2.3. As a preliminary motivation for our work, we first perform a benchmark analysis. We pre-train representative SSL methods on the same web image-text dataset, for the same number of training epochs. Then, we evaluate the transfer performance on downstream tasks by fine-tuning the resulting pre-trained models.
Experimental setup. We categorize methods according to their pre-training paradigm, as discussod in Fig, 1. For single-modal SSL methods, we choose MoCoV3 (Chen et al., 2021) and SimCLR (Chen et al., 2020b) as representative discriminative methods, and MAE (He et al., 2022) as representative gen- erative method. For image-text multi-modal SSL, we choose CLIP (Radford et al., 2021) and SLIP (Mu et al., 2022) as discriminative methods, and also include CoCa (Yu et al., 2022), which is composed of both discriminative and generative largets. Notably, in addition to these methods, we further implement another multi-modal baseline namely Masked Captioner (MAC), which generates text from masked images. It is trained using only a purely generative cross-modal objective. The details of MAC will be introduced in Soc- tion 4. For the pre- pre-training byper-parameterss we strictly follow the recommended setting in the original paper of each method. For fair comparison, we pre-train all methods web dataset CC3M (Sharma
4

---

Published in Transactions on Machine Learning Research (August/202))
narrow bottleneck
wide buttleneck
a wider buttleneck
X or x", xt
encoder
X
encoder
T
x
encoder
T
(ii) multi-modal generative Figure 2: Left: Single/ mullimmmmlliiaaall discriminative met have narrow bottleneck and thus learn less informative representation. Middle: Single-modal generative methods have a wide bottleneck and thus learn a more informative representation. Right: Multi-modal generative methods have a wider bottlenock for generating (e.g., recovering) the joint distribution of both modalities, and as a result learn an even more informative representation.
(i) singlo/multi-modaa discriminative
(ii) single-modal generative
Table 1: Like-for-like comparison cxisting SSL pro-training methods. All models are pre-trained on the CC3M dataset (Sharma et al., 2018), and evaluated on ImageNet-lk (IN-IK) (Deng al., 2009) via end- to-end fine-tuning, "gen. and dist." are short hand for generative and discriminative. "I and T denote images and text. For instance, SimCLR (Chen al., 2020b) is pre-trained by discriminating images (disc. I). CLIP (Radford et al., 2021) is pre-trained by discriminating image-text pairs (disc. I&T). CoCa (Yu et al., 2022) has two objectives: discriminaling image-text pairs (disc. I&T) and generating text from images (gen. T w. Z). We also include a MAsked Captioner (MAC) baseline that generates text from masked image patches (gen. T w. I).
method
pre-training paradigm
IN-1K (Deng et al., 2009)
single-modal pre-training
MAE (He al., 2022)
gen. I disc. I
83.0
SimCLR (Chen et al., 2020b)
82.7
MoCoV3 (Chen al., 2021)
disc. I
82.6
multi-modal pre-training
MAC
gen. T w. I disc. I&T
81.7
CLIP (Radford al., 2021)
79.7
SLIP (Mu et al., 2022)
disc. I&T disc. I disc. I&T + gen. w. I
80.9
CoCa. (Yu et al., 2022)
79.5
al., 2018) for 400 epochs. When transferring the representation to ImageNet-Ik (Deng et al., 2009), we follow the widely used fine-tuning recipe introduced by Bao et al. (2022); He et al. (2022).
3.2 Key Observations
We make the following key observations from the results of the experiments in Table 1:
Generative methods (e.g., MAE) achieve the best results. Across all methods, we can observe that MAE (He et al., 2022) achieves the best performance. As mentioned in Section 2.1 and Section 2.2, we argue that discriminative pre-training and generative pre-training approach the InfoMax (Hjelm et al., 2019) principle via different mechanisms. Specifically, discriminative pre-training optimizes [(X; Za) H(Z) H(ZalX) via maximizing HZa),, while generative pre-training optimizes I(X; Zg) = H HX) via minimizing H (,) (Cui et al., 2022a), here Za is the representation learned by discriminative pre-training, and Zg is the representation learned by generative pre-training. Suppose the models Icarned by generative pre-training and discriminative pre-training both achieve zero training loss, then л(X; Za) will be (Za)) and 1(x; Z,) will be H(X). In the discriminative pre- traininii scenario, we consider the Markov Chain of + Za, we have T(X; X) 2 T(X; Za) from the data process incquality, í.e., H(X) 2 T(X; Za) = H(Za). Thus generative pre-training has better transferability upper bound compared discriminative pre-training, and results in more informative and beneficial features for downstream tasks.
5

---

Published in Transactions on Machine Learning Research (August/2024)
Disc.: Z(X; ((aa)
X
upperhound
H((a))
(2)
maximixing
Gen.: Zg) (X) 2)
zero training loss, +X(X
(3)
minimizing
Current multi-modal methods do not outperform single-modal ones. We can observe that the best single-modal pro-training method outperforms the best multi-modal pre-training method by 1.3% the ImageNet-Ik accuracy. Compared with our introduced MAC baseline, multi-modal discriminative methods (e.g., CLIP and CoCa) yield worse results. We believe the reason is that the information from the tert modality is greatly limited A typical text annotation only contains partial information about the contents of the corresponding image. prominent researcher has commented that "language is a very low- now-bandwidth method for transmitting information: isolated words or sentences, shorn of context, convey little." " (Browning & LeCun, 2022) It suggests the H (X²) could be much lower than H (XV). In the case of the multi-moda discriminative method CLIP, low (X²) directly limits I(XL; Za)- Thus, poor transferring performance is observed. When comparing SimCLR and SLIP we observe that SLIP falls behind SimCLR by 1.8% accuracy, which further supports the potential limitations of multi-modal discriminative training objectives. However, MAC is generative and alleviates this problem, but it still cannot outperform single-modal methods.
H(x²) < H(X))
(4)
Discussions and opportunities. Inspired by the information- 2015; Shwartz-Ziv & Tishby, 2017; Alemi et al., 2017; Cui et al., 2022a; Hjelm et al., 2019), we illustrate to help understand our observations in Fig. 2. The first observation that generative methods can help the model learn more transferable visual representations compared to discriminative ones due a superior 1(x; Z) upper bound. In Fig. 2 (left), we argue that discriminative methods have narrow bottleneck and low I(X; Z), thus less informative representation learned. More detailed discussions are provided in Appendix Appendix A.6.
theory (Tishby & Zaslavsky,
In Fig. 2 (middle), generative methods have a wider bottleneck, leading to more informative representa- tions. Given the side eflects of multi-modal discriminative objectives (the second observation), developing multiple generative objectives will encourage the model to learn more informative representations. As shown in Fig. 2 (right), for an imago-text pair, gencrating the joint distribution of both the image and text modal- ity will help the model to "absorb" as much information possible from the data and approach a larger 7).
Note that here we are using the term bottleneck to refer to the variance in the representation space Z with respect to the input images X, é.e., how much the representation changes when the input is changed. One way to understand this bottleneck is from the perspective of the rank of the feature matrix (Garrido et al., 2022). A higher rank would indicates that the model can capture more variance in the input, whereas a collapsed lower rank captures less information about the input. In contrast, a completely collapsed representation would map all inputs to a same feature which has rank of zero. In Appendix A.4, we provide qualitative mcasure of the rank of different pro-trained models.
4 Approach
4.1
Motivation
Following the above observations, in this section we describe a more effective vision learner that learns from web sourced image-text paired training data. We first formulate the mechanism that multi-modal generative objective uses to learn more transferable representation, in line with Fig. (right). We believe that introducing multiple generative objectives is feasible, where now the optimization objective is to learn

---

Published in Transactions on Machine Learning Research (Auuust/222)
Algorithm 1 Pseudocode for MUG.
key, value
image encoder
image decoder
text decoder
query
ing, txt: image-taxt paired data # txt_nask: causal naak for captioning tezt_decoder(e, kv, mask):
- single, mad_oddd__aat attnnq, mask) cap_res multinodal, _attn(q, kv) cap_res
2
MAMMNN
patch ing patchify(ing)
maaked_ token nasking(patch, ing) latent vit_encoder vis_encoder(mmseed_.co (nacked [i L,
Generative objective recon ing mae jas__decoderrateee (latent) recon loss - mse nse_loss (ing, recon_ing)
inage
[X, L, D]
We generated caption
generated image
original
image
caption
Generative objactive
text.
label, txt txtl:, 1:, :l, txtl:, 1-1, :]
ext_ tokenizer (txxt) a [X, L, D]
MSE loss
CE loss
cap_res text (etooer(grtxxt_fee ext_nask) cap_loss (l_bees cap_res)
kv-latent, mask-
genensing glals
gasersting words
sakked Lkeeo
genenned decedor
leaa recon recon_veight weight recon_loss + cap_reight cap_lose
Figure 3:
Illustaation of our multi-modal self- supervised approach MUG.
how to generate the joint distribution p(XV, x from a single masked input distribution X as our focus is on learning a visual encoder.
To generate the joint distribution p(XV, x4h from the latent representation == fXXV extracted from the non-masked visual input &V, the model maximizes the following mutual information:
T(XV, x =X(XV .x HIXV 1z)
2 max
- ( x
(5)
max HH HH(X)) (zero training loss)
(6)
If we consider the fully optimized model the optimal, then H(XV x41Z) = As the lower bound of [(XV xx: Z) greater than or equal to the lower bound of 1(X; ;Z) because of the added variable X we can observe that generating the joint distribution of p(XV, X²4 can least learn an equally transferable representation single-modal generative pre-training. In practice, observe that multi-modal generative pre-training achieves superior transferability. This design motivation is also in line with the discussion in Section 3.2. Overall this intuition motivates us to design generative pre-training framework with multiple generation tasks using only the visual input and representation.
4.2 MUG: MUlti-modal Generator
Here describe novel multi-modal generative vision learner, named MUlti-modas Generator (MUG). The framework of MUG is presented in Fig. 3. It consists of an image encoder, an image decoder, and text decoder. Two tasks are involved, f.e., raw pixel and original caption generation.
Raw pixel generation. Given a masked visual input &Y = M @ @xY. where M is the random generated (-1 mask following the procedure in MAE (He et al., 2022) where 0 indicates that the image region is masked out and 1 indicates that is visible to the model, @ denotes the element-wise product, the visual feature encoder maps the masked input image to latent. representation Ex f(a)') The latent representation Ey is then fed into visual decoder gV to generate æ. gv (t.") which is the reconstruction of the non-masked visual input. We adopt the reconstruction loss from MAE (He et al., 2022) to train both f and gv:
Lv
((a)) ([ - M) - M) 2 O
where ((a)) ) is the number of masked elements in aV This loss encourages the generation of the visual part of the input.

---

Published in Transactions on Machine Learning Research (August/202))
Caption generation. We further propose to generate the paired text (t.e. the image caption) a from the feature Ef of the masked visual input aÝ. To do this, we feed the feature 1y to a text decoder gr. to generate the text captions = = g1 (2h ) in an auto-regressive fashion, driven by the following loss:
Cr. ===-
Σ log P( :
where J is the maximum length of a textual caption, oi,e is the first j - tokens of the caption and ats is the j-th token of the caption. The text generation is conditioned on the image features, which helps to improve the transferability of the image encoder. Thus, the proposed text decoder has two parts, i.e., single-modal and multi-modal layers. The single-modal part is constructed as an auto-regressive model via a causal mask (Vaswani et al., 2017), where each word can only attend to the words before itself in a caption sequence. It only provides the text decoder with contextual information when gencrating the caption. We design the multi-modal part using the image features as the key and value and the output from single-modal layers as the query in cross-attention layer (Vaswani et al., 2017). This ensures the text generation is derived from the image features as the contextual information only serves the query. Thus, the entire text decoder can be seen regrouping the image features LY with contextual information to generate captions.
Optimization. Our multi-modal pre-training framework is trained by optimizing the above two losses:
L= Ly + ArCi,
(9)
where bv and dr are different weights for the image and text generation losses. Additionally, for the MAC method we evaluate in Section 3, only the caption loss L1 is used to optimize the model.
4.3 Understanding the Loss Function
If we consider the MUG image decoder as modeling the image using Gaussian prior (Zhao et al., 2015), then the loss in Fq. (7) is actually maximizing the log-likelihood of the generated image:
(-1 a - x )) (-)((==)))
(-2 (-2(0²
1 -T
log Pay Pay(2 = log
cxp
M |aY - x 12
(10)
2
where a is the standard deviation of the distribution. M and C are constants that do not affect the optimiza- tion process. Together with LL, we can see that our loss function in Eq. (9) is maximizing the likeliihl of both the generated image and the generated paired text. Thus the latent representation Z extracted by the model is optimized to be more predictive of the image X and the caption X In other words, the conditional entropy H(XV, x is minimized via the two losses in our framework, maximizing the mutual information 1(XV, x 2), and leading to a more transferable representation.
5 Experiments
5.1
Implementation Details
Encoders and decoders. Our framework contains an image encoder f, and two decoders gv and gr for decoding the latent representation to reconstruct the image and the textual caption, respectively. The image encoder f is implemented using a Vision Transformer (Dosovitskiy et al., 2021) backbone, and the image decoder is implemented with the same architecture as in MAF (He et al., 2022). We implement the text decoder 9L following SimVLM (Wang et al., 2022). For efficient training, we train our encoder- decoder architecture using teacher-foreing (Williams & Zipser, 1989). Detailed implementation details, including architectures and hyper parameters, are provided in the Appendix. The code is available at https: / //huggingface.. co. co/ sttssspaccssetaaeee htte//tnnttttnuaaatteett
Pre-training datasets. We train models on the publicly available CC3M (Sharma et al., 2018) and LAION400M (Schuhmann et al., 2021) datasets. Additionally, we also collect 200M web image-text pairs to study the scalability of MUG. The collected data has no filtering processing and as a result potentially contains a large amount of noise which presents a challenge for modti-modal methods. We denote this privately collected dataset as W200M.
8

---

Published in Transactions on Machine Learning Research (August/2024)
Table 2: Comparison of different methods pre-trained on CC3M (Sharma et al., 2018) and then evaluated via fino-tuning on ImageNet-lk (IN-1K) (Deng et al., 2009). Top-1 accuracy results are reported. In addition, we evaluate the IN-lk fine-tuned models on SuageNee-Adversariaa (IN-A) (Hendrycks al., 2021b) and GaageNet-Reenditinn (IN-R) (Hendrycks et al., 2021a) to assess out-of-distribution performance.
method
data
#epoch IN-1K IN-A IN-R
single-modal pre-training
SimCLR
CC3M
400
82.7
31.2
49.1
MoCoV3
CC3M
400
82.6
29.7
46.9
MAE
CC3M
400
83.0
32.9
49.4
malli-modal pre- pre-inngning
CLIP
CC3M
400
79.7
23.6
42.6
SLIP
CC3M
400
80.5
20.5
44.4
CoCa
CC3M
400
79.5
18.1
42.9
MAC (impl.)
CC3M
400
81.7
27.5
44.3
MUG (ours)
CC3M
400
83.5
36.3
50.4
5.2 Transfer Learning
We begin by evaluating the transfer performance of different learned reprosentations on a wide range of downstream lasks. Transfer learning is performed with conventional full end-to-end fine-tuning of the model weights on target task (He et al., 2022). This highlights the superiority of MUG compared to other methods.
Image classification on ImageNet- First, we transfer Icarned representations to general image clas- sification on ImageNet-1kk (Deng et al., 2009) using end-to-end fine-tuning. In addition to reporting perfor- mance the validation images in ImageNet- 1k, also evaluate ImageNet- 1kk fine-tuned models out-of- distribution datasets SoageNet--Advesaarii (IN-A) (Hendrycks et al., 2021b) and Srngeeeet-Rendttiin (IN- R) (Hendrycks et al., 2021a). Results are reported in Table 2 and Table 3. Compared to previous methods, MUG achieves the best top-1 accuracy 83.5% (with CC3M). When training MUG with ViT-S, ViT-B, and VIT-L backbones, we also observe consistent improvements over the previous single-modal best performing MAE (He et al., 2022). Both comparisons illustrates that MUG surpasses previous methods by making use of scalable web image-text paired data. Notably for the results on IN-A and IN-R, MUG outperforms the previous multi-modal methods by large margin (i.., >10% IN-A) even through our model leverages the same supervision signal these other multi-modal methods. This indicates that MUG is much more effective for learning representations that generalize to hard ciat----ditribution cases.
Next, we evaluate pre-trained models using lincar probing evaluation on ImageNet-Ik. Results are reported in Ta- ble and Table 6. Compared with discriminative pre- training methods, most generative methods obtain unsat- isfactory IN-1k linear probing results. MUG also suffers from unsatisfactory linear probing performance common to other masked transformer models (He et al., 2022), where it has been hypothesized that linear probing re- quires discriminative representations, which cannot be satisfied by generative methods. For instance, SimCLR and MUG achieve 63.3% and 61.3% on the IN-1k val- idation set, respectively. However, in the case of IN- A and IN-R, generative methods like MUG achieve the best results. We conjecture this is due to the fact that discriminative mot can easily be overfit to the in- domain data, but conversely cannot achieve good out-of- distribution generalization. Additionally, compared with the original MAE, MUG outperforms by a large margin. Notably, when using the backbone ViT-S in Ta- ble 6, method perforrsms MAE by 1 11.3% top-1 accuracy. This indicates that MAE pre-trained models
Table 3: Comparison of different backbone sizes (i.e., ViT-S, ViT-B, and ViT-L) for models pre- trained with CC3M for 400 epochs and then fine-tune on IN-IK. Top-1 accuracy is reported. Across all backbone sizes, we observe that MUG is superior to MAE in almost all cases.
method
backbone
IN-IK
IN-A
IN-R
MAE
ViT-S
80.9
22.7
44.1
MUG (ours)
VIT-S
81.6
25.0
42.5
MAE
VIT-B
83.0
32.9
49.4
MUG (ours)
VIT-B
83.5
36.3
50.4
MAE
VIT-L
85.0
49.4
59.6
MUG (ours)
VIT-L
85.4
53.8
63.1
9

---

Published in Transactions on Machine Learning Research (August/2024)
can suffer from poor results due lower model capacity, whereas MUG pro-trained models can alleviate this drawback with the help of external data information from the text modality.
Table 4: Comparison of pre-training methods on ImageNet-1k -kk (IN-1K) using linear probing. Top- accuracy is reported.
Table 5: Comparison of pre-training methods when fine-tuning on fine-grained iNat-17, iNal- 18, and P365, Top-1 accuracy reported. MAE= is pre- trained on IN-1K.
method
data #epoch IN-IK IN-A IN-R
method
backbone
iNat-17
iNat-18
P365
single-modal pre-traiin training
SimCLR.
CC3M
400
63.3
12.5
23.0
MAE
VIT-S
65.4
70.0
57.9
MAE
CC3M
400
57.4
9.1
17.8
MUG (ours)
VIT-S
65.5
70.7
58.3
multi-model pre-training
MAE+
VIT-B
70.5
75.4
57.9
CLIP
CC3M
400
55.1
11.2
18.9
MAE
VIT-B
67.9
73.0
58.3
CoCa
CC3M
400
56.9
13.8
20.3
MUG (ours)
ViT-B
68.4
73.4
58.6
MAC (impl.) CC3M
400
59.3
15.2
21.3
MAE
VIT-L
73.0
76.8
59.4
MUG (ours) CC3M
400
61.3
16.7
24.2
MUG (ours)
VIT-L 75.1
77.8
59.8
Fine-grained image classification. To illustrate that our pre-trained models do not over-fit on ImageNet- 1k- based evaluations, we further transfer learned representa- tions to fine-grained image datasets. Specifically, we re- port performance on iNaturalist-17 (iNat-17) (Van Horn et al., 2018), iNaturalist-18 (iNat-18), and Places365- Small (P365) (Zhou et al., 2017a) with end-to-end fine- tuning. Results are presented in Table 5. Similar to the results of ImageNet- 1k, MUG still outperforms the previ-
Table 6: Comparison of different backbone sizes (i.e., ViT-S, ViT-B, and ViT-L). Models are pre- trained on CC3M for 400 epochs and evaluated using linear probing on IN-1K.
method
VIT-S
VIT-B
VIT-L
MAE MUG (ours)
41.0
57.4
63.5
52.3
61.3
67.6
ous SOTA method MAE all evaluated cases. This further validates the success and robustness of MUG in transferring to image classification tasks. Moreover, we notice that the performance of MUG is relevant to the pre-training data domain. Concretely, iNat-17 and iNat-18 share similar domain with IN-1K. Thus, the IN-1K MAE models perform better than CC3M MAE models. P365 shares similar domain with CC3M, which leads to better results of CC3M-based models. Regardless of the fine-tuning data domain, the improvements from MUG are stable and generalize across datasets.
Semantic segmentation on ADE20K. We transfer MUG to semantic segmentation task using the ADE20K dataset, (Zhou et al., 2017b). Experiments on ADE20K use the decode head from UperNet. (Xiao et al, 2018), and training recipes follow the default setting provided in nmsegmentation Here, the training resolution is set to 512 x 512, and the number of iterations is 160k. We evaluate models with different capacities in Table 7. We observe that our MUG approach results in significantly improved transfer performance across the VIT-S, VIT-B, and VIT-L backbones. We can see that the VIT-S MAE model only achieves 40.7% mloU, Consistent, with the previous IN-1K linear probing results, MUG significantly alleviates the performance drop on the small backbone by improving mloU by 2.6%. Notably when using the larger VTT-I backbone, MUG still improves over MAF by 2.1% mloU. Compared to CLIP and MoCoV3, the superiority of MUG supports our cettmaatioonthhooetiin motivation, i.e., generative pre-training results in better transferability upper bound compared to discriminative pre-training. Consequently, we observe that generative pre-training greatly benefits downstream fine-tuned performance.
(oaeetttttiiiiooiinnaiiiiiaaa (Contributors, 2020).
5.3 Ablation Experiments
Ablation: Impact of image reconstruction and text generation losses. The most important ablation is the trade-off between the image reconstruction loss weight Xy and the caption generation loss A₁. We fix λy and ablate the value of Å IN-1K fine-tuning experimental results are reported in Table 8.
The best transfer performance is achieved when λ₂ 0.1, and both larger and smaller values perform worse. This rates the trade-off between image reconstruction and text generation.
10

---

Published in Transactions on Machine Learning Research (August/202))
Table 7: Semantic segmentation fine-tuned results (reported as mloU) comparing different pre-training
methods on ADE20K (Zhou et al., 2017b). MoCoV3+ and MAE* are pre-trained on IN-1K.
method
backbone
data
#epoch
mloU
MAE
ViT-S
CC3M
400
MUG (ours)
VIT-S
CC3M
400
43.3
CLIP
VIT-B
CC3M
400
39.4
SLIP
VIT-B
CC3M
400
38.1
MoCoV3=
VIT-B
IN-1K
1600
47.3
MAEs
VIT-B
IN-1K
1600
48.1
MoCoV3
VIT-B
CC3M
100
46.5
MAE
VIT-B
CC3M
400
45.5
MUG (ours)
VIT-B
CC3M
400
47.5
MAE
ViT-L
CC3M
400
50.0
MUG (ours)
VIT-L
CC3M
400
52.1
Oriainal capioe: red parked baa-a
Origiaal capiees youra valina bei phone. cpption persen taklnng phone
Original aptions grour drinis posing ricture. exproo
Origind eptiios raa sipting weise glos ber. Gesavited expliee. wine glass
Odigial oap A eoseas and tho young dellcce siine heads Gesernzd optian, wassem lice.
Origial eaption: whii hoc solff picture glas. Gcocraaal eqptien driving highway.
Haxtod
Nakee
Gevente captior wit
Gescozze table the well
dihiin
Figure 4: Reconstructions of masked images and captions from our MUG approach from the MS-COCO (top) and PASCAL-VOC (bottom) datasets.
Setting '₁ 0.0 makes MUG equivalent MAF. Increasing Å₁ in the range of (0.05, 0.3] increases the transfer performance, illustrating the effectiveness of our introduced text generation task. The reasons for the performance drop are two-fold: (i) As the CC3M pre-training data is web sourced, increasing Å emphasizes the noisy text generation task. This can be problematic as captions coupled with web images can be often irrelevant, e.g., an image contains a river, but its caption may be about hiking. (ii) As discussed in Section 3.2, the information contained in the text modality is greatly limited. Increasing \z. unavoidably leads to over-fitting fitting, which is unfavorable in representation learning. Therefore, a trade-off between image reconstruction and text generation exists.
We further study the effects of the image reconstruction target. We simply set Xy to 0.0 and set A 1.0. which only permits the text generation task. Interestingly, setting λy to 0.0 degrades MUG to our MAC bascline. Results are reported in Table 8 (right), where notable performance drop is observed. This validates that (1) the information in text modality alone is insufficient, and (2) motivates the need for image generation for effective visual representation learning with image-text, pairs.
Ablation: Impact of the number of layers in text encoder. For the text generation task, both single-modal and multi-modal layers are involved. We set the total number of layers to ten, and perform a grid search for the best result, Table 9. We observe that setting the number of single-layers to one two
11

---

Published in Transactions on Machine Learning Research (August/202))
Table 8: Left: Ablation of the language generation loss weight for MUG pre-trained on CC3M and fine-
tuned on ImageNet- 1k. Right: If we disable the visual generation task, performance suffers.
X
0.0
0.05
0.1
0.2
0.3
w/o Lv w/ Lv
ft. top-1 acc. 83.0 83.3
83.5
83.2
83.0
81.7 83.5
achieves comparable performance. However, when the number of multi-layers decreases to five and six, the performance drop is significant. The reasons are likely two-fold: (i) A few single-modal layers support the text generation task since the single-modal layers can serve has a small influence on visual representation learning. (ii) A certain number of multi-modal layers could be required to support the text generation task. An inadequate number of multi-modal layers further hurts the learning image reconstruction because the performances becomes worse than the MAE baseline (83.0%).
key entity word extractor, which only
Table 9: Ablation of the number of textual layers for MUG when pre-trained CC3M and fine-tuned ImageNet-1k.
#single / laule- F.T. top-1 acc.
1/9
2/8
3/7
4/6
5/5
83.3
83.5
83.0
82.5
82.1
Ablation: Impact of form of text generation. Here we compare different text generation methods. There are two methods for text generation, i.e., masked language modeling (MLM) and auto-regressive mod- eling (captioning). We set. the optimal Å5. values for both methods, and observe that captioning outperforms MLM by 0.5% top-1 accuracy (83.5% v.s. 83.0%) on IN-1k. A harder task could help to approach higher T(XV, X); Captioning regularized to scquentially predict words, while MLM is not regularized. The text generator is composed of single-modal and cross-modal parts, and the text generation methods actu- ally affects the learning mechanism of the single-modal part. Concretely, the ruto-regressive learning target forces the single-modal encoder to extract key words, e.g., the word after an article could be a key entity noun. For masked language modeling, contextual words are possibly masked and predicted. However, such noisy and potentially meaningless reconstruction tasks likely do not benefit the vision encoder as much.
Reconstructed images with coupled text. In Fig. 4, we provide generated images and captions produced by MUG on the MS-COCO (Lin et al., 2014) and PASCAL-VOC datasets (Everingham et al., 2015). The input to MUG only consists of a masked image and start word (e.g., *a"). As observed, MUG can recover the main objects and entities in the raw images and annotated captions, even if most of the image pixels are masked. It is reasonable that MUG only describes visible pixels, e.g,, in the middle of the second row, the left child is masked, and thus MUG only describes "a woman and a man". These visualizations show that the representation learned by MUG can generate the joint distribution p(XV, X
Visualization:
6 Conclusion
In this work, we first benchmarked different self-uueerviee representation learning methods and outlined two main observations: (i) generative pre-training achieves the best transfer performance web sourced datasets, and (ii) current multi-modal methods do not outperform single-modal ones. An informattion-theretiie. view was developed to understand these benchmarking results and our observations. This inspired us to propose a new vision learner, titled MUlti-modal Generator (MUG), that makes use of web sourced image-text paired data. MUG learns visual latent representations using two generative objectives and demonstrates strong generalizability across multiple transfer learning lasks with satisfactory scaling performance, and validates our infrrnation-thherretiall perspective.
References
Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards. alpha-req: Assessing repre- sentation quality in self-supervised learning by measuring eigenspectrum decay. NeurIPS, 2022. 18
Jcan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, Tain Barr, Yana Hasson, Karel Lone, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In NewrIPS, 2022. 4
12

---

Published in Transactions on Machine Learning Research (August/202))
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottle- neck. In ICLR, 2017. 2, 6
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022.
Hangbo Bao, Li Dong, and Furu Wei. BETT: BERT pre-training of image transformers. In ICLR, 2022. 1, 3, 5, 17, 20
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianee-invaaiacccccooaannnn regularization for self-supervised learning. In ICLR, 2022.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In ICML, 2018. 18
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020. 3
Jacob Browning and Yann LeCun. Ai and the limits of language. Noema Magazine, 2022. 6
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Gener- alive pretraining from pixels. In ICML, 2020a. 3
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020b. 1, 2, 3, 4, 5, 20
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In NewrIPS, 2020c. 1,
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv:2003.0 04297, 2020d. 1,
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers.
In CVPR, 2021. 1, 3, 4,
M.SSegmentation Contributors, MMSSegmentttion: Openmmlab semantic segmentation toolbox and bench- mark. https://github. com/ open
http:///..ooooooooaaaaaa//////aa m 2020. 10
Quan Cui, Bingchen Zhao, Zhao-Min Chen, Borui Zhao, Renjie Song, Jiajun Liang, Boyan Zhou, and Osamu Yoshic. Dicrimiaabllty-rrrsffraallll tradc-off: An information-thcoretia perspective. In ECCV, 2022a. 2, 3,
Quan Cui, Boyan Zhou, Yu Guo, Weidong Yin, Hao Wu, Osamu Yoshie, and Yubo Chen. Contrastive vision-language pre-training with limited resources. In ECCV, 2022b. 20
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A largo-scale hicrarchical image database. In CVPR, 2009. 1, 5,9
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021. 3
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina N. Toutanova. Bert: Pro-training of deep bidi- rectional transformers for language understanding. In NAACL, 2018. 3
13

---

Published in Transactions on Machine Learning Research (August/202))
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, In ICLR, 2021. 3, 8
Mark Everingham, SM Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015. 12
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the down- stream performance of pretrained self-supervised representations by their rank. In ICML, 2022. 6, 18
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In ICCV, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised
visual representation learning. In CVPR, 2020. 1, 2
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1, 2, 3, 4, 5, 7, 8, 9, 17, 20
Olivier J Hénaff, Skanda Koppula, Caan-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient. Visual Pretraining with Contrastive Delection. In ICCV, 2021. 3,3
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR, 2021a. 9
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. In CVPR, 2021b. 9
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019. 1, 2, 3, 5,
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 1
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022. 17, 18
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NewrlPS, 2020. 3
Wonjae Kim, Bokyung Son, and Tldoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In JCML, 2021. 3,
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. In ICML, 2023. 4
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-inage pre-training for unified vision-language understanding and generation. In ICML, 2022a. 4
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 4
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In ICLR, 2022b. 3
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft. coco: Common objects in context. In ECCV, 2014. 12
14

---

Published in Transactions on Machine Learning Research (August/2024)
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-
image pre-training. In ECCV, 2022. 1, 4,
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BETT v2: Masked image modeling with vector-quantized visual tokenizers. In NeurIPS, 2022. 1, 20
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2, 3, 4,
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nla Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 3
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. orXiv:2204.0111 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NewrIPS Workshop, 2021. 1, 3, 8, 19
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: cleaned, hyper- nymed, image alt-text. dataset for automatic image captioning. In ACL, 2018. 1, 3, 4, 8,
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. In AAAI, 2022. 1, 3
Ravid Shwartz-Zii Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv:1703.000 703.00810, 2017. 2,
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multinedia research. Communications of the ACM, 2016.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. ITW, 2015. 2,
H Touvron, M Cord, M Douze, F Massa, and A Sablay-rolles. Training data-cfficient image transformers & distillation through attention. In ICML, 2020. 3, 17
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. In NeurIPS, 2023. 4
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 2017. 3
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongic. The inaturalist species classification and detection dataset. In CVPR, 2018. 10
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, 2017. 3, 8
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervised visual pre-training, In CVPR, 2021. 1, 3
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. SimVLM: Simple visual language model pretraining with weak supervision. In ICLR, 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
15

---

Published in Transactions on Machine Learning Research (August/202))
Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual represen- tation learning with semantic grouping. In NewrIPS, 2022. 1, 3
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Newral computation, 1989. 8
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding, In ECCV, 2018. 10
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object-level representation learning from scene images. In NeurIPS, 2021a.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021b. 1,
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022. 3
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: "'ine-graine interactive language-image pre-training, In ICLR, 2022. 1, 3
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. TMLR, 2022. 3, 4,
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Danicl Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text luning, In CVPR, 2022.
Bingchen Zhao and Xin Wen. Distilling visual priors from self-supervised learning. In ECCV, 2020. 3
Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for neural networks for image processing. arXi::1511.08861, 2015. 8
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 2017a.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017b. 10, 11
Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrvstive learning by visualizing feature transformation. In ICCV, 2021. 3
Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Sepe-supe multimodal learning: A survey. TPAMI, 2024.
16

---

Published in Transactions on Machine Learning Research (August/2022 2004)
A Appendix
A.1 Implementation Details
Pro-training. The default setting is in Table 10, and hyper-paraneters mainly follow He et al. (2022) for lair comparisons. For each caption, we set the max length to 70 and use a percentage of 20% input words for processing, For each word, we mask it, replace with a random word, or delete it with a probability of 50%, 10,, and 40%, respectively.
End-to-end fine-tuning IN-1K, iNat-17, iNat- 18, and P365. We fine-tune models with the widely used recipe He ct al. (2022); Bao et al. (2022) in Table 11 except for the layer-wise decay rate is set to 0.7. On all evaluated datasets and pre-training models, we use the same setting for fair comparisous.
Table 10: Pre-training settings.
config
value
optimizer
AdamW
base learning
1.e-4
weight decay
0.05
optimizer momentum
91, 9900.9,0.00
batch size
4096
Linear probing. Settings are provided in Table 12. We follow He et al. (2022) to introduce extra BatchNorm layer without affine transformation for improving linear probing performances. For fair comparisons, all evaluated models are equipped with the extra layer.
learning rate schedule
cosine decay
warmup spochs
40 Raudom CropzeCppp
Image augmentation
text augmentation
mack/replacee replace/deleeeeeee
Av Ap
1.0
0.1
Semantic segmentation on ADE20K. The ADE20K We advise settings for ViT-S and ViT-I. by following Touvron
finettniin setting can be found this website. et al. (2020) and Bao et al. (2022), respectively. All evaluated pre-training models are fine-tuned with the same setting, and thus comparisons are fair.
Table 11: End-to-end fine-tuning settings.
config
value
Table 12: Linear probing settings.
optimizer
AdamW
base learning rate
1e-3
config
value
weight. docay
0.05
optimizer base learning
LARS
optimizer uomentum
31, 82=0.9,0.990
0.1
layer-wise decay batch size learning rate schedule
0.7
weight decay
0
1024
optimizer momentun
0.9
cosine decay
hatch size
16384
warmup epochs
5
training epochs
100 (B), 50 (L)
learning rate schedule
cosine decay
augmentation
RandAug 19, 0.5)
warmup epochs
10
label smoothing
0.1
training epochs
augmentation
90
RandomResizedCrop
mixup
0.8
cutmix
1.0
drop path
0.1
A.2 Transfer Learning with Visual Prompts
Recent progresses in transfer learning in the visual domain has demonstrated the effectiveness of using visual prompt tuning (VPT) (Jia et al., 2022). By adding small set of learnable tokens to the visual token sequence, VPT (Jia et al., 2022) can match, even outperform, the performance of fully end-to-end fine- tuning of the pre-trained model. Thus VPT has been seen as an effective method for tuning a pre-trained model. In this section, we demonstrate that MUG learned representations are effective for use by VPT by evaluating on several fine-grained benchmarks from the original VPT paper (Jia et al., 2022). The results are presented in Table 13. We can see that MUG improves the performance on all the data that we tested, indicating that the features learned by MUG are not only effective for full fine-tuning, but also effective for parameter efficient tuning, which is more practical in real applications.
A.3 More Diverse Evaluation
In this section, we provide linear probing results and visual prompt tuning results on additional datasets. These results complement the probing results in Table 4 and prompt tuning results in Table 13. We
17

---

Published in Transactions on Machine Learning Research (August/202))
Table 13: Comparison classification performance using visual prompt tuning (Jia et al., 2022) across different
pre-training methods and datascts.
method
data
#epoch
CUB
SCars
NABirds
MAE*
IN-1K
1600
68.4
67.7
65.2
MAE
CC3M
400
69.7
68.8
67.1
CLIP
CC3M
400
67.8
68.4
66.3
MUG (ours)
CC3M
400
70.1
69.3
67.4
MAE
W200M
10
70.2
69.0
67.2
CLIP
W200M
10
69.0
68.7
66.1
MUG (ours)
W20M
10
70.5
69.5
68.0
MAE
LAION100M
69.6
68.1
CLIP
LAION400M
5
69.8
68.9
67.3
MUG (ours)
LAION400M
5
70.6
69.7
68.8
that in terms of linear probing, both MAE and MUG do not achieve a good performance compared to other methods. However, in the visual prompt tuning experiments, MUG demonstrates strong performance. As demonstrated in the original MAE paper, linear probing results are not always correlated with the final transfer performance of the learned representations, as linear probing misses the opportunity to make use of strong, but non-lincar, features. Thus, we are more interested in the visual prompt tuning results as they belter demonstrate the learned representationns quality. The effective rank experiments in Table 16 also demonstrate that MUG can learn high quality representations.
Table 14: Linear probing results.
Linear Probing
Data
Food101
CIFAR100
Aireraft
DTD
Pets
Caltech 101
MoCoV3
CC3M
75.6
68.5
30.5
64.5
68.9
82.3
CLIP
CC3M
77.6
70.7
32.0
66.0
70.5
85.3
SLIP
CC3M
83.1
71.5
37.0
75.9
75.8
90.7
MAE
CC3M
70.4
63.4
25.6
61.3
63.1
76.9
MUG
CC3M
72.3
64.5
28.7
63.4
67.0
79.8
Table 15: Visual prompt tuning results.
Visual Prompt Tuning
Data
Food101
CIFAR100
Aircraft
DTD
Pets
Caltech 101
MoCoV3
CC3M
81.4
76.7
37.6
71.2
84.2
86.1
CLIP
CC3M
83.5
77.5
38.9
74.4
85.6
88.3
SLIP
CC3M
85.6
77.9
39.4
76.1
86.3
90.4
MAE
CC3M
85.1
76.8
39.0
75.3
84.7
88.7
MUG
CC3M
86.7
78.4
41.3
77.4
85.3
91.9
A.4 Assessing Transferability using RankMe
There have been a number of works that assess the performance of a pre-trained model on a downstream task prior to fine-tuning the model on the actual downstream dataset, e.g., Garrido et (2022); Agrawal et al. (2022). RankMe (Garrido al., 2022) one such method that assesses the performance of model using the elfective rank of the features extracted by the model a rewnstream task. This be viewed measuring part of the mutual information I(X; Z), as the rank of the features describe the variation of the features across dimensions. T(X; Z) describes not only the dimension-wise variation of the feature, but also the magnitude of variation within each of the dimensions.
As the measure of mutual information in high dimensional space is difficult (Belghazi et al., 2018), we leverage RankMe show that MUG can indoed learn superior visual representation. Here, this mcasured terms of having higher effective rank compared to other methods on features extracted from the ImageNet- 1k dataset. The results shown in Table 16. Overall, observe that discriminative methods like MoCoV3
18

---

Published in Transactions on Machine Learning Research (August/202))
and CLIP have lower effective rank than generative methods like MAE and MUG. Furthermore, MUG has the highest effective rank, indicating its ability to enlarge the information bottleneck of the model.
Table 16: Comparison of the effective rank of different pre-trained models ImageNet-Ik. rank is better.
Here, higher
method
data
#epoch
Rank
M3CoV3
W200M
10
413
MAE
W200M
10
674
CLIP
W200M
10
312
MUG (ours)
W200M
10
704
MAE
LAION400M
5
689
CLIP
LA1ON400M
425
MUG (ours)
LATON400M
5
792
A.5 Pre-training on Larger Datasets
In this section we demonstrate the scability of MUG by pre-training models larger datasets such LA1ON400M (Schuhmann et al., 2021) and W200M. We mainly evaluate the performance of models using ViT-B backbone. For pre-training models on different sized datascts, we keep the total number of itcrations the same across different datasets for a fair comparison.
Fine-tuning on ImageNet-1k. We first compare the results of larger pre-training datasets for different pro-training methods on ImageNet-1k using end-to-end fine-tuiig. Top-1 accuracy performance is reported in Table 17. We observe that similar trends from before still hold. Notably, MUG achieves the best results across IN-1k, IN-A, and IN-R. This demonstrate the scalability of MUG, as the performance of MUG still grows in line with the size of the pre-training dataset. In addition, MUG is significantly better than other multi-modal baselines.
Table 17: Comparison methods on larger pre-training datasets on ImageNet- 1k (IN- 1K) using end-to-end fine-tuning.
Table 18: Comparison of different pre-training methods using larger datasets ADE-20k using end-to-end semantic segmentation fine- tuning.
method
data
#epoch IN-IK IN-A IN-R
method
data
#epoch
mIoU
MAEs
IN-1K
1600
83.6
35.9
48.3
MoCoV3
W200M
10
82.1
27.9
41.7
MAEs
IN-1K
1600
48.1
MAE
W200M
83.3
35.3
50.4
MoCoV3
W200M
46.0
MAE
W200M
10
46.3
CLIP
W200M
79.8
25.6
12.7
CoCa
W200M
10
78.2
23.1
43.2
CLIP
W200M
10
40.3
MUG (ours)
W200M
10
83.7
39.0
51.8
MUG (ours)
W200M
10
48.5
MAE
LAION400M
83.4
35.6
51.0
MAE
LAION400M
5
48.9
CLIP
LATON400M
80.1
29.7
45.2
CLIP
LAION 400M
43.2
MUG (ours)
LATON400M
83.9
42.1
54.7
MUG (ours)
LATON400M
50.0
Fine-tuning on ADE-20k. We also provide comparison using larger pre-training datasets when fine- tuning for semantic segmentation downstream task ADF-20k. The results presented in Table 18. It can be observed that MUG can indeed improve the transfer learning performance the pre-training dataset grows. We note that MUG achieves the best transfer learning results, again indicating the effectiveness of MUG.
A.6 Further Discussions on the Information Bottleneck View
In Section 3.2, we discussed about the inspirations of preliminary benchmark experiments. In this part, we extend the discussion from the following perspectives:
Why does the single/ single/multi-man multimmodal discriminative method have narrow bottleneck? Since popular discriminative methods based the contrastive learning, the nature of such methods could be
19

---

Published in Transactions on Machine Learning Research (August/2024)
regarded the few-shot classification task, and the reason is follows. In each batch, we can regard each sample as category, and the few-shot learning setting is B-way 1-shot, where B represents the batch size. For instance, in SimCLR (Chen et al., 2020b), an image is processed by two data augmentations as positive pair, and "pulling close" the positive pair is actually minimizing a classification loss, i.e., the cross entropy loss. We believe the information compression rate of classification lask is relatively large, since the model only requires the most representative patterns to distinguish defined categories. Information irrelevant to the classification task will be compressed in the training process. Therefore, regarding discriminative methods classification tasks would explain the narrow information bottleneck, and thus the [[X; Za) would be relatively small because of significantly compressed input information. In addition, it also explains that contrastive learning pre-training methods require a large batch size (Cui et al., 2022b), since increasing batch size will increase the difficulty of the few-shot classification task, which further alleviates the information compression. The narrow bottleneck also suggests that such methods require a certain amount of pre-training data, and insufficient data could result in the over-fitting problem.
Why does the single-modal generative method have wide bottleneck? Popular single-modal generative methods are based on masked image modeling in encoder-decoder fashion, and the working mechanism of such methods could be regarded as the image reconstruction task. The image reconstruction task is much more challenging than the contrastive learning (or few-shot classification) task, because the model is driven to preserve as much information as possible for recovering details (concrete pixel values) in the raw input image. It explains the wide bottleneck, since the decoder cannot finish the reconstruction task without sufficient information of the encoder's output. Thus, the [(:; Zg) would be larger than I(X; Za), and the upper bound is HX).. Therefore, better transferring results are achieved (He et al., 2022; Bao et al., 2022; Peng et al., 2022) by using single-modal generative methods than discriminative methods. The wide bottleneck also explains that generative methods can still perform well with relatively small amount of pre-training data, as proven in He et al. (2022).
Why does the multi-modal generative method have a wider bottleneck? This work requires the model to reconstruct the raw input image and the coupled caption. It drives the model to preserve more information than the single-modal generative methods, since two objectives are involved and the decoder generates a joint distribution p(X", x²). can be wider than that of single-modal generative one. In this way, drive the [(X; Z) to approach H(XV, X4), which has the potential to learn more transferable representations.
( The information bottleneck of multi-modal generative method
How about scaling pre-training data? Note that the above discussions are based on a hypothesis that the pre-training data is fixed, e.g., CC3M in this work. Moreover, results in Section 5 prove that the transferring results of our proposed multi-modal generative method are the best. We further discuss the choice of pre-training methods when the pre-training data is scalable.
As aforementioned, single-modal discriminative methods could suffer from large information compression rate; however, we can increase the amount of pre-training data as compensation. To prove this, we pre- train the SimCLR (Chen et al., 2020b) with our collected W200M data, and then we transfer the model via TetgeNet-1k end-to-end fine-tuning. MAE (with W200M) and SimCLR (with W200M) achieve 83.3% and 83.4% top-1 accuracies, respectively. It means that, with large-scale data, the drawback of single-modal discriminative methods could be overcome. Besides, it also suggests that the scalability of single-modal generative methods can be inferior. However, our method MUG has good scalability by achieving 83. 7%. Therefore, we suppose that our proposed method can perform well different scales of pre-training data, which is an additional advantage.
A.7 Additional Visualization Results
We further provide visualizations of the reconstructed images as well as the generated captions produced by our model on randomly selected images on the MS-COCO and PASCAL-VOC datasets in Figs. and for qualitatively understand the model's behaviour.
20

---

Published in Transactions on Machine Learning Research (August/202/)
Gen treadced
Can Duo parced buidig
Gen Capo poaon slan next brown hoss
Gen poson atore leble with leate
Gon Eapula Hate bear aurerg wcroos luan gran Boa
Cars Cuss Cop Teer notts
Cer Cop
aamall progsBer plane aiiiin on
Con Cap- Icedy bear altling io windon
Cazs donn the with
acoe with
Ses. slardin fort burch
Capu sleshants stand in ted
Gan Cap blcck aftll phoro
offre atles
Cop ribre AUT ohithe De ELEEIN
Gen Cep. person on cour tatris
Cec. atroct SCOTe
couple peope wak
San Cap-a Sunting mater en the road
Sen Se
E4C siling a cTal! wh bont
Sen Cap. clone that soma gratte
Gan Cap. B cty Atrest at rigst Ap frait:
unzrell
Cas.:
the
srakebaudd
Cep: shancing
lsptop
San. CAp. Vara thiigg de ataia
Ger. Cao person AnCrn aCTOE a (OOW couered ground
Oup ballvorr Enc. mim ard
Cap. puzl: Iranet
m thre
unp missleo
Gat wowe 2itia af
F2h station wht
Trocks
Sabe dhe tonri ewaat
Ois PatzinAA Alrg sotowarzat tapoo
Usts
Cun Eapir LoUEUs
Lowess novse
Ocn Capr man Stnnd rg
bunue
Ger Cap. WoU man MasTo JANCOM SoTI
Cor. testt Bny 10l to taletpued
Oct. sunh teach
2and
PULK carked
Figure 5: Uncurated random samples on COCO images. For each triplet, we show the masked image (left). MUG reconstruction (middle), the ground-truth (right), and the generated captions by MUG. The masking ratio is 75%.
21

---

Published in Transactions on Machine Learning Research AAugust/202/)
Gen Cabu smmu motor sccote
Gen Capo whacpaaappppacoo surding
Cepo Lotrel beot tod middl
Sat. Capl man nairy aklisberr coon
Con Cap Hlac
witee day Hottn on couth
Can Cap zan truned ing cown oun truda rear Heee
Dan Cap callage of pactes nomarii
Con Cap. mun mrunaaaadit in the miedss di part
Cap. Iis EIaRinn
Cap- hone trak
nhe dit
Con. learing
bluc taz
app
Cap.l ciltt track
Cap: &E and nost
Cet. PIEET MRIS Stding reed
Gan. Cap siow rigs
Sen. Capura man AnC Women EEE postg
San Cap: paros zandhh in micdl of
Cen Cab. aitling tacia noctlo
Gen Dap man and nomann daanchng nent BRES
Gen. maanding
San
Gen Bid
fumm tur
Cep.
Cap. paman dancing
Ca dark wit slmp
Cap locking out sincow
Gar, Cap
ma Layfng with
AFEC wwd. of
1/19 couch windww
2.0 bout sillews
Den Cop Attreeronee Hell
REDALU
Gitt Cap J.n.o UsL DULC
Stat Sap. stry tr salty lool
CotT Cobure Aftss SiAll in poo Maaer
CAT Cup Sima
Gun Sapr Murz imoge of ou the SIUU
Oct Capo lad Th parding Rithg190
OLn Cups mun (
Gan Crpco Datt AELAEA of
Ocn. boxt mile miliee Bel
und in
Figure 6: Uncurated random samples on PASCAL-VOC images. For each triplet, we show the masked image (left), MUG reconstruction (middle), the ground-truth (right), and the generated captions by MUG. The masking ratio is 75%.
22

---

Published in Transactions on Machine Learning Research (August/202))
A.8 Limitations and Border Impact
One common limitation of models pre-trained on web sourced image-text pairs is that the models will inevitably be influenced by the open nature of internet data, such as biases or harmful speech. Thus the modols may learn unwanted behaviors that may cause actual harm to people. Given this limitation, resulting models should tested thoroughly before they are deployed in real-world applications.
23


Reviews:
[{"review_id": "mgfJEe407o", "reviewer": "Reviewer_FNLK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. **Extensive Experiments:** The paper includes a comprehensive set of experiments, comparing multiple state-of-the-art self-supervised learning methods using the same web-sourced dataset. This thorough comparison provides valuable insights into the strengths and weaknesses of different approaches.\n2. **Simple and Effective Method:** The proposed method, MUG, is straightforward yet effective, achieving superior performance across a variety of benchmarks. The design of MUG, leveraging both image and text modalities, demonstrates strong generalization capabilities.\n\n## Weaknesses\n1. **Lack of Novelty:** While the method is effective, it lacks significant novelty. The approach is mainly an incremental improvement over existing methods, combining ideas from both single-modal and multi-modal self-supervised learning, such as MAE and VirTex. The motivation of this work needs to be clarified.\n2. **Linear Probing:** This experiment can be conducted on more diverse datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper titled “Vision Learners Meet Web Image-Text Pairs” explores self-supervised pre-training methods using large-scale web-sourced image-text paired data. The authors conduct a benchmark study comparing various single-modal and multi-modal self-supervised learning methods. They introduce a new visual representation pre-training method, MUlti-modal Generator (MUG), which consists of masked image modeling and language modeling. The paper includes extensive experiments and thorough analyses to validate the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3GSAEMg3JS", "reviewer": "Reviewer_8qb7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper offers clear and comprehensible explanations.\n\n2. It demonstrates strong performance across a wide range of tasks.\n\n## Weaknesses\n1. The approach presented seems to be a simple combination of existing methods, the image captioner and MAE, lacking substantial technical insights. The Image Captioner (MAC) framework appears to be essentially a version of a previously published work [1] with the addition of masks, and the final solution MUG is essentially MAC combined with MAE. Prior research, such as SLIP [2], has already demonstrated the benefits of joint training across different modalities, making the performance of MUG somewhat expected. The authour should give a more clear technical insight for this paper.\n\n2. The linear probing experiments should be evaluated on more datasets (e.g. Food, Cifar, Cars, Aircraft, DTD, Pets, Caltech ...)\n\n3. Some of the result is missing in Table 4, e.g. SLIP gets 65.4% in Linear Probing.\n\n4. Important reference is missing [3]\n\n[1] Image captioners are scalable vision learners too. NeurIPS 2023.\n\n[2] SLIP: Self-supervision meets language-image pre-training. ECCV 2022.\n\n[3] RedCaps: Web-curated image-text data created by the people, for the people. CVPR2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an innovative pretraining framework for visual representation learning. Initially, it presents the Masked Captioner (MAC) as a pretext task using image-text pairs. Moreover, it enhances this framework by integrating the Masked Image Modelling (MAE) method. This combined approach demonstrates good performance across various downstream tasks, including linear probing, fine-tuning on In-1k, and image segmentation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "See Weakness", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kJLdBXyR7T", "reviewer": "Reviewer_pTSC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper includes extensive benchmark study of self-supervised methods on image-text pairs, confirming the superiority of generative pretraining approach. \n- The paper considers an information-theoretical perspective in explaining their experiments.\n- The proposed MUG seems to be effective in vision representation learning.\n\n### Weakness\n- I think there are more reference self-supervised learning tasks that authors should consider. For example, DINO [1, 2] is a powerful self-supervised learning method that has shown empirical performance of visual transfer learning. Also, the proposed MAC is similar to Cappa [3], where the difference is in masking the image and using bi-directional attention masks. \n- The benchmark study seems to be extensive in that it considers various methods, yet the evaluation is only made on ImageNet fine-tuning tasks with only Top-1 accuracy is reported. If the goal is to achieve transferable visual representation, the benchmark results should consider performance on various transfer learning tasks. Also, it would be helpful to show the scaling behavior of various methods. \n- The author considers information bottleneck theory, which seems viable, yet it would be better to show quantitative comparison that validates the authors’ claim. Most of the claims are based on assuming optimal cases, which might not be held in practice. Thus, to better validate the authors’ claim that generative pretraining with multi-modal data achieves wider bottleneck and it results in better transferable representations, the authors should consider some quantitative metrics to show it.\n\n### Reference\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\\\n[2] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\\\n[3] Tschannen, Michael, et al. \"Image captioners are scalable vision learners too.\" Advances in Neural Information Processing Systems 36 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers visual self-supervised representation learning given the image-text pairs. First, the authors conducted a benchmark study to compare different self-supervised methods, e.g., contrastive learning (image only), masked autoencoders (MAE), and image-text contrastive learning (i.e., CLIP). They observe that generative approaches e.g., MAE or masked captioners (MAC) perform the best in ImageNet fine-tuning experiments. Based on their benchmark study, the paper introduces Multi-modal Generator (MUG), which learns visual representation by reconstructing an image and a caption given a masked image. They show that MUG outperforms single-modal and multi-modal pretraining on ImageNet fine-tuning, and various transfer learning tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Most of my concerns are written in Weakness. In specific, the requested changes are given as follows:\n- Adding missing but relevant methods to comprehensive benchmarks, e.g., DINO and CapPa. \n- Also, add multiple evaluation tasks in benchmark study.\n- If the authors can show the scaling behavior in benchmark study, it would be much better.\n- To validate their theory based on information bottleneck, add a quantitative metric that correlates with information bottleneck.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qmXjBbS5fn", "reviewer": "Reviewer_SaAu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper is well written and easy to follow\n2) The paper attempts to pre-train models on the same web-scale dataset which helps in performing direct comparisons. \n3) A simple approach (MUG)  is presented which outperforms the compared works.\n4) Interesting observations that might be useful to the community\n\nWeaknesses:\n1) How does the presented new approach compare with works like [a,b] and that line of work ? \n2) While, I do understand that this area has received a lot of interest recently and there are too many approaches out there, it would also be great to comment on approaches that have obtained improvements over SimCLR, CLIP and MAE perhaps by including comparisons with the state-of-the-art in each category. \n3) It would also be beneficial to include some discussion on overlap of the pre-training datasets with ImageNet-1K and such. \n4) Figure 2, theoretical insights: I am not sure if it is completely obvious that say single-modal generative approaches for example learn better representations than discriminative ones [c] \n5) Some comparison on time and/or resources needed to train will also be helpful  \n\n\n[a] Multimodal Masked Autoencoders Learn Transferable Representations\n\n[b] Scaling Language-Image Pre-training via Masking\n\n[c] What Do Self-Supervised Vision Transformers Learn?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper presents a benchmark study on various self-supervised pre-trained models including contrastive learning, masking-based and image-text multi-modal learning. \n- Theoretical insights are presented on potential reasons for why multi-modal approaches underperform single-modality ones.\n- A new approach called MUG is presented that is shown tu be beneficial on various downstream tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses section. I am particularly interested in answers to questions (1,3,4 and 5). (2) is good to have but perhaps difficult to finish during a rebuttal.", "broader_impact_concerns": "Not present in the paper but I do not expect any obvious concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers agree that the observations are interesting and the experiments are comprehensive. The primary weakness noted by the reviewers is the technical novelty of combining masked image modeling and language modeling as the proposed pre-training method MUlti-modal Generator. While this strategy seems straightforward and incremental, it has been extensively validated to show its effectiveness. According to the TMLR's evaluation criteria, the AE believes this work still has merits for publication. 

The AE also checked the authors' responses to reviewers' comments and found additional results and discussions adequate. The AE thus recommends "Accept with minor revision". In the revision, please update the paper with new results and promised modifications/clarification.
Decision: Accept (poster)