Improved baselines for vision-language pre-training
Authors: Enrico Fini
Pietro Astolfi
Adriana Romero-Soriano
Jakob Verbeek
Michal Drozdzal
Venue: Accepted by TMLR ()

Abstract: Contrastive learning has emerged as an efficient framework to learn multimodal representations. CLIP, a seminal work in this area, achieved impressive results by training on paired image-text data  using the contrastive loss. Recent work claims  improvements over CLIP using additional  non-contrastive losses inspired from self-supervised learning. 
However, it is sometimes hard to disentangle the contribution of these additional losses from other implementation details, \eg, data augmentation or regularization techniques, used to train the model. To shed light on this matter, in this paper, we first propose, implement and evaluate several baselines obtained by combining contrastive learning with recent advances in self-supervised learning. 
In particular, we use the loss functions that were proven successful for visual self-supervised learning to align image and text modalities. We find that these baselines outperform a basic implementation of CLIP. However, when a stronger training recipe is employed, the advantage disappears. Indeed, we find that a simple CLIP baseline can also be improved substantially, up to a 25\% relative improvement on downstream zero-shot tasks, by using well-known training techniques that are popular in other subfields. Moreover, we discover that it is enough to apply image and text augmentations to make up for most of the improvement attained by prior works. With our improved training recipe for CLIP, we obtain state-of-the-art performance on four standard datasets, and consistently outperform prior work (up to +4\% on the largest dataset), while being substantially simpler.

Published in Transactions on Machine Learning Research (10/2023)
Improved baselines for vision-language prenraininn
Enrico Fini Pietro Astolfi- Adriana Romero-Soriano 3,,,,4 Jakob Verbeek Michal Drozdzal
FAIRR, Meta, 9 UUniversity of Trento, Mila, Quebec AT Institate, 4MeGill Universiby, & Canada CIFAR AI Chair A enrico, enniccofini@@matt fini@gmail. com, ppietoastoli@mmeta.co +. --ootributeed equally
Reviewed on OpenReview: http:://ppemreeseeeee//ffrrrfr net/
Abstract
Contrastive learning has emerged as an efficient framework to learn multimodal representa- tions. CLIP, a seminal work in this arca, achieved impressive results by training on paired image-text data using the contrastive loss. Recent work claims improvements over CLIP using additional non-contrastive losses inspired from self-supervised learning. However, it is sometimes hard to disentangle the contribution of these additional losses from other implo- mentation details, e.g., data augmentation or regularization techmiques, used to train the model. To shed light on this matter, in this paper, we first propose, implement and eval- uate several baselines obtained by combining contrastive learning with recent advances in self-supervised learning. In particular, we use the loss functions that were proven successful for visual self-supervised learning to align image and text modalities. We find that these basclincs outperform a basic implementation of CLIP. However, when stronger training recipe is employed, the advantage disappears. Indeed, we find that simple CLIP base- line can also be improved substantially, up to a 25% relative improvement on downstream zero-shot tasks, by using well-known training techniques that are popular in other subfields. Moreover, we discover that it is enough to apply image and text augmentations to make up for most of the improvement attained by prior works. With our improved training recipe for CLIP, we obtain saate-of-the-ate performance on four standard datasets, and consistently outperform prior work (up to +4% on the largest datasct), while being substantially simpler. The code is available at https: //githb//bbb/t/// con/facceookkeeeeardee clip-rocket.
1 Introduction
Vision-language pre-training (VLP) is recent learning paradigm that enables neural networks to learn multi- modal representations from images and text. The learned representations are inherently semantic and can be easily transferred to several tasks, such as zero-shot image classification and multimodal retrieval. Arguably, the most prominent approach to VLP is CLIP (Radford et al., 2021), which leverages a contrastive loss to learn aligned image and text representations from large-scale image-text datasets scraped from the internet..
Notwithssanding its success, CLIP is not without its limitations. These include high computational cost and a strong sensitivity to the size and composition of data batches used during training due to the contrastive approach, as well as other limitations such as lack of fine-grained alignment between image regions and words in a sentence, and the absence of cross-modality feature fusion. Morcover, the standard implementation of CLIP adopts a rather basic training recipe, which does nol. incorporate commonly used training techniques such as image and text augmentations (Shorten & Khoshgoftaar, 2019), label smoothing (Müller al., 2019), and architectural improvements like projector networks (Chen et al., 2020; Grill et al., 2020), which have been shown enhance generalization and robustness in supervised and self-supervised representation learning (SST.).

---

Published in Transactions on Machine Learning Research (10/2023)
Motivated by these limitations, recent VLP research has sought to improve upon CLIP by modifying the contrastive objective and integrating advanced training techniques. Tn particular, to improve the contrastive objective, some works (Yao et al., 2021; Gao et al., 2022; Fürst et al., 2022; You et al., 2022) have tried to tackle the fino-grained imago-text alignment or cross-modal feature fusion. Others, e.g.. SLIP (Mu et al., 2022) and DeCLIP (Li el al., 2021), have regularized the objective with auxiliary sell-supervised image-to- image losses. However, the contrastive approach remains a limiting factor. For this reason, some rescarchers have suggested using vision-language non-contrastive losses (Chen et al., 2022; Zhou et al., 2022; Singh et al., 2022). These losses reminiscent of SSL, where they have been theoretically well-studied (Tian et al., 2021; Balestriero & LeCun, 2022), and their translation to VLP has the potential to solve the challenge of the batch size sensitivity and heavily reduce the computational cost. However, such existing methods do not provide faithful VLP translations of SSL losses and have not explored many well-known ron-contrrsttive losses successful in visual SSL, e.g., SwAV (Caron et al., 2020), BYOL (Grill et al., 2020), SimSiam (Chen & He, 2021), Barlow Twins (Zbontar et al., 2021). Furthermore, as in many CLIP extensions, the dual adoption of modified objectives and advanced training techniques, which despite re-implementations (Ilaarco el al., 2021; Li et al., 2021) are not employed by baseline CLIP, makes it hard to evaluate the impact of such methods. It emerges that the VLP literature is far from standardizing the training recipe and would benefit from the analysis of advanced training recipes to clarify the landscape of the ofate-of-the-art.
In this paper, we aim to elucidate the state of the art for VLP by systematically investigating ron-cont extensions of CLIP, and tuning an improved training recipe for CLIP that can reveal the actual impact of ours well as existing CLIP extensions. In particular, we present four new VLP basclines: SiamLIP, BYOLIP, BarLIP, and Sw by translating the non-constrastive losses of the SSL methods, SimSiam, BYOL, Barlow Twins, and SwAV to the multimodal case. We test these methods in the zero-shot image classification task and we find that they cannot learn strong aligned multimodal features on their own (as also found by concurrent work (Zhou et al., 2022)), but can be used in composition with the CLIP loss to boost the performance compared to a standard CLIP baseline. However, we also find that this improvement is much smaller than the increase in performance brought about by advanced training techniques. Indcod, we carefully explore training recipes for VLP, by combining and evaluating the training techniques that are best practices in the supervised (Dosovitskiy et al., 2020; Wightman et al., 2021) and self-supervised (Chen et al., 2020) literature. This results in the proposal of an improved training recipe, dubbed H. that comprises image and text augmentations (Shorten & Khoshgoftaar, 2019; Shorten et al., 2021), dropout (Srivastava et al., 2014), and label smoothing (Mäller et al., 2019), combined with architectural design choices (i.e. design of the projection layers). By pre-training on four public datasets of various sizes, we discover that when employed for CLIP (CLIP& 3). this recipe substantially improves its zero-shot ImageNet performance (up to 11%), so much that it surpasses of up to 4% all previously published results obtained with more complex CLIP extensions, including our non-contrastive baselines also trained with the same improved recipe. We conclude that, although methodologically interesting, these more complicated models do not provide enhanced results. In practice it is better to stick to well-tuned implementation of CLIP than rely on existing additional losses.
Our contributions can be summarized as follows:
We systematically analyze non-contrastive losses for vision- language pre-training by proposing four baselines that outperform a standard implementation of CLIP by few points (122%) on Tetg
We collect and study advanced training recipes for CLIP, and propose a well-tuned improved recipe, CLIP 7, that dramatically boosts its performance up to 11% on ImageNet (relative boost ~25%);
We reset the state of the art for VLP by showing that CLIP o is better than met employing additional modified contrastive and non-contrastive losses when pro-trained four datascts, ranging from 3M to 29M image-text pairs. In particular, using the largest dataset we outperform by 4% the best related method.
2 Background
Let us consider a dataset D = {( WM of M paired data points x, and y, representing the same underlying concept. In our manuscript, D; represents an image and y, represents either a data augmentation
2

---

Published in Transactions on Machine Learning Research (10/2023)
textual description of z;- Moreover, let us define two encoding functions A and B that project input data into representations: z = A(x;) and = B(y,). Note that if x; and y, are from the same modality (e.g.. images), A and B may represent the same encoding. The objective of representation learning is to learn robust encoding function that produces representations that gencralize well to diverse downstream tasks.
2.1 Contrastive learning
In contrastive learning, the main idea is to minimize the distance between different representations 2 and z of the same data pair (mi, U:) - positive pairs and maximize the distance between representations z and z from different data pairs (x,,) and (z) - negative pairs. Formally, this is achieved by optimizing the following objective:
N
exp (sim (2 22 /T) , /T)
c CA-BB
N log ΣØ= exp (sim
(1)
where the summation is over N elements in the batch, T is the lemperature hyper-parameler and sim(s.) is the cosine similarity between two vectors. Note that if A and B are different, the loss is not symmetric; we use the superscript A B, to denote the direction of similarity computation. Nevertheless, the contrastive loss can be made symmetric by considering both similarity computation directions:
Lo C + L
(2)
The contrastive objective in Eq. 1 is used both SST. where y; represents an augmented version of æ; and VLP where y, represents the text description associated with æs, CLIP (Radford al., 2021) is a de facto standard for VLP which leverages cross-modal contrastive learning. In practice, the model leverages vision encoder and text encoder to obtain image and text representations, respectively. In the typical implementations, the vision encoder is either a ResNet (He et al., 2016) or a vision transformer (Dosovitskiy et al., 2020) while the text. encoder is a text (ransformer that processes the input lext to produce a fixed-length vector representation.
Limitations of contrastive Icarning. One disadvantage of the contrastive learning in CLIP that the quality and diversity of the learned features can be affected by the choice of negalive samples (Chen et al., 2020; He et al., 2020). If the negative samples are not diverse enough, the model may learn to distinguish between only small subset of dissimilar samples, leading to poor performance. For this reason, the contrastive loss is usually optimized using large batch sizes, which are more likely to carry larger sample diversity (Pham et al., 2021; Zhai et al., 2023). However, increasing the batch size also increases the memory footprint of the model. This issue is exacerbated by the fact that the size of the similarity matrix scales quadratically with the batch size. Another disadvantage of the contrastive loss that may not be tolerant to semantically similar samples (Zhong et al., 2021). For instance, the batch contains many instances of the same concept it may map them far apart in the feature space, breaking the underlying semantic structure and harming the emergence of features that are useful for downstream tasks. These limitations apply to both contrastive SSL and VLP approaches, including CLIP. However, in practice, the second limitation may be mitigated when training on very broad data distributions, as typical for CLIP, because it is less likely to simultancously sample rodundant concepts. However, CLIP is not exempt from this issuc when trained on small da datasets, and it is worth mentioning that a recent study, Abbas et al. (2023) observe a high degree of redundancy also in well-known large-scale datasets such as LAION (Schuhmann al., 2022).
2.2 Non-contrastive learning
To overcome the limitations of contrastive learning discussed in Sec. 2.1, non-contrastive learning approaches have boen introduced in the literature. In particular, non-contrastive techniques have emergod and boen widely explored in self-supervised visual represenlation learning, where they have repeatedly shown to be effective. These approaches do not rely on the use of negative samples during training, but instead impose consistency among positive pairs (Chen & He, 2021; Grill et al., 2020), replace negatives with redundancy reduction terms (Zbontar et al., 2021; Bardes et al., 2021) or cluster prototypes (Caron et al., 2018; 2021).
3

---

where À is weight and Published in Transactions on Machine Learning Research (10/2023)
Consistency-based methods. Consistency-basee methods, such as BYOL (Grill et al., 2020) and Sim- Siam (Chen & He, 2021), optimize consistency objectives like mean squared error (MSE) negative cosine similarity loss only between positive samples, therefore avoiding the use of negative samples in the loss. This not only reduces the memory requirements during training as the similarity between all pairs of samples in a batch no longer needs to be computed -, but also results in better performance when training with small batch sizes. The loss function of these methods can be written as follows:
N Σ
AABB
=
11. - 2,1
(3)
1=1
where of is the output of a predictor network that takes as input z The encoder B is either a momentum encoder in the case of BYOL or the same encoder as A in the case of SimSiam. In principle, this loss function is commutative, however, since these methods exhibit an asymmetric architecture - duc to the presence of the predictor the loss can be made symmetric by swapping the encoders and B and attaching the predictor network to the output of the encoder B, similar to Eq. 2.
Redundancy reduction methods. Redundancy reduction methods represent another example of meth- ods using a non-contrastive loss in visual SSL. We identify three methods that belong to this category: Barlow Twins (Zboutar et al., 2021), VICReg (Bardes et al., 2021) and W-MSE (Ermolov al., 2021). For simplicity, here we describe the mechanism and the loss function in Barlow Twins, the most influential of these three methods, but similar arguments can be made for all of them.
Barlow Twins (Zbontar et al., 2021) uses a cross-correlation loss that decorrelates feature components and maximizes variability of learned representatious. More precisely, the loss measures the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of sample and tries to make it as close to the identity matrix as possible. This causes the represenlations of distorted versions of sample to be similar while minimizing redundancy between the components of these representations. The loss function reads as:
Exc = (1- Exo=E(1) c +1 ++EΣ a
C 1D9
(4)
a
Disl 2 2,
N
B
Cuv A
Em (z
2
EN (x
is the cross-correlation matrix between x/ and N along the batch dimension." In the case of Barlow Twins, A and B represent the same encoding function.
ceattering-based methods. The third and last family of visual SSL methods is based on clustering. We identify three methods that belong to this category: SwAV (Caron et al., 2020), DINO (Caron et al., 2021), and DeepClusterV2 (Caron el al., 2018). By means of clustering, these methods disccretize the feature space to learn representations, which enable them to use the cross-entropy loss to compare cluster assignments of different views of the same sample, follows:
K
exp (sim (2 Ps) /T) ΣK= exp (sim (z .P.) /T)
AA X
R a log
(5)
where Ps is the k-th learnable cluster prototype, af the vector of cluster assignments of 2,, and K is the number of clusters. In the case of SwAV, A and B represent the same encoding function, while in DINO B is a momentum encoder. Similarly to consistency-based methods, this cooss-entopyy-bsee loss is asymmetric, but can be made symmetric by swapping A and B. In the remainder of the manuscript we focus on SwAV. In SwAV, assignments are obtained by using the Sinkhorn- Knopp algorithm (Cuturi, 2013), which approximates optimal transport by iteratively normalizing the rows and the column of the matrix containing the similarities of samples in the batch, ". and the prototypes PR.Vec(....)) This equates to imposing that the clusters are roughly equally represented in the batch.
We use the double subscript notation 2, indicate j-th element of the vector 21-
4

---

Published in Transactions on Machine Learning Research (10/2023)
CLIP Loss
Weak Projector
Projector
"aaychede browpal
Image Encoder
Text Encoder
Ieom anothez dimene
Strong Aug.
Strong Projector
CLIP Loss
Strong Projector
Strong Aug.
Figure 1 Overall architecture of our CLIP
model.
3 Improved training recipe
Our improved training recipe, which we denote with H, is based on the original CLIP training recipe but incorporates the following important modifications: we incorporate stronger data augmentations, introduce new projector, add label smoothing and additional regularization for the lext encoder.
Image augmentations. In CLIP, the only augmentation, i.e. weak augmentation, is obtained by ran- domly cropping large portion of the imput image. Instead, in our improved recipe we generate additional views of the same input image by applying multiple image transformations that produce strong augmen- tations. Specifically, inspired by the best practices in supervised (sce, e.g., Wightman et al. (2021)) and self-supervised learning (see, e.g-; Chen et al. (2020)), we first crop the image by randomly sampling the crop scale from a wider range than the one used in the weak augmentation; then, we sequentially apply a random combination of color distortions such as contrast, brightness and saturation jittering and color dropping (greyscaling). Gaussian blurring, and horizontal flipping.
Text augmentations. We borrow text augmentations from the NLP litcrature, where variety of trans- formations that do not alter semantic have been presented, see, e.g., Shorten et al. (2021). We generate the weak augmentation by simply removing stop-words with an arbitrary probability. Note that this trans- formation has been already shown to be useful for vision-language pre-training by Tejankar et al. (2021). Moreover, to generate stronger augmentations, independently and uniformly sample one caption among them for each augnentation, herwiss,, we use the same caption for all the augmentations. Then, we combine stop-word removal with a one randomly sampled transformation among synonym replacement, word insertion deletion, insertion/deletion, and sentence shuffling, as suggested by Wei & Zou (2019). Note that the original CLIP training recipe does not exploit any text augmentations.
multiple captions are available for the same image, we
Projector design. CLIP employs linear projectors to embed text and image representations into the same space. However, in the vision literature multi- projectors have emerged as an indispensable element when employed in combination with strong data augmentation. This fact has been empirically proved in multiple learning scenario, e.g. supervised (Khosla et al., 2020; Sariyildiz et al., 2023), semiisupprrvsee (Fini et al., 2021; 2023; Assran et al., 2021), and self-supervised (Caron el al., 2021), and an intuition of the explanation is provided by Bordes et al. (2021). Grounded by this, we adopt multi-layer perceptrons (MLP), consisting of two linear layers separated by BatchNorm and ReLu layer strong projectors for both image and text, while we keep linear projectors as in original CLIP to embed weak image and text augmentations.
Label smoothing. Label smoothing (Szegedy et al., 2016) discourages the model from being overconfident in its predictions. This is especially helpful when the dalaset is noisy or the data augmentation recipe is so strong that it can change the semantics of the input, or the model overfits the training distribution. CLIP since we have both strong augmentations and deep projectors, we propose to soften the identity matrix otherwise used target in the contrastive loss (see Sec. 2.1). With the same logic, we do not apply label smoothing when computing the contrastive loss between weak image and text augmentations embedded with linear projectors.
5

---

Published in Transactions on Machine Learning Research (10/2023)
Overview of our recipe An overview of CLIP & approach is depicted in Fig. 1 and in the pseudo-code of Pseudo-code (see appendix for the exact implementation). At each training step, we load batch of lext-image pairs. Each pair augmented creating one weak and one or more strong augmentations of pair 2, The aug- mented images and texts are then fed to the im- age and text encoders, respectively. Importantly, we apply dropout to the text encoder (trans- former) to regularize it. Then, the embedded weak and strong angmentations are projected into the shared cross-modal space using two sepa- rale projectors (weak and strong projector). The model is trained with the standard CLIP loss (see Eq. 2) applied to the corresponding views of image and text, i.e. projected weak image augmentations are contrasted to projected weak text augmentation, and vice-versa for projected strong image and text augmentations. When computing the loss between strong augmenta- tions we apply label smoothing. At inference time, average the similarities predicted by the two projectors (weak and strong) to obtain the final similarity score between image and text.
Pseudo-code 1 CLIP / training procedure
This code describes the case of seak and strong aug x, y: inage and correaponding tau_veak, rau_strong learnable tenperatures label_smooth: smooth: label smoothing paraneter image_ text_ encodera def training_step(a. y):
(posaibly w/ dropout)
a weak sugnentations
xeaa anage_augnent.e (x, mode="geakk")
y_veak text_ text_augnent(y, node="Reak")
strong augnentations
x s_atrong = inage augnent (x, sodo="strong") y y_etrong text_augnent (y, mode='strong")
# forvard seak
s_inage_vert _eek prage (ena (x_ (x_
# forvard strong
2_inage_atron inaga_prot panga_proj_aaronne (inage end attrngg))
2_text_etrong test_proj__sroog((teee (text___eeeeeeeeeee (y_stry))og))
compute lass
less _eak contrastive_loge
z_inage_vesi,
z_text_vaak,
tonperaturestau_vek
loss_ctrong contrastiva_loss( 2. sanage_atrong,
2_text_strong
conperaturertan.ttee label snooth=label. smooth
)
return loss_Neak loss_atrons strong
4 Non-contrastive baselines for vision-language pre-training
In Sec. 2 we reviewed the use of non-contrastive losses in SSL. Here, we propose to incorporate such losses into the vision-language pre-training of CLIP, with the idea of mitigating the issues of the CLIP contrastive training objective. In our non-contrastive baselines, we consider the following training objective:
L = aLc + BLNC,
(6)
where Le is the contrastive loss of CLIP, Lxc is a non-contrastive auxiliary loss and a, 3 E [0, 1] are weights. Note that both Le and Lxe are intended as cross-modals between image and text. Note that, setting a = and 3 = equivalent to using only CLIP loss only, while a 0 and 3 results in non-contrastive multi-modal learning. In our experiments, we found that setting = B worked best. See supplementary for the full ablation.
In this section, we consider four types of non-contrastive losses and combine them with CLIP to build four models: SiamLIP, BYOLIP, BarLIP, and SwaVLIP. Unless otherwise specified, all the basclines use multi-layer projector to refine the features.
SiamLIP & BYOLIP. SiamLIP BYOLIP (see Fig. 2a) are two VLP models that complement the contrastive training objective of CLIP with a consistency-bused objective. In the case SiamLIP & BYOLIP, the complemenaary training objective inspired by SimSiam (Chen He, 2021) and BYOL (Grill et al., 2020). For these models, we define Lxc to be the cross-modal version of LMSE (sce Eq. 3), computed in both directions A + B and B + A, where A is the image encoder and B is the text encoder. The use of an asymmetric training objective requires the architecture to have two projectors followed by two predictors, mapping from A + B and B + A, respectively. Moreover, following BYOL, in BYOLIP we use momentum encodors for both image and text modalitics.
2'hroughout manuscript, we refer to original CLIP sugmentations as weak augmentations.
6

---

Published in Transactions on Machine Learning Research (10/2023)
Cross- Entropy Loss
Cross- Entropy Loss
MSE Loss
MSE Loss
Cross- Loss
detach()
detach()
Pred
Pred
Proto
Proto
Image Encoder
Text Encoder
Image Encoder
Text Encoder
Image Encoder
Text Encoder
"Esychedelis brosool1 from ansthez cimension"
"Paychedei ie brecmll from enother dimensiion
Faychedee lig besewel from: anothez dimension"
(a) SiamLIP & BYOLIP
(b) BarLIP
(c) SwALIP
Figure 2: Overview of the proposed improved baselines: SiamLIP, BYOLIP, BarLIP and SwALIP. For the sake of visualization, we do not show the CLIP loss used in combination to the non-contrastive loss and the projector networks all the modcls. For BYOLIP we do not show the additional momentum encoder.
BarLIP. BarTIP
(see Fig. 2b) VLP model that combines CLIP objective with a redundancy reduction criteria. in particular, we take the inspiration from Barlow Twins (Zbontar et al., 2021) and adapt Eq. the VLP scenario and use it as the non-contrastive loss (CNc). Contrary to Barlow Twins, in BarLIP we use two encoders - A for vision and B for text inpuls. The multimodal version of Eq. 4 aligns the feature space of both data modalities while also decorrelating its components, which should help when pre-training on noisy and duplicate-rich datasets typical of vision-language pre-training.
SwALIP. SWALIP (see Fig. 2c) is a VLP model that combines CLIP objective with clustering- criteria. particular, we get inspiration from SwAV (Caron et al., 2020) and introduce the cross-modal adaptation of the cross-entropy loss, LxE (see Eq. 5), to compare the cluster assignments of different rep- resentations of the works texts. The learned set of prototypes are shared across the two modalities, and basically partitions concepts in both modalities in the aligned feature space. The cross-entropy loss is computed in both direc- tions enforcing that paired image and text samples belong to the same cluster. However, in practice, this formulation for AWALIP turns out to be quite unstable, and for most our experiments we adopt a slightly modified version SwALIP that does not exhibit learned protolypes, rather, it uses representations from the text modality as prototypes. Note that this modified formulation computes the swapped assignments using correlated views of each image-text pair; hence, it requires multiple views as in our improved training recipe, and cannot be used otherwise. We refer to the appendix for additional details and the formalization of modified SwALIP.
concept. In this case, the encoder A takes
input images and the encoder B
5 Experiments
5.1 Datasets and evaluation
We pre-train the presented CLIP variants on the datasets described below, ranging in size from 3M to 29M. We test the models in the zero-shot image classification task, which is performed by computing the cosine similarity between the image representation and the representation of all the classes encoded as text and choosing the most similar class. The performance is measured in terms of accuracy on the ImageNet- 1000 (Deng ct al., 2009) validation set. Morcover, following Radford et al. (2021), we investigate the modcls' generalization ion using extended set of 22 vision benchmarks of different kinds most of them belong to the widely adopted Visual Task Adaption Benchmark (VTAB) (Zhai et al., 2019).

---

Published in Transactions on Machine Learning Research (10/2023)
The Conceptual Captions dataset is composed of image-caption pairs that have been filtered based on: (i) image properties and content; (ii) caption language, information richness, and content; (iii) alignment between image and caption; and (iv) hypernymization and text refinement. Given this pipeline, two datasets have been published in the literature: CC3M (Sharma et al., 2018) composed of 3.3M image-text pairs obtained by applying the full filtering pipeline, and CC12M (Changpinyo el al., 2021) comprising 12.4M pairs obtained from a relaxed version of the pipeline (relaxed uni-modal filters (i), (ii), and (iv)).
The Yahoo Flickr Creative Commons datasct is composed of 100M of image-text pairs (Thomoe et al., 2016). Noisy lext-image pairs have been filtered by Radford et al. (2021) to obtain a cleaner version of 15M pairs. This dataset is referred to as YFCC15M.
To experiment with larger scale datasets we follow Li et al. (2021) and create a dataset as the union of CC3M, CC12M, and YFCCI5M, obtaining 29M image-text pairs. We refer to this dataset as Open29M. Moreover, we further scale the dataset size by adopting PMD46M a 46M subset of the PMD70M dataset (Singh et al., 2022) containing image-text pairs gathered from different data sources including the whole Open29M.
5.2 Implementation details
We train models using both the base recipe used by CLIP and our improved training recipe. We provide further implementation details below.
Architecture. We focus our main analysis CLIP variants using ResNet-50 (He et al., 2016) the vision backbone. However, we also investigate vision transformers (Dosovitskiy et al., 2020) like ViT-S/16 16 and ViT-B/16. The text encoder is kept as in the original CLIP, except that we change its number of layers based on the dataset size, e.g., for smaller scale dataset like CC3M we use 6-layer transformer, while for larger datasets we use 12 layers. Morcover, the strong text and vision projectors are two separate MLPs with BatchNorm and ReLU, each having a hidden and output dimension of 4096 and 256, respectively.
Optimization. In most experiments, we pre-train the model for 32 epochs following Li et al. (2021), using the AdamW optimizer (Loshchilov & Hutter, 2017) (betas 0.9 and 0.98), with learning rate 0.003 (or 0.002 for experiments on the 29M datasct) regulated by lincar warmup (1 epoch) plus cosine scheduler (final Icarning rate 50. Mini- batche are composed of 4096 image-text pairs. To provide regularization to the training, weight decay is applied with magnitude 0.1 all parameters except for biases and normalization layers. For the smaller CC3M dataset, we use weight decay 0.5. The dropout probability in text encoder varies depending on the dataset size, e.g., no dropout on YFCC15M and probability 0.2 on CC3M, while label smoothing is applied with a smoothing factor of 0.1 regardless of the dataset.
Data augmentation. Both images and text are augmented to produce one weak and two strong augmen- tations. For the sake of clarity, we report the exact configuration adopted in Pseudo-code 2.
Pseudo-code Config for image and text augmentations
x: image c, b, 5, h: constrast, brightness, saturation, bue daf anage_angment((x (x, roda="seak"):
List captione
def text_augnee (y, mode='veak") :
randon multi-caption choice choice(y)
if mode "veak";
transfern
SandonneeizeddOopp(iizeez scale=(0.5, 1))
renove stop-zorda probability ra_stopwords prob-0.8)
elif mode =- 'strong":
transferm Conpose
if node "xeax": return y
Basdoniesiiee02rpp(seeee-e scale- 1)),
RandenColorJitter.
draz one EDA sugnentation transform choica([ synonyn_ replacomntt..c. aadon_awap. randon_deletion
4,0.4, b=0.4, a=0.4, b=0.1, prob=0.8),
Baddandrayscallapprooo...
RandonGaussiannilur (signa 2), prob=0.5),
Bacdonnlorisonntallli (p
})
J. [0.4, 0.4, 0.2]) return transforn(y)
raturn tranaform (x)
Note that strong image augmentations are SimCLR-like (Chen et al., 2020). Stopword removal is set to 0.9
for YFCC15M experiments, and EDA refers to "easy data augmentation" (Wei & Zou, 2019).
We use smaller subset of PMD70M as we were able reproduce part of the dataset.
8

---

Published in Transactions on Machine Learning Research (10/2023)
Table 1: Imagenet zero-shot classification accuracy obtained using CLIP in combinations with other non- contrastive losses. Note that all these runs do not use multi-caption and results for CLIP are obtained using our implementation. " This score differs from the one in Tab. 2, because here multi-caption is not used.
(a) Base vs. improved recipe on YFCC15M
(b) Scaling pre-training data with ResNet-50
Base
Improved
$
Improved
Method
Method
VIT-8/16
Re<Net-50
VIT-S/16
ResNet-50
CCSM
CC12M
YFCC15M
CLIP
32.4
32.6
43.1
43.2*
CT.IP
27.4
44.4
43.2*
BarLIP
34.4
32.8
42.5
42.6
BarLIP
28.0
44.4
42.6
SiamLIP
33.4
32.4
42.7
42.8
SiamLIP
27.9
43.4
42.8
BYOLIP
33.8
32.7
42.9
42.8
BYOLIP
28.2
43.9
42.8
SwALIP
32.9
32.2
41.8
42.6
SwALIP
27.8
43.7
42.6
5.3 Experimental results
In this section, we report on our experiments to disentangle the use of advanced data-augmentation techniques from the introduction of alternative losses for vision-language pre-training. In Sec. 5.3.1, we report on our main experiments that compare CLIP with variants that include additional non-contrastive SSI~inspired losses (BarLIP, SiamLIP, BYOLIP SwALIP), and consider them using the original training recipe and our improved recipe. We then compare CLIP with our improved training recipe with state-of-art results from the literature in Sec. 5.3.2. Finally, we ablate the different components of the improved recipe in Sec. 5.3.3.
5.3.1 Comparison with baselines
We start by evaluating the impact of non-contrastive losses on cross-modal learning. In Tab. la we report the results obtained with the base training recipe and our improved training recipe (marked with #) using VIT-S/16 and ResNet-50 vision backbones. From the base recipe columns, it can be observed that when employing the VIT backbone, BarLIP, SiamLIP, BYOLIP, and AwILIP all outperform the original CLIP with a gap ranging between 0.5% and 2.4%. However, they perform on par or slightly worse (up to -0.5%) when employing the ResNet backbone, presumably due to the stronger inductive bias of convolutions. Then, looking at the improved recipe columns, our recipe provides a great boost to all methods, and especially to original CLIP (+10.7% and 10.5% with VIT-S and ResNet-50, respectively); +8.1 9.3% and +9.8 10.4% for the improved non-contrastive CLIP variants with VIT and ResNet-50, respectively. Note that, the results for SwALIP with our improved recipe utilize the modified version of SwALIP described in the appendix, which requires multiple views to optimize its additional slustering-based loss. Moreover, using the improved training recipe, BarLIP, SiamLII, BYOLIP, and SwALIP all perform similarly (the accuracy is in the ranges 41.8 42.9% and 42.6 42.8% with ViT and ResNet-50, respectively), and worse than CLIP # (43.1% and 43.2%) - additional zero-shot ovaluations are reported the appendix. To validate this result we ran experiments on CC3M and CC12M, which contains different amount of data and are known to be less noisy than YFCC15M, see Tab. 1b. To limit the use of computational resources, we ran this investigation only using ResNet-50. From the results, we can notice comparable performance across all the methods, with CLIP & obtaining slightly worse results with CC3M pre-training (-0.8%), the best results (on-par with BarI.PP) with CC12M, and the best results on YFCC15M.
Taken together, these results clearly demonstrate the importance of a strong training recipe relative to com- bining contrastive and non-contrastive loss functions. This finding is in line with concurrent work investigat- ing a combination of CLIP with MAE (Weers et al., 2023). In the following, we focus on the best performing CLIP compare the statc-of-the-art, and ablate the components of our improvod training recipe.
5.3.2 Comparison with the state-of-the-art
Given the conspicuous boost of CLIP & compared to the original CLIP by simply changing the training recipe, we compare it against other state-of-the-ar methods built on CLIP. We compare to the approaches based on the ResNet-50 architecture, as well as to the approaches based on vision transformers.
9

---

Published in Transactions on Machine Learning Research (10/2023)
Table 2: Comparison of CLIP * with state-of-the-art zero- shot ImageNet classification accuracy. ResNet-50 backbone, unless otherwise specified. Best results in first block in bold. *: Uses larger YFCC100M subset; 1: 50 epochs pre-training.
55
CLIP
50
DeCLIP
CLIP *
45
Method
Pretraining
ToP3 0..1 accuracy
40
CC3M
CC12M
YFCC15M
29M
CLIP (Radford et al., 2021)
20.6
36.5
32.7
41.2
35
ProtoCLIP (Chen et al., 2022)
21.5
32.0
CyCLIP (Goel et al., 2022)
22.1
CLOOB (Fünst al,, 2022)
24.0
35.7
DeCLIP (Li et al., 2021)
27.2
11.0
41.9
49.3
CLIP (owss)
27.4
44.4
43.4
53.3
25
Using pee: Anzined models or additionall supervision
20
BoW (Tejankar et al., 2021)
30.3
UniCL (Yang al., 2022)
40.5
5
10
25
30
PyramidCLIIP (Gao et al., 2022)
43.7
Dataset size (M)
SotCLIP (Gao al., 2023)
24.2
43.2
Using ViT B/38
Figure 3: ImmageNet performance trend when scaling the dataset size when pro- training on CC3M, CC12M, YFCC15M and Open29M respectively.
FILIP (Yao et sl., 2021)
37.8'
MS. CLIP-S (You al. 2022)
36.7*
UniCLIP (Lee al., 2022)
42.87
51.2
Our main comparison is presented in Tab. 2, where we compare the methods based on ResNet-50 or VIT with similar FLOPs e.g. VYT-B /32. Analyzing the results in the first block of rows in Tab. 2, CLIP & not only boosts original CLIP when pre-trained on all datasets, but it also obtains consistently higher accuracy than all the prior works having the same architecture. Broadening the comparison to methods employing pre-trained networks, we observe slightly worse performance, -2.1% and -0.3% on CC3M and YFCC15M. Although negative, this small gap can be considered encouraging as our model could also benefit from initialization and potentially overturn the gap. In second broader comparison with methods that use VIT-B/32, we find the best results for CLIP # YFCC15M (+0.6%), despite the longer pro-training of UniCLIP (50 vs. 32 epochs), and slightly worse results on Open29M (-0.9%). This is explainable as vision transformers have less inductive bias and thus benefit more from a larger dataset size.
In Fig. 3 where we consider the scaling ability of CLIP & compared the best competitor DeCLIP (Li al., 2021) and baseline CLIP, we notice the biggest gap with DeCLIP when pre-training on the largest dataset (+4.0% on Open29M vs. +0.6%, +3.2%, +1.5%, for CC3M, CC12M, and YFCC15M, respectively). This result highlights the scalability of our training recipe, which is fundamental feature given the prominent research direction toward large-scale training. For this reason, we further solidify this scaling investigation by reporting in Tab. 3 the results when pre-training on PMD46M. Here, we compare CLIP # with various baselines of CLIP and FLAVA (Singh et al., 2022). For a fair comparison, we use the VIT B/16 vision backbone as in Singh et al. (2022) and train by sceing a comparable number of image-text pairs. The analysis of the results showcases a significant gap between CLIP & CLIP baseline (-5.8%) and FLAVA (-3.3 5.3%). Moreover, CLIP # matches the performance of the open-clip implementation trained by Cherti et al. (2023) LAION (400M image-text pairs) for 13B samples seen. This is remarkable achievement considering that our model is trained on ~00 less data with ~10x less samples seen. Together with the trend shown in Fig. 3 these results validate a consistent boost provided by our recipe when scaling data.
Orthogonally, we also perform scaling on the model size. Results are reported in Tab. 4 and Fig, 4. We make comparisons with prior work using image encoders based on the ViT-(S,B.L)//1 architectures. As it can be observed from the ImageNet zero-shot accuracy (first column), our improved recipe is again very beneficial compared the original recipe for CLIP: boosting accuracy by 8.8%, 7.4%, and 9.5% when using VYT-(S,B,L) backbones, respectively. Moreover, it outperforms SLIP (Mu el al., 2022), by 3.2%, 2.2%, and 2.4% on (ST.(S,B,L) respectively, when pre-trained for the same amount of epochs (25), and when pre-trained for a significantly less epochs, i.c., 32 vs 100, improving by 2.0%, 3.6%, 2.1% on ViT-(S,B.L) respectively. These results further highlight the benefit of our improved training recipe that includes multiple views with separate projectors, which allow the model to be both better performing and more efficient. in the number of epochs.
10

---

Published in Transactions on Machine Learning Research (10/2023)
Table 3: Linear ovaluation on ImageNet after pro-training on the PMD dataset (unless otherwise specified). Scores of CLIP and FLAVA are taken from Singh et al. (2022), open-clip (Cherti et al., 2023) and the original CLIP pa- per (Radford et al., 2021). Note that, indicatess training on PMD augmented with ImageNet, CCNews and Book-
51
Top-1 accuracy (9)
45
42
Corpus, while marks initialization with pre-trained text and vision eucoders. "I-T" stands for image-lext pairs.
39
Method
Dataset
Batch
I-T seen
Lin eval
CUP (25ep)
SLIP (25ep)
CLIP FLAVA
70M
8K
1.3B
73.0
SLIP (100ep)
>70M
8K
1.3B
73.5*
33
(25ep)
70M
8K
>1.3
74.31
(32ep)
770M
8K
>1.3B
75.5*t
30
50
100
150
200
250
300
CLIP *
46M
4K
1.5B
78.8
Model size (Millions of params)
CLIP
400M (CLIP-WIT)
32K
13B
80.2
Figure 4: ImaaeNet. zero-shot with ViT- [S.B,L]/16 scaling, trained on YFCC15M.
CLIP
400M (LAION)
86-88K
13B
78.7
Table 4: Zero-shot on 23 vision datasets after YFCC15M pre-training for 25 epochs, unless noted otherwise. CLIP and SLIP scores are from the SL.I paper. The average considers also ImageNet, as in Mu et al. (2022).
CIRARA CITRAIO 2011
Stanfed Car
Ocford Attraft
Onford Pets
Calted-- 100
Ondard Flount
RESISC45
Ximi detance
Country111
Parsh Canedren
CLEVE counts
Holede Xlnnes
Ramibod 33T2
Average
SINISI
Fooi101
STLI0
EaniBat
CTSIBB
Mathodl
9.8
43.4
61.0
20.9
17.9
25.0
58.3
47.8
50.8
22.3
191
34.1
64.8
14.7
00.1
40.5
20.5
VI--1/16 VPF-8/2 VST-3/16
SLIP
41.8
0.0
7:2
59.8
51.6
70.0
61.3
05.4
90.5
37.6
36,3
5.8
35-7
20.3
90.9
8774
04.2
91.6
223
38.3
21.9
41.4
39.9
13.0
31.7
50.8
14,9
15.1
51.5
50.4
50..
13.8
SLIP (10000)
2.7
08.4
00.3
36.5
36.6
91-8
18-8
21.2
13.8
38.4
100
40.4
26.6
CLIP (S2ap)
0.4
57.0
62.0
31.3
40.3
6.2
10.5
28.4
$4.4
68.2
50.7
93.8
10.9
23.7
36.6
11.8
30.8
57:2
16.1
59:2
61.9
97.1
8.4
23:22
21.5
15.1
58.5
SLIP
0.8
76.0
6.7
36-1
$1.1
71.0
91.4
20-3
25.0
34.0
55.4
37.7
17.5
51.1
58.6
CLIP
45.0
9.4
61.5
71.3
42.7
6.7
9.9
28.0
311.7
72.8
58.4
85.5
11.6
37/2
10.0
44.2
14.6
51.1
41.1
12.4
52.9
50.1
30.0
(100g))
17.1
70.3
50.4
9.4
34.7
U5.4
27.5
11.7
35.2
52.1
00.9
20.0
CLIP (30pp)
47.0
9.5
133.0
07:3
37..5
45.7
8.22
11:2
-00.3
35.4
75.4
622.8
16.1
223.2
36.00
8.1
35.45
15.66
31.6
42.7
13.5
54.3
50.1
40.0
10.3
71.9
30.6
9412
15.5
54.9
12.1
10.3
SLIP
1822
87.8
58.4
29.5
76.6
30.4
56.6
27.7
96.5
25.8
54.4
42.6
16.1
50.1
41.1
CLPP
68.6
16.5
69.1
83.5
19.9
46.2
9.99
11.8
30.4
32.9
78.5
65.8
06.7
100.7
41.3
8.4
35.0
18.2
51.5
45.2
15.8
50.0
50.3
42.2
(100000)
14.0
87.5
20.5
9.0
0.5
20.0
80.0
60.3
06.2
46.0
30.7
30.0
09.5
43.1
CLIP (30pp)
50.0
19.3
70.3
64.7
52.9
46.9
11.8
11.2
32.1
41.7
77.5
08.3
87.00
08.3
64.1
10.3
37.0
17.9
56.2
11.0
13.8
50.0
50.3
43.5
Next, we consider the generalization of CLIP # to different types of vision benchmarks. To do so, we follow the setting proposed by Radford et al. (2021), testing the zero-shot transfer of CLIP representations on 22 vision benchmarks (plus ImageNet) performance (see last column), in all the settings considered, i.c., (iT-(S,B,L) with 25 or more pre-training epochs. In particular, focusing on the 25 epochs pre-training, CLIP # provides a boost in most of the benchnarks; it improves over baseline CLIP in 18/23, 20/23, and 18/23 benchmarks with VIT-(S,B,L.. respectively. Moreover, compared with SLIP, regardless of the pre-training duration, CLIP & obtains the best performance in 13 to 16 of the 23 benchmarks. As a final note, we notice that the types of benchmarks positively affected by our recipe remain almost the same across different. ViT sizes and number of epochs. This might be explained by the induced invariances of the strong data augmentations used in our recipe.
reported in Tab. 4. We observe improvement of the average
5.3.3 Ablations
We quantify the impact of the different components of our improved training recipe by isolating and ablating them. We first ablate the regularization lechniques, and then the projector design. Finally, we analyze the computational cost of our recipe. For all the ablations, we use CLIP * pre-trained on YFCC15M with ResNet-50 as the vision backbone. For these experiments, we use the zero-shot ImageNet top-1 classification accuracy as a metric. Also, note that not all the ablations were run on the final recipe: some of them have been performed earlier in our exploration, and thus provide results different from the one reported in Sec. 5.3.2.
11

---

Published in Transactions on Machine Learning Research (10/2023)
(a) Text augmentation
(b) Vision augmentation
(c) Projectors design
EDA
RMSW
Multi-cap
Top-1
SimCLR aug.
Multi-view
BYOL aug.
Top-1
Text-pro)
Vislon-proj
Top-1
42.8
32.7
Lin Traharedd
Lin Sin
40.9
43.0
35.2
MLPI-shared
MLPY-ebared
39.6
/
43.1
/
43.4
Lin"
Lin"
41.0
/
43.4
/
43.2
MLPF
Lin"
41.0
Lin"
MLPY
42.6
MLP"
MLP'
43.1
(2, 224)
43
(2, 192)
(2. 160)
43
65
Top accuracy (%)
Top-1 acouracy (9))
Tep-1 1 accuracy (%)
42
(2, 126)
(1. 224
41
42
50
Logit
40
Mem footprint
29GB
41
3.5
2EGB
CCC
39
23GB
CC12M
21GB
(2, 96)
CC3M
19GB
40 0.00
0.02
0.04
0.06
0.08
0.10
20
0.00
0.05
0.10
0.15
0.20
1.00
1.05
1.10
1.35
1.20
1.25
Label smoothing
Dropout probability
Speedup
(d) Label smoothing
(e) Dropout
(f) Efficiency
Figure 5: Ablations. (b) EDA: Easy Data Augmentation (Wei & Zou, 2019); RMSW: remove stop-words (Te-
jankar et al., 2021); Multi-cap: random multi-caption sampling; (e) the accuracy is relative to having dropout set to 0; (f) the reported values were computed using eight. Nvidia V100-SMX2 32GB GPUs and our recipe with ResNet-50 backbone (see Sec. 5.2). The annotation of each point indicates (# of views, view resolution).
Regularization. In Fig. 5a and 5b we break down the data augmentation recipe into its main components. For the text augmentation recipe (see Fig. 5a), we evaluate the impact of EDA (Wei & Zou, 2019), removing stop-words (RMSW) (Tejankar et al., 2021), and random multi-caption sampling (see Sec. 5.2). We notice that each of the three techniques provides a small boost, summing up in total to +0.6% to CLIP *. In particular, while RMSW only provides +0.1% it is helpful to reduce paiss//ninformative image- captions from the training. In general, we may explain the relatively small boost given by text augmentations as our recipe is more focused on the vision part due to the introduction of multiple views. Indeed, in Fig. 5b we showcase the improvement of using multiple views combined with SimCLR-style (Chen et al., 2020) image transformations that results in +10.7% w.r.t. the original CLIP. On the contrary, the use of stronger and asymmetric BYOL-like (Grill et al., 2020) image transformations are not beneficial in the multi-modal CLIP pre-training. We conjecture that the implicit. regularization provided by these augmentations is too strong in the presence of the noisy supervisory signal provided by image-text pairs.
In addition to implicit data augmentation regularization, we also study the effect of the explicit regularization techniques present. in our improved recipe in Fig. 5d and 5e. Specifically, Fig. 5d depicts the trend of the accuracy (blue line) when incrcasing label smoothing in the contrastive computation between strong sugmentations. From the plot, it is easy to observe that higher smoothing factors generate better results, +2.8% by increasing the smoothing from 0.0 to 0.1. To explain this significant boost, we report in the same plot the trend of the logit scale (orange curve), which corresponds to the inverse of the learned temperature, 1/7. The logit scale is negatively correlated with the label smoothing factor, suggesting that label smoothing improves accuracy by mitigating overfitting as indicated by reduced logit scales. This is also corroborated by concurrent work (Gao et al., 2023). In Fig, 5e, we analyze the effect of the dropout probability in the text encoder when pre-training on three different datasets. We find that YFCC15M, the larger and more noisy dataset among the three, applying dropout is dimental. On the contrary, the datasets become smaller and less noisy, dropout provides slight accuracy boosts.
12

---

Published in Transactions on Machine Learning Research (10/2023)
Projector design. A crucial modification that we propose in our improved recipe is the use of two sepa- rated non-linear projectors to project strong/local augmentations//ien /views into the CLIP cross-modal embed- ding. We ablate the possible architectural choices for the projector in Fig. 5c. In the first two rows, the same projectors are used for the weakly and strongly augmented text, and similarly for the image projectors. We observe that with such shared projectors having a linear projection (Lin) is beneficial compared to non-linear (MLP). However, in the following rows, we show that if we employ separate projectors for the weakly- and the strongly -agggeeee text and images, using non-linear projections (last row, MLP" and MLP") produces a boost of +2.2% from the baseline case (first row, shared linear projection). This finding is perhaps not surprising as strongly augmented text-image pairs are more challenging to align, providing enough regularization such that the MLPs can be trained without overfitting.
Efficiency. Our improved training recipe increases the computational costs due to the simultaneous use of multiple image and text diews/augmentaien how the accuracy and the training time are impacted by the resolution of the multiple image views. As can be noted, we can reduce the training time by using lower resolution image views. However, we observe that decreasing the resolution of the image views impacts negatively the performance. In particular, substantial downsampling i.e., < 128, causes a drastic drop in the performance, while less aggressive downsampling, ie., 1922 and 160?, still provides results competitive with the state of the art, but at a training cost. reduced by 10% to 20%.
combined with more complex projectors. In Fig. 5f, we show
6 Related work
Contrastive vision-language pro-training. Seminal works like CLIP (Radford et al., 2021) and ALIGN (Jia al., 2021) have proven how contrastive VLP at scale provides rich and general representa- tions for multitude of downstream tasks. However, these methods apply the contrastive loss only between the entire image and its caption, ignoring possible local alignments, and do not use distorted views of the input data limiting the robustness of the learned representations. To this end, FILIP (Yao et al., 2021) and PyramidCLIP (Gao et. al., 2022), have tried to refine the contrastive loss to consider image and text parts beatch-tokend saanns//tttp /tags pairs), and SoftCLIP (Gao et al., 2023) use image-text parts alignment combined with a softening loss computed over the negatives pairs. Instead, other works tackled a better robustness of the learned representations: CLOOB (Fürst al., 2022) using Hopfield networks to regulate the covariance of the learned representations, MS-CTIP (You et al., 2022) and FIBER (Dou et al., 2022), tried to overcome the use of dual modality encoders in CLIP to obtain better fused multimodal features, while FLIP (Li et al., 2022b) and MaskCLIP (Dong et al., 2022) used image masking. More related to CLIP H some works improved representation robustness using multiple views: SLIP (Mu et al., 2022) used multiple views of images, but unlike us, they were used to optimize an auxiliary unimodal SSL contrastive loss; UniCLIP (Lec et al., 2022) and ViewCo (Ren el al., 2023) instead, used multiple image views for both the unimodal and multimodal contrastive losses. However, this is different from CLIP & since we do not use unimodal losses while we also use multiple views of the text to generate aligned weakly and strongly augmented image-text pairs. Finally, since image-text pairs are often noisy in web-crawled datasets, UniCL (Yang et al., 2022) proposed a solution that integrates imaee-label pairs from supervised datasets, while CoCa (Yu et al., 2022), OTTER (Wu et al., 2021), BLIP (Li et al., 2022a), and BoW-CLIP (Tejankar et al., 2021) proposed to denoise text captions by gencrating them or substituting them with bag of words (BoW). Inspired by BoW-CLIP, in CLIP # we also remove stop-words from the image captions.
SSL-inspired non-contrastive losses in VLP. All the aforementioned contrastive VLP approaches are highly sensitive to the quality and diversity of negative samples present in the batch. To overcome this issue, several recent works have explored the use non-contrastive losses inspired by recent SSL advancements. However, as shown by the non-contrastive basclines in our experiments, none of the methods in the literature obtained better results without keeping the standard CLIP contrastive loss. Differently to our baselines, some existing methods only adopt non-contrastive losses for the unimodal encoders: DeCLIP (Li et al., 2021) use a consistency loss for the image encoder, while VoLTA (Pramanick et al., 2022) adopt the Barlow Twins loss (Zbontar et al., 2021) separately on both the vision and text encoders. More similar to our basclines some works integrates the CLIP contrastive loss with multimodal non-contrastive loss: ProtoCTIIP (Chen
13

---

Published in Transactions on Machine Learning Research (10/2023)
et al., 2022) and xCLIP (Zhou et al., 2022) adopt clustering-based loss, while CyCLIP (Goel et al., 2022) combines both unimodal and multimodal consistency losses for vision and text. However, these methods employ custom non-constrastive losses that do not reflect any existing SSL approaches such as our baselines. Finally, FLAVA (Singh et al., 2022) and ViCHA (Shukor et al., 2022) use image and text masking combinod with multimodal masking and/or image-lext matching, but the masking approach is out of the soope of this work as usually requires fine-tuning, as shown by He al. (2022), while we focus on zero-shot transfer.
7 Conclusion
In conclusion, this paper investigated the effectiveness of combining contrastive learning with recent ad- vances in self-supervised and supervised learning. Our study includes several baselines using successful non-contrastive loss functions from visual self-supervised learning, and compares their performance with CLIP, seminal work in this area. Our findings indicate that the SSL-inspired baselines outperform ba- sic implementation of CLIP, but the advantage disappears when an improved training recipe is employed. Noticeably, we show that a simple CLIP baseline can be substantially improved by applying this improved training recipe, composed of image and text augmentations, label smoothing, dropout, and deeper projector networks. Our improved training recipe for CLIP achieves state-of-the performance on four datasets and outperforms related methods by significant margins.
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic doduplication. arXiv preprint fii:233300054400 2023.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual fcatures by non-parametrically predicting view as- siguments with support samples. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 8443-88 8452, 2021.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint vrXiu:2205.11000 11508, 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-nnariance-ovvaaa regularization for self-supervised learning. arXiv preprint arXiv:2105.04 04906, 2021.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self- supervised representation knows about. arXiv preprint arXi::2112.09164, 2021.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9999 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-96600 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail tail visual concepts. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3558-3568 2021.
Delong Chen, Zhao Wu, Fan Liu, Zaiquan Yang, Yixiang Huang, Yiping Bao, and Erjin Zhou. Prototypical contrastive language image pretraining. Xiviv preprint arXiv:2206. 10996, 2022.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMT.R, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 2829, 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Jía Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 255. leee, 2009.
Xiaoyi Dong, Yinglin Zheng, Jianmin Bao, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self-distillation advances contrastive language-image pretraining. arXiv preprint 220: 1 12262, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa, Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXim:2010. 1 11929, 2020.
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, al. Coarse-to-fine vision-language pre-training with fusion in the backbone. arXiv preprint ar arXie:2206.076. 07644,, 2022.
Aleksandr Ermoolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning, pp. 3015 3024. PMLR, 2021.
Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9284 9292, 2021.
Enrico Fini, Pietro Astolfi, Karteek Alahari, Xavier Alameda-Pinedd Julien Mairal, Moin Nabi, and Elisa Ricci. Semi-supervised learning made simple with self-supervised clustering. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognation, 2023.
Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner, Viet T Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp, Günter Klambauer, Angela Bitto, et al. Cloob: Modern hopfield networks with infoloob outperform clip. Advances in neural information processing systems, 35:20450-2000 204688, 2022.
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hierarchical feature alignment for vision- language model pretraining. Advances in Neural Information Processing Systems, 35:35959-35970 2022.
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wa, Wei Liu, Jie Yang, Ke Li, and Xing Sun. Softelip: Softer cross-modal alignment makes clip stronger. arXiv preprint arXiv:2303. 1766, 2023.
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35: 6704-6719, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 -21284, 2020.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer nision and pattern vecognilion, pp. 770-772 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hamnaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp.
PMLR, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Youglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661 18673, 2020.
Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for contrastive language-iman image pre-training. arXio preprint arXiv:2209. 19490, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre-training for unified laiiugg understanding and generation. In International Conference on Machine Learning, pp. 22888-12900 PMLR, 2022a.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive anguage-image pre-training paradigm. arXiv preprint arXim:2110.06 10.05208, 2021.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking, arXiv preprint orXiw:2212.0099/ 2022b.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Xi preprint 1rXi::111..01101 2017.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pro-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Parl XXVI, pp. 529 544. Springer, 2022.
Rafael Müller, Simon Kornblith, and GeofTrey E Hinton. When does label smoothing help? Adoances in neural information processing systems, 32, 2019.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050 2021.
Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik Shah, Yann LeCun, and Rama Chel- lappa. Volta: Vision-language transformer with weakly-supervised local-fcature alignment. arXiv preprint arXiv:2210.00 04155, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. PMLR, 2021.
16

---

Published in Transactions on Machine Learning Research (10/2023)
Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiao- dan Liang Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. In The Eleventh International Conference on Learning Representations, 2023.
Mert Bulent Sariyildiz, Yannis Kalantidis, Karteek Alahari, and Diane Larlus. No reason for supervision: Improved generalization in supervised models. In International Conference on Learning Representations, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: cleaned, hyper- nymod, image alt-text datasct for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Compulational Línguistics (Volume 1: Long Papers), pp. 2556 2565, 2018.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Connor Shorten, Taghi M Khoshgoftaar, and Borko Furht. Text data augmentation for deep learning, Journal of big Data, 8:1-34, 2021.
Mustafa Shukor, Guillaume Couairon, and Matthieu Cord. Efficient vision- language pretraining with visual concepts and hierarchical alignment. arXiv preprint Xiv:22200 13628, 2022.
Amanpreet Singh, Ronghang Hu, Vedamuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638-15650, 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
C. Szegedy, V. Vanhoucke, S. Toffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Ajinkya Tejankar, Bichen Wu, Saining Xie, Madian Khabsa, Hamed Pirsiavash, and Hamed Firooz. A fistful of words: Learning transferable visual models from bag-of-words supervision. ar preprint 2rXiv:2112. 13884, 2021.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Ylee100m: The new data in multimedia research. Communicntions of the ACM, 59 (2):64-73, 2016.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding sule-supe learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10278. PMLR, 2021.
Floris Weers, Vaishaal Shankar, Angelos Katharopoulos, Yinfei Yang, and Tom Gunter. Self supervision does not help natural language supervision at scale. arXie preprint arXiv:2301.07886 2023.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classi-
fication tasks. arXie preprinl arXiv: 11196, 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arxiv preprint Xiv:22110. 00476, 2021.
Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph E Gonzalez. Data efficient language- supervised zero-shot recognition with optimal transport distillation. arXiv preprint orXiv:2112.09441 09445, 2021.
17

---

Published in Transactions on Machine Learning Research (10/2023)
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 19163 19173, 2022.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive languago-image pro-training. arXiv preprint arXiv:211100778. 2021.
Haoxuan You, Luowei Zhou, Bin Xiao, Nocl Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-image pre-training. In Compuler Vision-ECCV 2022: 17th European Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Part XXVII, pp. 69 87. Springer, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Young Mojtaba Scycdhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-lext foundation models. aiviv preprint wrXiu:2205.01111 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310-12323 12320. PMLR, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of repre- sentation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.0458: 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, Sigmoid loss for language image pre-training, XrViv preprint arXio:2303. 15348, 2023.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood con- trastive learning for novel class discovery. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 11867-10877 2021.
Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastive learning meets language- image pro-training. arXiv preprint arxiv:2210. 09304, 2022.
A Modified SWALIP
To mitigate the instability issues of SwALIP (sec Sec. 4), we develop a slightly modified version that does not exhibit learned protolypes, rather, it uses representations from the other modality as protolypes, and clusters samples in the current modality around those. In practice, in order to implement this, need multiple views of cach image-text pair as in our improved recipe, where we have two strongly-angmented views. Considering a batch of samples containing such two sets of correlated views projected into latent space, 2 = ((= 2 ) and 2 = ((zA, = ) the new formulation reads as follows:
C = LAE CA AM, A = -ssk(sim (24, (B)) + - AII, pB= = 2
(7)
N C = Σ>
exp (sim (21²) .p²) /T) ΣV Ek=1 exp (sim (2 p /T)
x log
(8)
where a: E AA is the vector of cluster assignments for the correlated view, Σ of 21 in one modality, e.g., image, and each prototype D pB corresponds one view of the samples, 2 in the other modality, e.g., text. Assignment vectors are obtained by applying Sinkhorn-Knopp (sk) on the multimodal similarity matrix computed against the prototypes PB, The loss is first computed using the swapped assignments typical of SwAV, i.e., B AAAAAA PB) and AZA,(A AA,,A, pB), and then in both directions A + B and B + A, so that it becomes symmetric. Also, since the view of the samples in the "other" modality used
18

---

Published in Transactions on Machine Learning Research (10/2023)
prototypes is arbitrary, i.e., pd " and P - are both valid, in implementation average the loss considering both cases. Finally, to stabilize training and avoid collapsed solutions we also average the output of the Sinkhorn-Knopp algorithm with the identity matrix, I, to strengthen relationships on the diagonal (ground truth image-text pairs).
B Additional results
Table 5: Zero-shot transfer on ImageNet and several vision datasets taken from Radford et al. (2021). Pre-
training on YFCC15M for epochs with ResNet-50 as vision backbone.
ImuugeNNe
CHARC30
CIPAR-I0
CURR00 2010
Stanarr Cars
Abeemit
Onfond Pata
Called-- 101
Oxford Flonees
RESISCAS
Körd distaoo
Comntry211
Porch Camplpon
CLEVR counts
Haudal Menoe
Renieesi 9912
MNNSS
CTSRB
DTD
Method
CLIP x
43.1
9.8
50.3
47.0
21.5
37.8
6.2
5-
25.7
34.1
66.6
69.0
89.7
18.5
21.0
83
37.0
11.8
51.0
37.0
12.9
52.7
50.1
35.1
SoaLiP x
12.5
10.8
12.6
62.6
21.8
37.8
6.6
96
27g
$4.6
67.1
68.8
89.0
14.5
19.4
11.8
41.1
11.0
50.2
37.0
15.3
62.9
50.1
85.5
BaLiP
42.5
22.3
30.1
29.8
9.1
392
12.4
53.9
BYOLIP *
4288
10.8
30.8
54.1
22.3
30.1
6.9
27.2
32.4
68.2
5933
90.8
10.7
23.1
788
38.6
11.2
52.5
36.6
129
53.2
30.1
35.6
Generalization of non-contrastive baselines In the main paper compared our improved baselines (see Sec. 4) with CLIP s in terms of zero-shot image classification accuracy ImageNet. However, might ask whether the use of those non-contrastive losses is beneficial in terms of generalization. In Tab. 5, we provide an answer to this question, expanding the evaluation to a larger set of 22 vision datasets. We observe that the best results on the downstream datasets are rather spread among the different improved basclines. is hard to find a model overcoming others consistently, despite BYOLIP presents a slightly better overall average. general, this result confirm our finding that it is worth complicate the our well-tuned recipe, CLIP #.
model by using multiple objectives, but is enough
Sensitivity analysis wrt. and 3 Here, report ablation of the and hyper-parameters regulating the weights of the contrastive and non-contrastive terms, respectively (see Eq, 6). To reduce the com- putational burden, we carry out the analysis only for BarLIP H, pre- trained with ResNet-50 on YFCC15M for 32 epochs. We ablate values of while kcoping a fixed at and vice versa. Results are reported in Table 6, where we observe that increasing the value of 8 with fixed at slightly harms the performance of up to 0.8%, while doing the op- posite, i.c., increasing a with 3 fixed at 1, leads to substantial drop of accuracy, up to 3.7%. Notably, using only the non-contrastive loss (a 0, B 1) is problematic for zero-shot evaluation because the CLIP projector is not optimized and is hard to perform zero-shot classifi- cation on the non-contrastive projector. These results further confirm the inefTectiveness of adopting non-contrastive losses together with our improved recipe.
Table 6: Sensitivity to loss weights. ImageNet zero-shot results ob- tained with BarLIP
8
YFCC15M
43.2
1
0.25
42.8
1
0.50
42.9
1
0.75
42.4
0
0.1
0.25
38.9
0.50
41.6
0.75
42.5
42.6
C Pseudo-code for multi-view
In Pscudo-Code 3, show the actual implementation adopted in CLIP H when multiple (num_aud augs) strong augmentations adopted. It is important to note that when compute the contrastive among strongly- augmented image-text pairs, each strong augmentation is contrasted with all the strong augmentations of the other modality. For instance, given two strong augmentations (num_augs augs = 2), each image augmentation is contrasted with both the strong text augmentations, and vice versa.
19

---

Published in Transactions on Machine Learning Research (10/2023)
Pseudo-code 3 Detailed implementation of CLIP
bi inage and its corresponding text f.t, f.v: vision Lin, nip: Linear and MLP (hidden__d1p din) with (enbed_din proj_din)
encoders (in_din, _dim
tau_e, tau_s: learnable tenperature for weak and strong sugnentations sin_target: identity natrix
proj_v_R, proj.t. t.x lin(f (in()_v(x), bias=Xone), lin(f_t(x), bias=Mone) proj_v_a, proj_t_a normalize (ormallee(ll((t_vtae) (nip(f_v(x))), normalte@cal((..((e.) def tres,ing.steppe, b):
augnentations
a_v, b_M au, 'weak"), sb, 'weak')
strang augnentations
a_s, b_a [], D for to range((u__aus)) aug 'strang') b_s aug_t(b, 'strong")
# projecting into shared proj-t_*(b_u)
z_a,x, 2_b_0 proj_ z_a_s, z_b_a ( (+b__),
sugnentation.
tau_y 2_a_x 2.a_x22_0 z.b.r.t((
sim_b_to_a_0 tau_e a_b_ e a_a_x.t()
loes_a_y ezozeyy (eiz_a_to_b_w, target)
loss_b_v cross_ b_in_ sin_target)
strong sugaentation
sin_a_to_b_s
for rango(Bum_augg))
sin_a_to_b_a [tau_s a. +0 for in a_a_a]
sin_b_to_a_0 +- [tau_e x 2.a_e(i).t( :( for in 2_b_s]
loss_a_s sum([cross_entroo
sin_target, label_snoothing) for sin_a_to_b_el) len
loss_b_a aun entrrops_nttrpyCC (x, aim sin_target, label amoothing) for sin_b_to_a_aj] (ati__b_to____)
loss_a (loss_a_z loss_a_s nun_augs) (1 nus_augs) loss_ (loss_b_2
aun_augs)
nun_auga)
lose (los8_a loss_b)
return
20


Reviews:
[{"review_id": "unK5cR8zgy", "reviewer": "Reviewer_Z9sJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This work is a welcome deviation from traditional leaderboard chase. It takes a step back and nicely analyzes the possibility of SSL approaches to top up the normal contrastive training of CLIP. The twist comes late and it tells that resorting to well-established training recipes reaps more benefit compared to SSL losses. The finding can be useful for typical leaderboard chase as well.\n2. The paper does a good job in summarizing and concisely describing the working principles of relevant literature.\n3. The experiments are well thought, well designed and well executed. 3 different pretraining datasets and 3 different vision backbones are tested with. The pseudo-codes add to the ease of understanding. Different ablations especially the gradual addition of augmentations and label smoothing variation are nice additions to the main experiments.\n\nWeaknesses:\n1. Although the experimental plans are pretty good, a sensitivity analysis of the performance (on imagenet zero-shot accuracy) on $\\alpha$ and $\\beta$ hyperparameters (ref. eqn. (6)) would have been good to have.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Building on the success of contrastive vision language models (primarily CLIP) and self-supervised pretraining strategies, the paper exercises the option of combining the two for better vision language training. For this purpose, the authors propose to extend normal CLIP training with the addition of four popular self-supervised learning (SSL) non-contrastive losses. The extension showed improved vision-language representation learning and this is manifested in the improved model’s capability of zero-shot transfer to benchmark datasets. The authors, though, exercised additional option of sticking to contrastive loss only (i.e., normal CLIP style training) but using stronger training recipes well-known in other works but not employed in CLIP. The second framework yielded even better performance at the cost of slightly increased computation. The experimental design was well thought and the experiments were well-executed. The relevant literature was well studied and summarized with pinpointed differences and similarities of the proposed work with them. In summary, with the simplicity in the training pipeline, the presentation and the easy-flowing experiments, I think the work can make an useful contribution to the community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. One sensitivity analysis related to weaknesses (point 1) can be added.\n2. A few typos need to be corrected.\n - Page 2, second line: The acronym SSL is not defined earlier. Though, it can be interpreted, still it is better to define the term. I have seen Semi-supervised as well as Self-supervised learning both getting the same acronym SSL.\n - Page 5, last line: ‘when computing the contrastive between’ -> ‘when computing the contrastive loss between’", "broader_impact_concerns": "I can’t think of any immediate concerns. However, it is good to have a discussion on it.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZdWF64N4Nj", "reviewer": "Reviewer_WDqT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The training recipe proposed in this paper is very instructive for practice and saves a lot of hyper-parameter search work.\n2. Many techniques, including image augmentation, text augmentation, label smooth, projector design, have been empirically proven to improve representation quality.\n3. Extensive ablation experiments analyze the effect of each component.\n\nWeaknesses:\n1. The significance of this paper is worthy of recognition, but many experimental phenomena have not been analyzed in detail. For instance, why does applying self-supervised representation learning techniques not lead to further performance gains when using advanced recipe? I'm guessing it might have something to do with the use of image and text augmentation.\n2. Considering that the title of the transaction includes \"Machine Learning Research\", a qualified paper should not only have comprehensive and impressive experiments, but also include explanations of experimental phenomena and some theoretical analysis. I believe doing so will greatly improve the quality of the article and make it more meaningful to the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a large number of experiments to analyze the effect of combining CLIP with various self-supervised representation learning techniques. Experiments show that combining different self-supervised representation learning techniques can achieve performance gains on downstream tasks compared to the original implementation of CLIP. In addition, this paper also proposes a advanced training recipe, which has achieved significant improvements compared to the original implementation. However, when combined with self-supervised techniques, there is no obvious improvement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. If the data scale is further expanded, is the advanced recipe better than the original implementation?\n2. please present more analysis of experimental results.", "broader_impact_concerns": "Nan", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HYOeUPz9jB", "reviewer": "Reviewer_sXof", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- Proposes a simple yet effective improved training recipe which involves improved image and text augmentations, label smoothing, and stronger projector models. The strong and weak losses are averaged.\n- Demonstrates the effectiveness of the training strategy over most complicated training objectives. The proposed method outperforms CLIP models optimized on more complex loss functions (such as the consistency-based objectives, the Barlow Twins loss, or clustering criteria).\n- The proposed method has a convincing scaling curve (Figure 3), which suggests that the gains will still hold when models are trained on even more data (i.e., the proposed method will still be helpful even when the full CLIP 400M dataset is available).\n- The paper conducts careful ablations on the objectives and projections to justify design choices.\n\n\n## Weaknesses:\n- There are limited experiments on scaling the CLIP model. Table 3 shows some favorable results with ViT-B against ViT-S. It would be beneficial to conduct even more scaling experiments, to show that the proposed method does indeed benefit models even at larger parameter sizes. If possible, having ViT-L, ResNet-101 and ResNet-152 results would make the findings substantially stronger. It would also be valuable to scale the text encoder similarly.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors critically analyze improved self-supervised learning procedures that claim to improve over CLIP contrastive learning. They show that many of these methods do not improve over the vanilla CLIP model when better data augmentations (and other improvements) are applied. The result is a simpler training objective that outperforms more complicated methods. They conduct extensive analysis to justify their design choices, such as augmentation strategies and projection models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Strengthen the work:** As mentioned above, I think that including further scaling experiments (along the model parameters dimension) would be very valuable to the paper.\n\nI think the paper in general is very polished and well written, and do not have any other requested changes.", "broader_impact_concerns": "I have no major concerns about the ethical implications of the work: it is research on self-supervised learning, primarily applied downstream to image classification. It is an analysis paper which does not introduce significant risk.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The study re-evaluates the leaderboard race, emphasizing the superiority of traditional training over SSL for enhancing CLIP. With a clear summary of related works and robust experiments across multiple datasets, it introduces an efficient training recipe combining image/text augmentations and projector designs. This method surpasses complex CLIP models and shows potential for scalability. Design choices are solidly backed by detailed ablation studies.

Initially, the reviewers raised concerns about in-depth hyperparameter analysis and ablations of self-supervised learning. One reviewer is also expecting a deeper theoretical insight and broader scaling experiments, especially with larger models. After the revision, all the reviewers were satisfied and voted to accept.

A Feature Certification is recommended for this work. Indeed, all the reviewers acknowledged that this paper is a non-trivial deviation from the leaderboard chase. The idea is simple yet effective. Its design philosophy may shed light on future vision-language pretraining research.
Decision: Accept (poster)