CAN: A simple, efficient and scalable contrastive masked autoencoder framework for learning visual representations
Authors: Shlok Kumar Mishra
Joshua David Robinson
Huiwen Chang
David Jacobs
Weicheng Kuo
Aaron Sarna
Aaron Maschinot
Dilip Krishnan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self supervised learning
contrastive learning
masked autoencoders

Abstract: We introduce CAN, a simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is a minimal and conceptually clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N) the noise prediction approach used in diffusion models. The learning mechanisms are \emph{complementary} to one another: contrastive learning shapes the embedding space across a batch of image samples; masked autoencoders focus on reconstruction of the low-frequency spatial correlations in a single image sample; and noise prediction encourages the reconstruction of the high-frequency components of an image. The combined approach results in a robust, scalable and simple-to-implement algorithm. The training process is symmetric, with $50\%$ of patches in \emph{both views} being masked at random, yielding a considerable efficiency improvement over prior contrastive learning methods. Extensive empirical studies on linear evaluation, finetuning, transfer learning, and robustness demonstrate that our approach achieves strong downstream performance. For instance, when pre-training ViT-B encoders on the curated ImageNet dataset, CAN achieves $74.8\%$ top-1 linear probing accuracy, an absolute improvement of $6.8\%$ over MAE and $1.3\%$ over SimCLR with the same architecture and data augmentations. CAN is especially useful for pre-training on larger uncurated datasets such as JFT-300M: the finetuned performance on ImageNet of our ViT-L model is $85.9\%$, compared to $85.0\%$ for SimCLR, and $85.4\%$ for MAE. For linear probe on ImageNet, CAN achieves $75.4\%$ compared to $71.8\%$ for SimCLR and $64.1\%$ for MAE. The overall FLOPs load is $41\%$ \emph{lower} than SimCLR\footnote{Our code will be released at \url{www.xxx.yyy}.}. 

Under review as a conference paper at ICLR 2023
CAN: A SIMPLE, EFFICIENT AND SCALABLE CON- TRASTIVE MASKED AUTOENCODER FRAMEWORK FOR LEARNING VISUAL REPRESENTATIONS
Anonymous authors Paper under double-bling review
ABSTRACT
We introduce CAN, simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is minimal and conceptually clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N) the noise prediction approach used in diffusion models. The learning mechanisms are complementary to one another: contrastive learning shapes the embedding space across batch of image samples; masked autoencoders focus on reconstruction of the low-frequency spatial correlations in a single image sample; and noise predic- tion encourages the reconstruction of the high- figh-firyyeecy components of an image. The combined approach results robust, scalable and simpletto-ipplement al- gorithm. The training process is symmetric, with 50% of patches in both views be- ing masked at random, yielding a considerable efficiency improvement over prior contrastive learning methods. Extensive empirical studies demonstrate that CAN achieves strong downstream performance under both linear and finetuning evalu- ations on transfer learning and robustness tasks. For instance, when pre-training ViT-B models on the curated ImageNet dataset, CAN achieves 74.8% top-1 linear probing accuracy, an absolute improvement of 6.8% over MAE and 1.3% over SimCLR with the same architecture and data augmentations. CAN is especially useful for pre-training on larger uncurated datasets such as JFT-300M: for linear probe on ImageNet, CAN achieves 75.4% compared to 73.4% for SimCLR and 64.1% for MAE, Finetuning our ViT-L model on ImageNet attains 86.1% top-1, compared to 85.5% for SimCLR, and 85.4% for MAE. The overall FLOPs load of SimCLR is 70% higher than CAN for ViT-L models!.
INTRODUCTION
Self-supervised learning promises continued advances in the state of the art by enabling the use of increasingly large models and datasets. However, interest in larger datasets has precipitated an in- creased reliance on web-scraped data collection processes, which result in heterogeneous and "uneu- rated" datasets (Yu et al., 2022; Radford et al., 2021; Jia et al., 2021). Extreme image heterogeneity has made scaling vision models to uncurated datasets a non-trivial challenge (Tian et al., 202 I; Cole al., 2022). There are two families of self-supervised methods for images which have both proven highly effective on curated datasets (c.g., ImageNet), and are therefore natural candidates for scal- ing to large, uncurated data. First, masked image models such the masked autoencoder (MAE) (He et al., 2022) are a nascent set of methods based on a mask-anddreconstrucc training mechanism. This classical idea (Ballard, 1987) is enjoying a rejuvenation thanks to favourable efficiency when combined with the vision transformer architecture (Dosovitskiy et al., 2021b). Second, contrastive learning (van den Oord et al., 2018; Chen et al., 2020b; He et al., 2020) trains an encoder dis- tinguish between pairs of positive samples generated with data augmentations and negative pairs sampled at random. Both approaches have proven to be very powerful supervupe methods.
Contrastive learning and masked autoencoders (MAE) employ very different learning mechanisms: the former train the encoder to be invariant to semanties- peantrii--reserving data variations, while MAES learn spatial statistical correlations. Furthermore, MAE methods treat each sample independently in
Cooee will he released soon.

---

Under review as a conference paper at ICLR 2023
Computation cost
Linear probe
Finetune
lours)
01.00
gFLOPs
68
MA
CAN (ours)
6654
SinCLR
VIT
VIUU
0500
CAN (ours)
200
Model size
423
590
Pretraining JFT-300M
4x0
3K3
Pretraining JFT-300M
2003
Parameters (M)
(IN-1K equivalent epochs)
(IN-1K equivalent epochs)
Figure 1: Left: CAN scales better than SimCLR since it uses masked inputs. Middle and right: CAN outperforms SimCLR and MAE on ImageNet linear probe and finetune evaluations for ViT-L models when pre-training on uncurated data such as JFT-300M.
the loss function, while contrastive methods explicitly look the relationship between all samples in the batch, by either reducing or increasing embedding distance. Given this, we hypothesize that these two approaches are complementary, extracting different discriminative features for given input. If this hypothesis holds, then we expect to see improved performance on various downstream tasks based on the extracted features. This motivates our exploration of combined method.
Further, inspired by advances in diffusion models (Ho et al., 2020; Song et al., 2021), we introduce a third loss based on noise prediction during the masked autoencoder reconstruction. We add Gaussian noise to unmasked input patches, and train the model to predict the noise added to each patch. Denoising encourages the encoder to extract autoencoder reconstructions tend to focus on low-frequency information (Hou et al., 2017). This additional loss has two purposes: it improves downstream performance; and it addresses a source of wasted computation in MAE with negligible impact on FLOPs: that reconstruction of unmasked patches is thrown away unused.
information from the input, while
Combining these ingredients we present CAN, minimal fusion of contrastive learning, masked autoencoders and denoising diffusion training loss. Our method enjoys stronger performance than its constituent parts do their own, especially pronounced benefits more uncurated datasets such as JFT-300M, which contains 300 million highly heterogeneous images, often containing artifacts (e.g., watermarks). For instance, evaluating JFT-trained ViT-L models using the top-1 accuracy of an ImageNet-trrinedd linear probe, MAE achieves 64. 14.1% and SimCLR achieves 73.4%. while CAN achieves 75.4%. CAN masks 50% of patches in each view, making significantly more scalable than prior contrastive methods that use two full image views. Our contributions are:
1. We present CAN, a simple self- learning algorithm with good scaling properties, making suitable for training on very large image datasets, such as the JFT-300M dataset.
2. CAN is much more efficient than SimCLR (Figure 1). For instance, SimCLR uses 70% more FLOPs than CAN with ViT-L models.
3. CAN is more robust to distribution shifts than MAE or SimCLR, and performs better on wide range few-shot and linear transfer tasks.
2 RELATED WORK
Masked image models with Vision Transformers. The advent of the Vision Transformer (ViT) (Dosovitskiy et al., 2021b) provoked focused effort to develop strong self-supervised learning frameworks that use ViT backbones. Works such as DINO (Caron et al., 2021) and MoCo-v3 (Chen et al., 2021b) demonstrated that techniques developed with ConvNet backbones in mind could also perform competitively using ViTs after proper tuning to suit the new architecture. ViT-specific methods have emerged since then, particularly masked image modelling (Bao et al., 2022: Chen et al., 2022; Xie et al., 2022), which takes inspiration from pre-training methods used in NLP (Devlin et al., 2018). Notably MAE (He et al., 2022) showed that classical masked autoencoding approaches could be used to pre-train ViTs without passing masked tokens through the encoder. This provides significant efficiency boost; our method similarly takes advantage of this.
2

---

Under review as a conference paper at ICLR 2023
Contrastive learning in computer vision. Self-supervision has received significant attention in computer vision as it offers a way to extract general purpose features without supervision. In par- ticular, contrastive learning (van den Oord et al., 2018; Hénaff et al., 2020; Chen et al., 2020b; He et al., 2020; Tian et al., 2020; Chuang et al., 2020; Hénaff et al., 2021) has achieved state of the art performance by enforcing invariance to augmentations, whilst using negative samples (Robinson et al., 2021a; Ge et al., 2021) to avoid trivial solutions by spreading the embedding out uniformly on the sphere (Wang& & Isola, 2020). The contrastive pre-training task is conceptually very different from masked image models such as MAE, which learn spatial statistical dependencies. Another distinction is that autoencoders encourage information preservation in latent representations, whilst contrastive learning could suppress features (Chen et al., 202 la; Robinson et al., 2021b). This leads us to hypothesize that the two approaches learn different data features, and may therefore be com- plementary learning mechanisms. This motivates us to combine contrastive learning and masked image modelling so as to develop a reinforced pre-training task that enjoys the merits of each.
Denoising diffusion models. Denoising autoencoders (DAE) (Vincent et al., 2010) learn to recon- struct clean data given a noisy input. By learning to map low-density data regions to high-density regions, DAE learns the shape of the data manifold. This connection was made precise by Vincent (2011), who showed that DAES learn the score- ecore-function sxx) = Vx log p(x). This key observation underpins the significant recent advances in generative diffusion models, which use estimate of the score-function to generate samples (Ho et al., 2020; Song et al., 2021). )). The recent success of DAEs in generative modelling has not yet translated to representation learning, with some excep- tions (Asiedu et al., 2022: Zaidi et al., 2022). In this work we exploit a denoising autoencoder to climinate the MAE inefficiency of reconstructing unmasked patches but never using them.
Concurrent work. Several recent works propose approaches that combine ideas from masked im- age modelling and Siamese self-supervised learning. For instance, Huang et al. (2022) propose a combination of contrastive and masked reconstruction objectives using one masked view, and one full (unmasked) view. Other recent works (Tao et al., 2022; Chen et al., 2022; Assran et al., 2022) use similar asymmetric designs. The key distinction between CAN and concurrent work is that we strike a different balance between simplicity, efficiency, and performance: we focus on developing a simple, efficient and symmetric method: we use two masked views and no momentum encoder. We hope the simplicity and efficiency of CAN will make it easy to adapt and modify in future work.
3 A SIMPLE CONTRASTIVE MASKED AUTOENCODER FRAMEWORK
Our approach a minimal synthesis of contrastive learning, the masked autoencoder (He et al., 2022), and the denoising loss used in the training of diffusion models. We focus simplicity and scalability, aiming to design a hybrid with as few complex or costly components as possible. We also aim to minimize wasted computation: in particular, the MAE decoder requires reconstructions of all patches, but only those of masked patches are used in the loss. Below, first we detail the basic pipeline of generating views and passing masked inputs through the encoder and decoder. Then we explain the three different objectives we use: contrastive, reconstruction, and denoising. The penultimate section describes the combined objective, and the final section discusses scalability.
3.1 OVERVIEW OF METHOD
Given batch of n images {x] we generate two views x², x² E Rhxux3 of each image without supervision using the same data augmentations as Chen et al. (2020b). Each image is then split into T (h/p) x (w/p) patches of size p X p. xl.puch x2 x input to the ViT encoder. We always assume that p divides h and w. Two masks M e {0, 1)" are independently generated, with 1 in coordinate E (I,... T} indicating that the Ith patch is masked. Each patch is left unmasked independently with probability m. conditioned on always having exactly T" = m patches unmasked, which we assume is an integer. In all CAN experiments our default masking rate is m 50% unless explicitly stated otherwise (for MAE we use the default 75%). Following He et al. (2022), only the T' unmasked patches are passed to the VIT encoder, which processes the two views in parallel. Masking a large fraction of patches from both views make our method much more efficient (see Table 1) than contrastive methods that use two full views, and recent works that use one full view and one masked view (Assran et al., 2022: Huang et al., 2022). Finally, we collect the embeddings of unmasked tokens 2/1 2? E R and
7'xpxxx3 in preparation for
3

---

Under review as a conference paper at ICLR 2023
Mask, augment
Gauesian nose
Replace masked patches
Reconstruct
Input
View
VIT
encoder
Avg. pool
Proj MIPP
InfoNCE
VIT encoder
VIT decoder
Misik, augment
Gaussian naise
Replace masked patches
Reconstruct
Figure 2:
Two views of an image are generated, 50% of patches randomly masked in each, and noise is added to patches. An encoder is trained to solve three tasks: 1) decoder that reconstructs missing patches, 2) Denoise: reconstructs the noise added to unmasked patches, and 3) Contrast: pooled patches are passed to a contrastive loss, using in-batch samples as negatives (Chen al., 2020b).
The CAN framework:
Reconstruction: encoded patches are passed
reshape into I d tensors by adding a learned [M] embedding to positions corresponding to masked tokens. The result passed through a comparatively lightweight VIT decoder to produce outputs x x in image space R
3.2 CONTRASTIVE LEARNING OBJECTIVE
The embeddings 2 zi E RT xd returned by the encoder are pooled via simple mean along the first dimension to form d-dimensional embeddings. which are passed through lightweight MLP projection head that maps into a lower dimension space E" r < d, and normalized to unit length to produce embeddings u. u1 E R² for = =1,.. ......r n. For the ith batch item we collect the other 2n 2 samples in-batch N₁ (u). uz) di ; to use as negatives, and compute the InfoNCE loss:
e nf/F
LIfONCE
ΣΣ log eu! u//7 Eu-EN, EN, eui uu-/
where T > 0 is temperature parameter, which we set to 7 = 0.1 by default.
3.3 PATCH RECONSTRUCTION OBJECTIVE
The outputs x X3 1,... of the ViT decoder are trained to reconstruct the missing patches of each image. Corroborating the findings of He et al. (2022), we find it best to only compute the reconstruction loss on masked patches:
Crce
Σ 1=12 i=1
M (x" x()
2n
where o multiplies all pixels in the ith patch of the residual image x - x by (M)); E {0, 1}. Whilst computing the loss only on masked patches gives better performance, indicates wasted computation since the decoder also produces reconstructions for unmasked patches. To avoid waste we propose an alternative objective specifically for unmasked patches, which we discuss next.
3.4 DENOISING OBJECTIVE
Inspired by the significant advances in diffusion modelling using denoising training objectives (Ho et al., 2020; Kingma al., 2021) and their equivalent coore-based counterparts (Song et al., 2021; Vincent. 2011) we revisit the suitability of denoising for self-supervised learning. We add indepen- dent isotropic Gaussian noise each image x x + ofee with ef ~N(0, 7) and of uniformly

---

Under review as a conference paper at ICLR 2023
sampled from interval [0, mmx] This noisy imput is masked and passed to the encoder de- scribed in Section 3.1. When passing encoded patches to the decoder we make a small addition to the method in Section 3.1 provide the decoder with information separate noise from the ground truth image. This is motivated by denoising diffusion methods, which pass both the noisy image and the noise level as inputs to the denoising model (Ho et al., 2020). We achieve this by using of as a positional encoding in the decoder, similarly to Vaswani et al. (2017). First we produce a sinusoidal embedding of of E R which is passed through a lightweight 2 layer MLP with ReLU activations of constant width d to produce (learnable) em- bedding P E Red, whose dimension matches the latent dimension of z E RT'xd We add the result to each embedded token (including missing tokens [M] provide noise-level information: (z,); - (z)) ) + P. for = 1.... J. and pass the result to the decoder producing x,. We define our denoising loss function, which is computed only on unmasked pixels:
the noise level of to help
Lieenoie
1
Σ V=
Σ
1(11
MT) (ofe) x1
2n
1.2 1=1
where, is as defined in Section 3.2. Note that this denoising loss extremely lightweight, introducing only very small overhead due to the MLP. We emphasize that the re- construction of noise patches comes zero additional cost since the de- coder produces reconstructions of all patches, both masked and unmasked, though only reconstructions of masked patches are used in Lree Fi- nally, it has often been observed in the diffusion modelling literature that although it is equivalent train denoising model estimate the noise, to estimate the clean in- put itself (Vincent, 201 I), there is a big empirical gap between the two, with noise prediction faring better. While we do not pursue it further, our testing corroborates this.
Sumple
MLP
Norie level embrccling
VIT encoder
VIT deccddr
Sugment
.fl nolos
enbeddings
Figure 3: Denoising: Both the encoded patches and the
noise level are passed to the decoder by passing through an MI.P, and adding the result to each embedded token.
Ablation: Table 1 studies the effect of each of the components of the denoising method. We use ViT-B models trained for 100 epochs on ImageNet, and con- sider four settings, each adding in more parts of the method: 1) CAN with no denoising, 2) adding noise to the input only, 3) adding noise and using the denoising loss, and 4) the full method with all of the described components, including using o. as a positional encoding in the decoder. Results show that simply adding noise as data augmentation improves performance by 0.7%, which can be improved to 1% by adding a reconstruction loss with noise level passed as an argument. The noise level argument is necessar: the reconstruction loss without noise level argument performs worse (68.4%) than noise with no reconstruction at all (68.6%).
None
+noise +noise, +loss Full
67.9 68.6
68.4 68.9
Table 1: Ablating components of the de- noising objective. "Full" denotes the entire method as described in Section 3.4
We emphasize that the improvement from denoising comes at minimal cost to run time and memory during training, since it uses reconstructions produced by the decoder, which in the case of MAE are simply thrown away unused. Denoising prediction encourages the encoder extract high-frequency features, which we hypothesize is complementary to reconstruction and contrastive tasks.
3.5
THE COMBINED OBJECTIVE FUNCTION
The overall CAN objective trains the encoder and decoder to optimize three losses combined:
LCAN AIntoNCE + AreeCroe + AdenoiseCCenoiis
where 0 S AInfONCE Apec Adenoise= and ALafONCE Aree + Adenoise weight the objectives. In practice we parameterize the weights by eliminating one variable using the equality constraint, taking: Ares = X) where < X S 1. This parameterization
(1 ALaONCE) À and Adenaise = AIaONNC.)
5

---

Under review as a conference paper at ICLR 2023
makes it easy control the relative weighting between the two reconstruction losses Liee Lennosee the one hand, and the contrastive loss LIstONCE on the other. Empirically we find that performance is very robust to the choice of X, and many choices of AIfONCE also work well (sce Section 5).
3.6 DISCUSSION ON EFFICIENCY
The goal this work is to propose conceptually minimal combined contrastive masked autoen- coder approach, aiming to find better trade-offs between simplicity, efficiency, and performance. Consequently, we choose to omit number of commonly used self-supervised learning design com- ponents. For instance, we do not use momentum target network multiple views (multi-crop), since they both increase memory requirements and run time. Even without these commonly used components, minimal framework achieves very strong performance compared to prior work, and importantly improves performance over its contrastive and autoencoder constituent parts. We expect that wide range of modifications, such momentum target networks (He al., 2020) and multi-crop (Caron et al., 2020), will improve performance further on top of the core method.
4 RESULTS
4.1 PRE-TRAINING ON UNCURATED DATA: JFT-300M
key promise of self-suuer learn- ing is to allow models to be trained on extremely large scale image datasets collected from the Web. Not only such data likely to be unannotated, but also uncurated: images contain- ing many objects, variable lighting, ar- tifacts (e.g., watermarks) and on. The large variation in images found online presents major challenge self-supervised learning, and it is not guaranteed that methods that work well on curated (and comparatively smaller) datasets such as ImageNet will work equally well on less curated data. To study how CAN scales to large datasets use JFT-300M (Sun et al., 2017), dataset of around 300 million images.
Architscture
Epochs
IN-IK top-1
MoCLR (Tian et al., 2021)
R50
5000
67.6
BYril et al., 2020)
R50
5000
67.9
DnC (Tian
2021)
R50
1000
67.9
DaC (Tian et al., 2021)
R50
4500
70.7
MoC1. (Tian et al., 2021)
R200x2
5000
DnC (Tian al., 2021)
R200x2
3000
77.3
MAE' (He et al., 2022)
VIT-L
1600
50.5
MAE' (He et al., 2022)
VIT-L
5000
64.1
SimCLR (Chen al., 20206)
VIT-B
800
65.8
SinCLR' (Chen al., 2020b)
VIT-L
800
72.6
SunCLR! (Chen et al., 2020h)
VIT-L
1600
73.1
SimCL.R (Chen et al., 2020h)
VIT..
5000
73.4
CAN (ours)
VIT-B
800
67.1
CAN (ours)
VIT-L
800
72.8
CAN (ours)
ViT-L
1600
74.3
CAN (ours)
VIT-L
3000
75.3
CAN (ours)
VIT-L
5000
75.4
Comparison to the state of the art on ImageNet linear probe. CAN outper- forms all methods except DnC, which uses a complicated multi-stage training process. Computation is measured as ImageNet-equivatee epochs. "Our implementation.
Table 2: JFT-300M pre-training:
Setup. Training time is measured in ImageNet-equivallent epochs: 1 epoch equals 1281167/ (hathh size] steps, the number of steps in one IN-1K epoch. Models are evaluated using linear probe and finetuning on IN-1K. All hyperparameers were tuned on IN-IK, besides learning rate and weight decay which we cut by a factor of 4 and 2 respectively stabilize training on JFT-300M. See Appendix C and Section 5 for details.
Results. Figure compares CAN to SimCLR and MAE baselines using ViT-L models. CAN achieves much better trade-off between efficiency (measured in FLOPs) and performance using ViT-L models for all three methods: CAN uses 41% fewer FLOPs than SimCLR and consistently outperforms SimCLR and MAE; for training ViT-I. models for 5000 epochs, CAN achieves an IN-1K linear probe performance of 75.4%, compared to 71.8% for SimCLR and 64.1% for MAE. The relatively poorer linear probe performance of MAE on JFT-300M highlights the non-triviality of scaling from IN-IK to larger datasets and suggests that while MAE is scalable for model size, scalability to larger datasets requires further study. Figure (right) gives finetuning results. CAN performs favourably: for a 5000 epoch pre-training schedule, CAN achieves an IN-IK linear probe performance of 86.1%, compared to 85.5% for SimCLR and 85.4% for MAE. CAN also enjoys better scaling with training schedule length than either MAE SimCLR.

---

Under review as a conference paper at ICLR 2023
Method
Pro-training epochs
Encoder
No Additional params.
Masked image
Finetune
Linear probe
from scratch!
100
VIT-B
79.1
from scratch
Swin-T
81.3
McCo-v3 (Chen al., 2021b)
300
VIT-B
X
83.0
76.7
DINO (Caron 2021)
ViT-B
82.8
78.2
EsVIT (Li 2021)
300
Swin-T
78.1
CIM (Fang al. 2022)
300
VIT-B
83.1
CAE (Chen al. 2022)
800
VII-B
83.8
68.6
CAE (Chen 2022)
2022)
1600
VIT-B
83.9
70.4
BETT (Bao 2022)
VII-B
83.2
37.6'
SimMIM (Xie al., 2022)
800
VII-B
83.8
56.7
MAF (Ie al., 2022)
800
WTB
83.1
MAE (Hl 2022)
1600
ViT-B
83.6
68.0
CAN (ours)
800
W
83.4
74.0
CAN (ours)
1600
VIT-B
83.6
74.8
SimCLR! (Chen al., 2020b)
800
VIT-
83.4
73.9
MAE (Il 2023)
800
VIT.I.
84.9
73.5
MAE (He 2022)
800
VIT-L
83.7
71.4
(ours)
VILL
Table 3: Pre-training on ImageNet-IK. 'Our implementation. *Quoted from Chen et al. (2022).
We also compare CAN to the current state of the art on JFT-300M pre-training in Table 2. Our best performance, 75.4% with VIT-L outperforms all methods besides DrC, with 77.3% (Tian al., 2021) with x200 However we note that CAN considerably simpler than DnC, which involves multiple training steps including training 10 separate "expert" models (each large as the final model), and then using MoCLR (an improvement of SimCLR that adds a momentum encoder and more), and finally using distillation produce single model. Our calculations suggest that training a ViT-L with CAN is about 3x faster than training the considerably smaller ResNet50 with DnC in terms of wall clock time (see Appendix for explanation). CAN on ViT-L outperforms MoCLR with x2 backbone (similar parameter counts), where we note that MoCLR performs as well or better than BYOL and MoCo-v3 on IN-1K (Tian al., 2021).
4.2 TRE-NNAINING ON IMAGENET
Next evaluate our method using ImageNet (IN-IK) pre-training to verify that it remains com- petitive in this setting. Results in Table 3 record the top-1 accuracy on IN-1K classification of finetuned models, and linear probes. Finetuning CAN achieves 83.6% with ViT-B, outperforming other contrastive approaches such as MoCo-v3 (83.0%), and is competitive with other state-of-the- art approaches such as CAE (83.9)). The linear probe performance of CAN is 74.8% using ViT-B, beating all masked image modelling methods, the best of which is CAE with 70.4% (Chen et al., 2022). CAN only outperformed by MoCo-v3 and DINO, both which use momentum encoders and two full image views, and in the case of DINO further 10 multi-crop views. Note that the masked image column indicates whether a method uses one or more full image views as input to the model, and the no additional parameters column indicates whether method relies other parameters besides the main encoder, e.g., from a pre-trained tokenizer, or a momentum updated target encoder. We also report results for MAE implementation, which approximately matches the original numbers reported by He et al. (2022), validating our MAE results on JFT-300M.
4.3 FEW-SHOT LEARNING
We use linear probes to evaluate suitability of CAN for few-shot learning, following the protocol of Dosovitskiy et al. (2021a). We use the models pre-trained on JFT-300M for 5000 epochs whose ImageNet performance is recorded in Figure 1. Results in Figure 4 for few-shot transfer learning on other datasets show that the superior performance on IN-1K translates to strong performance on other tasks. We also note that 25-shot VIT-L models beat full-shot both DnC and BYOL ResNet50 models (also trained for 5000 epochs on JFT-300M) on 6 out of 8 datasets (Tian et al., 2021). See Appendix A for many additional results for different training schedules and model sizes.
4.4 ROBUSTNESS TO DISTRIBUTION SHIFT
Finally, consider the robustness of CAN to distribution shifts. We ViT-L backbones trained for 5000 epochs on JFT-300M, which have been finetuned on IN-IK. Model performance evalu- ated number of different validation sets with the same 1000 classes as IN-1K Mao et al. (2022).

---

Under review as a conference paper at ICLR 2023
JFT-300M 5000 epochs: 10-shot
top-1 (%)
100
MAC
StmcLa
6633
72.3 27.9
$1.5 46.
CAN
53.0
21.2342
12.7
Birds
IN-1K
Pets
CIFAR100
DID
Cars
UC-Merced Col Hist
Caltech
JFT-300M 5000 epochs: 25-shot
top-1 (%)
100
CAN (oazo)
MAE
StarCLR
66.5
21.5
dB3
703
33
50
Birds
IN-IK
Pets
CIFAR100
DTD
Cars
UC-Merced Col-Hist
Caltech
Figure 4: Few-shot: ViT-L models pre-trained on JFT-300M for 5000 epochs are evaluated on
datasets in few-shot setting (10-shot and 25-shot). CAN outperforms MAE and SimCLR.
JFT-300M: 5000 epochs
top-1 (%)
100
85.3 361
30.0
CAN
59.0 474
50
$1.
40.7
43.3 46.
32.6
IN-A
ObjectNet
IN Sketch
IN Rendition
IN-v2
IN-1K
IN-Real
IN-1K: 800 epochs
top-1 (%)
342
30.0 46.7 50.0
40.3
362 31
409
IN-A
ObjectNet
IN IN-Sketch
IN-Rendition
IN-v2
IN-1K
IN-Real
Figure 5: Robustness:
Evaluating performance under distribution shifts with respect models finetuned on IN-IK. Validation performance of ViT-L. models reported on 7 different datasets.
Figure reports results on the following 7 validation sets, which cover a large variety distribu- tion shifts: original IN-1K (Deng et al., 2009), IN-v2 (Recht et al., 2019), IN-Real. (Beyer et al., 2020), IN-Adversarial (Hendrycks et al., 2021b), IN-Rendition (Hendrycks et al., 2021a), Object- Net (Barbu et al., 2019). CAN performs favourably under both JFT-300M and IN-IK pre-training, beating SimCLR and MAE baselines in nearly all cases. See Appendix A for additional results.
HYPERPARAMETER ANALYSIS
We study the different components of CAN to better understand the effect of the different mecha- nisms, and determine optimal parameter configurations. All ablations use ViT-B models trained for 100 epochs on IN-IK, unless explicitly said otherwise. We use the best loss weights and noise level in these experiments for experiments in Section 4.
top-1 uocuracy
70
top-1 accuracy
67
86
CAN lours! SimCLR
CAN (ours) SImCLR
20
30
40
5o
60
70
80
10
20
30
40
50
Masking rate (IN-1K)
Masking rate (FF-300M)
Figure 6: CAN and SimCLR with different masking rates.
ViT-B models are pre-trained for 100
epochs on IN-IK (left), and 800 epochs on JFT-300M (right).
8

---

Under review as a conference paper at ICLR 2023
top-1 accuracy
70
top-1 accuracy
top-1 accuracy
60
50
68.7
40
68.6
0.00
0.05
0.10
0.15
0.20
0.0
0.1 02 03
0.0
0.1
0.2 0.5
0.4
Contrastive loss weight
Maximum noise level
Denoising loss weight
Figure 7: VIT-B models pre-trained on IN-1K for 100 epochs.
Left: The best contrastive loss
weight is small but non-negative. Middle: A wide range of max Right:
values improve over no-noise.
Performance is not sensitive the denoising loss weight.
Complementarity of contrastive and reconstruction losses. A key hypothesis motivating our work is that contrastive learning and masked autoencoder reconstruction may not only be compati- ble training objectives, but are complementary ones. Table 4 compares the final training value of the contrastive LIIANCE and reconstruction Lee when jointly trained (i.e., CAN) compared to only op- timizing LINONNE (SimCLR) or only feee (MAE). The results support the hypothesis: joint training achieves a lower loss on both objectives compared individual training.
Masking rate. Figure 6 reports the behavior of CAN and SimCLR under different masking rates IN-1K and JFT-300M pre-training (for JFT-300M we use 800 epochs). The performance of SimCLR decreases as the masking rate increases, suggesting that masking is not an effective data augmentation. contrast, perfor- mance of CAN peaks at non-zero masking rate, but at much lower rate than 75% used by MAE on IN-1K. This occurs since very low masking rates are preferred by the contrastive part of CAN, but severely damage the autocncoder part, which can learn trivial solutions. The considerable efficiency improvement from masking 50% of patches more than com- pensates for the small drop in performance for fixed number of epochs.
Method
Contrastive loss
Reconstruction loss
SimCLR MAE
9.157
0.1658
CAN (ours)
9.143
0.1633
Table 4: Complementary training: All methods use 50% masking for fair compar- ison. CAN training achieves lower training loss for both contrastive and reconstruction than individual training.
Contrastive loss weight. We vary the weighting AIiONCE used to weight the contribution of the contrastive and reconstruction losses.
Method
Image top-1 f
Recall that larger AtafNNCE: places higher weight on the contrastive loss. Results Figure show that the best weight AIsONNCE 0.03,
AN
42,8
CN
68.5
CA
67.9
which approximately balances the magnitudes of the two terms (see
CAN (ours)
68,9
Table 4).
Denoising loss weight and noise level. We study the noise level interval [0. maa]] from which sample input noise, and the weight A balancing the denoising and reconstruction losses. Results in Fig show that the best maximum noise level max = 0.05, and that similar performance is attained for number of different weights on the denoising loss.
Table 5: Ablating CAN: We remove each of the three loss terms in CAN one by one.
Ablating CAN: CAN is comprised of three components: (C) contrastive, (A) masked autoencoder, and (N) de-noising losses. We ablate each of the three components in Table 5, setting the loss weight to zero to "remove" component. We use ViT-B models pre- pre-trained for 100 epochs. Removing any component leads to worse performance, with contrastive loss hurting the most.
DISCUSSION
We present CAN, a simple, efficient and scalable self-supervised method for visual representation learning. CAN combines ideas from contrastive learning, masked autoencoding, and diffusion de- noising into a single high-performing method. Extensive empirical results show that CAN scales with minimal changes the large uncurated datasets, providing significant boost over SimCLR and MAE methods on a wide range of downstream tasks and evaluations, including linear probes, few-shot, robustness, and finetuning. Our results suggests that contrasting and reconstruction are complementary principles that can mutually reinforce one another.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Emmanuel Brempong Asiedu, Simon Kornblith, Ting Chen, Niki Parmar. Matthias Minderer, and Mohammad Norouzi. Decoder denoising pretraining for semantic segmentation. preprint arXiv:2205. 1423, 2022.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In preprint arXiv:2204.071441 2022.
Dana H Ballard. Modular learning in neural networks. In Association for the Advancement of Artificial Intelligence (AAAI), volume 647, pp. 279-284, 1987.
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. In Int. Conf. on Learning Representations (ICLR), 2022.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron den Oord. Are we done with ImageNet? In preprint arXiv:2006.07159, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances Neural Information Processing Systems (NeurIPS), volume 33, pp. 9912-9924 9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Int. Conference Computer Vision (ICCV), pp. 9650-9660, 2021.
Mark Chen, Alec Radford, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Int. Conference on Machine Learning (ICML), 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Int. Conference on Machine Learning (ICML), pp. 1597-1607. PMLR, 2020b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 11834-11845, 2021a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- 2022.
tion learning. In preprint arXiv:2202.034
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- supervised vision transformers. In Int. Conference on Computer Vision (ICCV), pp. 9640-9644, 2021b.
Ching-Yar Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanic Jegelka. De- biased contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. -77558775, 2020.
Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. When does contrastive visual representation learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14755-14764, 2022.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. RandAugment: Practical auto- mated data augmentation with reduced search space. In 2020 IEEF/CVF Conference on Com- puter Vision and Pattern Recognition Workshops (CVPRW), pp. 3008-3017, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 248-255 leee, 2009.
10

---

Under review as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics (NAACL), 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Int. Conf. on Learning Representations (ICLR), 2021a.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Int. Conf. Learning Representations (ICLR), 2021b.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training, preprint arXiv:2202.03382, 2022.
songwei Ge, Shlok Kumar Mishra, Haohan Wang, Chun-Liang Li, and David Jacobs. Robust con- trastive learning using negative samples with diminished semantics. In Advances in Neural Infor- mation Processing Systems (NeurIPS), volume abs/21 10.1111..111 14189, 2021.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training Ima- geNet in 1 hour. preprint arXiv: 0777700667777 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoeneoders are scalable vision learners. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Eslami S. M. Ali Doersch, Carl Do- ersch. and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. In Int. Conference on Machine Learning (ICML), pp. 4182-4192, 2020.
Olivier Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aäron Van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. In Int. Conference on Computer Vision (ICCV), 2021.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -f-iiiiiiitttiinnnnnn generalization. In Int. Conference Computer Vision (ICCV), pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15262- 15271, 2021b.
Charles Hermann, Kyle Sargent, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu, Dilip Krishnan, and Deqing Sun. Pyramid adversarial training improves ViT performance. IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 13419-13429, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851, 2020.
11

---

Under review as a conference paper at ICLR 2023
Xianxu Hou, Linlin Shen, Ke Sun, and Guoping Qiu. Deep feature consistent variational autoen- coder. 2017 IEEE winter conference on applications of computer vision (WACV), pp. 1133- 1141. IEEE, 2017.
Zhicheng Huang. Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xi- aohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. arXiv:2207.13532v1 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances Neural Information Processing Systems (NeurIPS), volume 34, pp. 21696-21707, 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. In Int. Conf. on Learning Representations (ICLR), 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in Adam. preprint 1rXiv:1711.055 2017a.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Int. Conf. on Learning Representations (ICLR), 2017b.
Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa. Discrete representations strengthen vision transformer robustness. In Int. Conf. on Learning Rep- resentations (ICLR), 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning. pp. 8748-8763. PMLR, 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classi- fiers generalize ImageNet? In Int. Conference Machine Learning (ICML), pp. 5389-5400. PMLR, 2019.
Joshua Robinson, Ching-Yat Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In Int. Conf. on Learning Representations (ICLR), 2021a.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 4984-4986 2021b.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- generative modeling through stochastic differential equations. In Int. Conf. on Learning Representations (ICLR), 2021.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lu- cas Beyer. How to train your ViT? Data, augmentation, and regularization in vision transformers. In Transactions on Machine Learning Research (TMLR), 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effec- tiveness of data in deep learning era. In Int. Conference Computer Vision (ICCV), pp. 843-852, 2017.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. In preprint trXiv:2206.0/2/)) 2022.
12

---

Under review as a conference paper at ICLR 2023
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Europ. Confer- ence on Computer Vision (ECCV), pp- 776-794, 2020.
Yonglong Tian, Olivier Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised
learning from uncurated data. In Int. Conference on Computer Vision (ICCV), pp. 10063-10074, 2021.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. preprint arXiv: 1807.03748, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurlPS), 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674 2011.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Int. Conference on Machine Learning (ICML), pp- 9929 9939. PMLR, 2020.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9653-9663, 2022.
Yang You, Igor Gitman, and Boris Ginsburg, Large batch training of convolutional networks. In ECCV, 2017.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners image-text foundation models. preprint arXiv:2205.01917, 2022.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee whye Tch, Alvaro Sanchez-Gonzaler. Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via de- noising for molecular property prediction. preprint arXiv:2206.000
2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Association for the Advancement of Artificial Intelligence (AAAI), 2020.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL TRANSFER LEARNING RESULTS
We report additional results for few-shot learning and robustness.
Robustness:
Sections 4.4 reports robustness results for ViT-L models pre-trained on JFT-300M for 5000 epochs, and VīT-L models pre-trained on IN-IK for 800 epochs. In both cases we report the performance of the models after finetuning on IN-IK.
Here we report the same robustness results for ViT-I. models trained on JFT-300M for 1600 and 800 epochs (Figure 8), and ViT-B models pre-trained for 800 epochs (Figure 9). Figure 9 also compares our ViT-B model to VIT-B models trained from scratch on ImageNet. We find that our model considerably more robust than training with cross-entropy and Mixup from scratch, and also outperforms PyramidAT (Herraan et al., 2022), an adversarial training method that introduces significant overheads compared to standard cross-entropy training. We emphasize that here there are two differences in the training: a) the training algorithm itself, and b) the data seen by the model. Our model sees extra JFT-300M data not seen by the other two approaches. This means that the methods not exactly comparable. It is, however, realistic setting showing the benefits to robustness of pre-training on large datasets.
JFT-300M: 800 epochs
top-1 (%)
MAE
B17
8&B
SunCLR
71.3 761 76.
50
20.3
CAN
J5.0
35.0
45- 112
51.0
IN-A
ObjectNet
IN Sketch
IN IN-Rendition
IN-v2
IN-1K
IN-Real
JFT-300M: 1600 epochs
top-1 (%)
100
35.0 800 274
SimCLR
CAN (ouns)
14.3
25-
IN-A
ObjectNet
IN- IN-Sketch
IN-Rendition
IN-v2
IN-1K
IN-Real
Figure 8: ViT-L models pre-trained on JFT-300M for 800 and 1600 epochs respectively, evaluated on 7 datasets with distribution shifts from IN-IK
JFT-300M: 800 epochs
top-1 (%)
MAR CAN (ours)
01.0 $3.0
50
150 46.1 464
21.
25.6
31.1
36.8
.7.4
35.0
13.8
IN-A
ObjectNet
-N-Sketch
IN-Rendition
IN-v2
IN-1K
IN-Real
Robustness: from scratch comparison
top-1 (%)
100
Supervised
PyramidAT
613
81.7
430
nia 882
(ours)
25
12.2
25.6
37.0
35.8 15.8
34.9
IN-A
ObjectNet
IN- IN-Sketch
IN IN-Rendition
IN-v2
IN-1K
IN-Real
Figure 9: Top: ViT-B models pre-trained on JFT-300M for 800 epochs, evaluated on 7 datasets with distribution shifts from IN-IK. Bottom: Comparison our JFT-300M pre-trained ViT-B model to training VIT-B from scratch IN-IK. We compare to standard supervised cross-entropy training with Mixup, and to PyramidAT (Herrmann et al., 2022), which uses an adversarial training method. CAN considerably outperforms supervised training, and beats PyramidAT in 6 out of cases without requiring adversarial training.
Few shot: Section 4.3 reports 10- and 25-shot results for ViT-L models pre- trained on JFT-300M for 5000 epochs. Here report 1- and 5-shot results for the same models in Figure 10. We additionally
14

---

Under review as a conference paper at ICLR 2023
show the full of {1, 10, 25)-shot results for VIT- models pre-trained JFT-300M for 800 and 1600 epochs (Figures 11 and 12 respectively), ViT-B models pre-trained on JFT-300M for 800 epochs (Figure 13), and VIT-L models pre-trained on IN-IK for 800 epochs (Figure 14).
JFT-300M 5000 epochs: 1-shot
top-1 (%)
100
MAB
SteCLL
20.
(numn)
35. 10.0 9214
74
21 21.7.0.1
16.
28.7304
zn
Birds
IN-1K
Pets
CIFAR100
Cars
UC-Merced
Col-Hist
Caltech
JFT-300M 5000 epochs: 5-shot
top-1 (%)
717
$1.5
38.5
456
8311
15
186224
26.5
21-1
Birds
IN-1K
Pets
CIFAR100
DID
Cars
UC-Merced Col-Hist
Caltech
Figure 10: Few shot: VIT- models pre-trained JFT-300M for 5000 epochs evaluated on 9 few- shot learning tasks. Results accompany the 10- and 25-shot results in Figure
JFT:300M 800 epochs: 1-shot
top-1 (%)
61
70.0)
56.6
26.7
19:114
12.4
Birds
Cars
IN-1K
Pets
CIFAR100
DTD
UC-Merced Col- Col-Hist
Caltech
JFT-300M 800 epochs: 5-shot
top-1 (%)
100
MAT StarLLR
22.4
ds. 72.72.477
50
CAN (osza)
33.1
36.1
to
59.2418
Birds
Cars
IN-1K
Pets
CIFAR100 00
DTD
UC-Merced
Col-Hist
Caltech
JFT-300M 800 epochs: 10-shot
top-1 (%)
100
MAE
814 16.98.22
as 4R.
StnCLR
(nera)
58.356
41.0
158
Birds
Cars
IN-1K
Pets
CIFAR100
DID
UC-Merced
Col-Hist
Caltech
JFT-300M 800 epochs: 25-shot
top-1 (%)
o 9295
053 66
1244 2:4
75
51.
21.3
36.0
20.1
62.
442225
25.4
Birds
Cars
IN-1K
Pets
CIFAR100
DID
Uc-Mereed Col-Hist
Caltech
Figure 11: Few shot: VIT-L models pre-trained on JFT-300M for 800 epochs are evaluated on few-shot learning tasks.
15

---

Under review as a conference paper at ICLR 2023
JFT-300M 1600 epochs: 1-shot
top-1 (%)
100
MAZ
StmcLa
CAN
51.900
33.
17.6352
12.2
Birds
Cars
IN-1K
Pets
CIFARI00
DTD
UC-Merced Col Hist
Caltech
JFT-300M 1600 epochs: 5-shot
top-1 (%)
100
MAE
#24
StarcLt
ass 719314
50
CAN (oaza)
50.6
20-
3633
37.2
31.9
37.6
13.0
21822 22.5
21.9
21.0
Birds
Cars
IN-1K
Pets
CIFAR100 00
DTD
UC-Merced
Col-Hist
Caltech
JFT-300M 1600 epochs: 10-shot
top-1 (%)
84 86.6
78.9
75-
62.7
46.
4244005
$2.5
17.8
12.0333 52.25
26.7
Birds
Cars
IN-1K
Pets
CIFAR100
DTD
UC-Merced Col-Hist
Caltech
JFT-300M 1600 epochs: 25-shot
100
top-1 (%)
01.
SinCLR
714243
CAN
55.6
36.2
46.3414
322
Birds
Cars
IN-1K
Pets
CIFARI00
DID
UC-Merced Col-Hist
Caltech
Figure 12: Few shot: ViT-L models pre-trained on JFT-300M for 1600 epochs are evaluated on few-shot learning tasks.
JFT-300M 800 epochs: 1-shot
top-1 (%)
$1.2 (0.61.1
31.233
30.1
48.
4.0 52 6.0.
4.
Birds
Cars
IN-1K
Pets
CIFAR100
DTD
UC-Merced Col- Col-Hist
Caltech
JFT-300M 800 epochs: 5-shot
top-1 (%)
100
MAT StarLL
24.5
d5.1 73.775
CAN (oszs)
6L.2
50
AB
10.4
11.5
Birds
Cars
IN-1K
Pets
CIFAR100 00
DTD
UC-Merced Col-Hist
Caltech
JFT:300M 800 epochs: 10-shot
top-1 (%)
100
MAE
StnCLR
22. 80.
35.8
(nerd)
$0.8 10
12.514
49.7
27.6280
10.8
21.2
36.
Birds
Cars
IN-1K
Pets
CIFAR100
DTD
UC-Merced
Col-Hist
Caltech
JFT-300M 800 epochs: 25-shot
top-1 (%)
0.221
812 03002
75
50
24.366
15
28.
Birds
Cars
IN-1K
Pets
CIFAR100
DID
UC-Merced Col-Hist
Caltech
Figure 13: Few shot: VIT-B models pre-trained on JFT-300M for 800 epochs are evaluated on few-shot learning tasks.
16

---

Under review as a conference paper at ICLR 2023
IN-1K 800 epochs: 1-shot
100
top-1 (%)
SimCl R
00.3
50
GAN
412458
52.5
30.2
22:233
214
2412111
5.7 10.21
Birds
Cars
IN-1K
Pets
CIFARI00
DTD
UC-Merced Col Hist
Caltech
IN-1K epochs: 5-shot
100
top-1 (%)
75.77711
76
02.
50
CAN (ours)
50.3
100
46.7
Birds
Cars
IN-1K
Pets
CIFAR1O0 00
DTD
UC-Merced Col-Hist
Caltech
IN-1K 800 epochs: 10-shot
top-1 (%)
75-
SimCLR
MAB
29.1
78.4
52 57.5 92.6
79.
20.2
20.
d5.3
(ours)
JB. 484
54.5
27.261
Birds
Cars
IN-1K
Pets
CIFARI00
DTD
UC-Merced Col-Hist
Caltech
IN-1K 800 epochs: 25-shot
top-1 (%)
1.0
02.0
83.8
6L.2
35.
SimCLR
02:7481
50
CAN
Birds
Cars
IN-1K
Pets
CIFARI00
DTD
UC-Merced Col-Hist
Caltech
Figure 14: Few shot: ViT-L models pre-trained on IN-1K for 800 epochs are evaluated on 9 few- shot learning tasks.
We make a number observations.
Across all settings CAN generally performs the best on JFT-300M pre-training.
2. The situation less consistent on IN-1K pre-training For instance, although MAE has comparatively poor few-shot performance on IN-1K, it is competitive others for 25-shot evaluation: this setting CAN only beats MAE on out of datasets. However, on 10-shot CAN outperforms MAE and SimCLR in 8 out of 9 cases, showing subtle picture.
3. JFT-300M pre-training often outperforms IN-IK pre-training. Comparing Figures 11 and 14, JFF-300M yields better 25-shot CAN performance on out of 9 datasets.
4. Model scale helps. Comparing Figures and 11, ViT-L models perform best in nearly all cases.
A.I IMAGENET-2IK PRE-TRAINING.
We also consider the performance of CAN on pre-training on ImageNet-2IK (IN-21K), a publicly available dataset of 14.2 million images, grouped into 21,000 different classes Deng al. (2009). We use the same hyperparamter settings as JFT-300M training to train ViT-L models on IN-21K for 800 (IN-1K equivalent) epochs.
We run a full set of evaluations on finetuning, linear probe, robustness (Figure 15), and few-shot learning (Figure 16.
17

---

Under review as a conference paper at ICLR 2023
IN-21K: 800 epochs
top-1 (%)
100
714 76a
83.3
36.0
CAN tound)
934
29.
352 413
33.8 12.6
IN-A
ObjectNet
IN -N-Sketch
IN N-Rendition
IN-v2
IN-1K
IN-Real
Figure 13: Robustness: VIT-L models pre-trained on IN-21K for 800 (IN- equivalent) epochs are first finetuned IN-1K. The models then evaluated test datasets with different distribution shifts from IN-IK.
IN-21K 800 epochs: 1-shot
100
top-1 (%)
MAE
SamCLK
67.
73.4 26
50
CAN (ours)
46.06.77
51.7224
12.5
372353
20.1
12.522
3.
11.
Birds
Cars
IN-1K
Pets
CIFAR100 00
DTD
UC-Merced Col-Hist
Caltech
IN-21K 800 epuchs: 5-shot
top-1 (%)
MAB
83.0556
re 20.4
22.2
CAN (ours)
59.3.77
20.1
62-5622
PS 13.2
19.
27.3
Birds
Cars
IN-1K
Pets
CIFAR100
DTD
UC-Merced Col-Hist
Caltech
IN-21K 800 epochs: 10-shot
top-1 (%)
na UIS
75
SimCLR
33.2.0.2
50
52.5
47.8
33 32
19.0
249
Birds
Cars
IN-1K
Pets
CIFARI00
DID
UC-Merced Col-Hist
Caltech
IN-21K 800 epochs: 25-shot
top-1 (%)
1.00
78.7
201
a6. 15.882
SimCLR
50
CAN (ours)
57.155.4
28.2
31.1
30.0
Birds
Cars
IN-1K
Pets
CIFAR100
DTD
UC-Merced Col-Hist
Caltech
Figure 16: Few shot: ViT-L models pre-trained on IN-21K for 800 (IN-1K equivalent) epochs are evaluated on 9 few-shot learning tasks.
B RUNTIME OF CAN COMPARED TO DNC
In the main paper we estimate our method is significantly faster than DnC (Tian et al., 2021). We determined this approximate comparison from the following two pieces of information: 1) DnC reports that 3000 ImageNet epochs takes 29 hours on 512 TPUs for a ResNet-50 model 25M parameters), and 2) 3000 ImageNet epochs of CAN take 78 hours 64 TPUs for ViT-L model ( 300M parameters). We assume a linear relationship between number of TPUs and runtime. Under this assumption, estimate that CAN would take approximately 10 hours train with 512 TPUs, compared to the 29 hours reported by Tian et al. (2021) for a model with 1/10th the number of parameters. We emphasize that this far from exact comparison and is only intended a very approximate guide.
C HYPERPARAMETER SETTINGS
We list hyperparameters used for CAN pre-training in Table 6 and Table For preprocessing we closely follow SimCLR Chen al. (2020b). We the same hyperparameters for SimCLR pre-
18

---

Under review as a conference paper at ICLR 2023
training. For MAE pre-training for the use of Glorot uniform initialization instead of LeCun initialization as done in He et al. (2022). We found that this provided better performance for our IAX-based MAE implementation. Table 10 lists the hyperparameters for finetuning evaluations. We use the same set of hyperparameters for each finetuning each pre-training method, and for both ViT-B and ViT-I. model sizes. For linear probing we list the hyperparameters in Table 11 for which we followed the settings in He et al. (2022). We use global average pool of the final representation instead of the cls token.
the same hyperparameters listed in He al. (2022), except
MAE longer training: MAE pre-training for longer training (5000 epochs) JFT becomes un- stable after about 500k steps (training loss oscillates); this results poorer fine-tuning performance. To overcome this, we decrease the base learning rate by 75% as shown in Table 9. However our model CAN is more stable and we use the same hyperparameters across different numbers of epochs.
the hyperparameters and pipeline Doso- vitskiy et al. (2021a). We use the same pre-processing as was done (Kolesnikov et al., 2020). We use base learning rate of 0.01 and train for 2500 steps, using an imput resolution of 384 384.
Few shot training: For few-shot learning
Hardware details: We use TPU-v4 for all of our experiments. CAN on VTT-B uses 64 TPUs for batch size 4096. SimCLR, on the other hand, uses 128 TPUs for the same batch size, and is more compute intensive than CAN.
Decoder architecture: Our decoder architecture the same as He et al. (2022). We use standard VIT with decoder depth of 8 and decoder width of 512. We use 16 heads and 2048 the dimension of the MLP.
Projection head architecture: We use 2 hidden layers our projection heads. Each layer has -ooneeeedd (FC) layer (dim 4096) followed by BatchNorm (momentum=0.9) followed by ReLU. After these layers we have FC layer which transforms the features to 128 dimensions. We apply contrastive learning on top of these 128 dimensional features.
JFT-300M specific hyperparameters: All hyperparameters were determined by training on IN- IK, and directly transferred with JFT-300M pre-training, with the exception of learning rate and weight decay, which found needed to be at a lower level for JFT-300M. For all methods we divided the learning rate by a factor of 4, and the weight decay by a factor of 2, except for MAE where found that the original weight decay tuned on ImageNet worked better. Specifically, for CAN and SimCLR we used following parameter choices: wd 0.1/2 0.05 and Ir 1.25 x 10 00-4//1 4/4 3. 1.125 10-5 and for MAE we used Ir = 1.5 x 10 10-4/4- = 3.75 x 10-5, and tried und = 0.05/2 0.025, but found that the original wel 0.05 worked better, so kept this value.
19

---

Under review as a conference paper at ICLR 2023
C.I CAN AND SIMCLR HYPERPARAMETERS
config
value
oplimizer
AdamW (Loshchilov & Hatterr, 2017a)
base learning ratet ratet VIT-B)
2.5c-4
leurning rate (VII-L)
1.25e-4
weight decay (ViT-B)
0.05
weight decny (VIT-L)
optimizer momeutum batch size
0I. 92.99, 0.95 (Chen al., 2020a)
4096
leamiigg rate schedule
cosine decay (Loshchilov Hotter, 2017b)
warmup epochs (Goyal al., 2017)
augmentation
RandomResizedCrop. Color Jinering(strength=11.0
GanySallejpoaahhltty-22... Gaussian Blurring (prohability-0.5)
Table 6: Hyperparameters for CAN pre-training on ImageNet. Note that we use lower learning rate for ViT-L as compared to VTT-B, following Steiner al. (2021). We use the same hyper-parameters for SimCLR training.
optimizer
AdamW (Loshchilov Hutter, 2017a)
base leurning rate (VII-B)
2.5e-4
base learning rate (VTT-L)
3.125o-5
weight decay
0.05
optimizer momentum batch size
01. 0..00,9990000 0.95 (Chen et al., 2020a)
learning schedule
cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017)
augmentation
RundomResizedCrop Color Jinering(strengthel..0
GrnySSaleepoobaabittg:...1 Goussian Blurring (probabliay=0.5)
Table 7: Hyperparameters for CAN pre-training JFT-300M. Note that use lower learning rate for ViT-L as compared to ViT-B, following Steiner et al. (2021). We use the same hyper-parameters for SimCLR pre-training.
C.2 MAE HYPERPARAMETERS
config
value
optimizer
AdamW (Loshchilov & Hutter, 2017a)
base learning rate (VTT-L
1.5e-4
weight decay
0.05
optimizer momentum
01. 0t..2,0.,00000 0.95 (Chen et al., 2020a)
hatch size
learning schedule
cusine decay (Loshchilov Hutterr, 2017b)
warmup epochs (Goyal et al., 2017)
40
sugmentation
RundomieeszeeCCopp Color (tteringgstrengghee1...)
GrayScale(prosshilttt...3 Gaussian Blurring (probabiling-055)
Table 8: Hyperparameters for MAE pre-training IN-1K. We follow the choices made by He al. (2022).
config
value
optimizer
AdamW (Loshchilov & Hutter, 2017s)
base leamning rate (VTT-L
3.75c-5
weight decay
optimizer momentum
01 02009, 0.95 (Chen al., 2020n)
hatch size
4096
leamning schedule
cosine decay (Loshchilov Hutter, 2017b)
warmup epochs (Goyal et al., 2017)
40
angmentation
RundomResizedCrop, Color Jinering(strengghhl1.000
GnySScalepposshiilyy42.. Gaussian Blurring (prohability-..5)
Table Hyperparameters for MAE pre-training on JFT-300M with ViT-L models. The only differ- ence from the IN-1K configuration is the learning rate, which we reduced since we found tarining to be unstable.
20

---

Under review as a conference paper at ICLR 2023
C.3 FUNETUNING AND LIENAR PROBE HYPERPARAMETERS
config
value
opiimizer
AdamW (Loshchilov & Hutterr, 2017a)
base learning rale
Sc-1
weight decay
optimizer momentum
01. 90=0.99 0.999 (Chen al., 2020a)
batch size
1024
leamnngg schedulc
decay (Loshchilov Hutter, 2017b)
warmup epochs (Goyal et al,, 2017)
training epochs
100
label smoothing
drop path
0.1
layer wise decay
0.65
sugmentation
RundomiessizedCipg Flip. KandAugtlayens=2,
magnitude=9) (Cubuk al., 2020), Random Frase
(Zhong al., 2020ppmhhhhlity-0225
Table 10: Hyperparameters for finetuning CAN pre-trained model on ImageNet. We use the same hyperparameters for ViT-B and ViT-L, for both JFT-300M and ImageNet pre-trainined models.
config
value
optimizer
LARS (You al., 2017)
base learning rale
0.1
weight decay
0
optimizer momentum
0.9
batch size
16384
leamning schedule
cosine decay (Loshchilov & Hutter, 2017b)
warmup epochs (Goyal et al., 2017)
10
training epochs
100
batch norm momentum
0.9
label smoothing
0
augmentation
RundomResiieeiCrop
Table 11: Hyperparameters for linear probing for CAN pre-trained model on ImageNet. We use the same hyperparameters for ViT-B and ViT-L, for both JFT-300M and ImageNet pre-trainined models. Note that these hyperparamters are same as reported in He al. (2022).
21


Reviews:
[{"review_id": "td_Y3MsPQS", "reviewer": "Reviewer_y8Yc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a visual representation learning framework combining contrastive learning, masked autoencoders, and diffusion noise prediction. It is interesting to introduce noise prediction for the unmasked tokens. The models pre-trained by the framework achieve competitive results on the image classification task.", "summary_of_the_review": "I am inclined to accept the paper if the author can dispel my concerns well.", "strength_and_weaknesses": "Strength:  \nInteresting idea to add the diffusion noise prediction.  \nGood results for the linear probe.  \nClear written.  \n\nWeaknesses:  \n(1) The Title of the paper does not cover the meaning of “noise” prediction accurately since it’s a synthesis of contrastive learning, masked autoencoders, and noise prediction.   \n(2) Choosing SimCLR as the primary baseline in Table 2 and the following Figures is actually not convincing. SimCLR is an early contrastive learning-based method and its performance has lagged behind other contrastive learning methods. How about DINO and MOCO v3?  \n(3) In Figure 6, masking 50% of patches is not the best choice, why the paper still uses 50%? The authors should better provide more explanations and quantitative analysis.  \n(4) How about the ablation study between contrastive learning loss, reconstruction loss, and denoise loss? (That’s the result of a pairwise combination)  \n(5) Lack of downstream experiments on Object detection and Semantic segmentation.  \nTypos:  \nPage 4: “see Table 1” --> “see Figure 1”  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jEAQl7xNaWw", "reviewer": "Reviewer_Jkgq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "CAN is a method of self-supervised learning for visual representations. The method builds on three families of self-supervised learning methods: Contrastive learning, Masked Autoencoding, and Noise prediction. The paper claims that the learning mechanisms of each family are complementary to one another, and includes some experiments with comparisons to selected methods in the masked autoencoding family (MAE) and the contrastive learning family (SimCLR). The paper claims empirical improvements over MAE and SimCLR on ImageNet, and when pre-training on JFT-300M and fine-tuning on ImageNet they achieve an 85.9% top-1 accuracy. \n", "summary_of_the_review": "The method seems promising and the topic is important and timely. However, the paper failed to make convincing comparisons (e.g., to EsViT, or to a strong MAE baseline with a fair masking ratio and FLOP comparison). In addition, the paper did not present experiments that illuminated the effects of combining masking, denoising, and contrastive learning methods in general, but only in the context of their method, CAN, and when comparing to their own MAE implementation. Further, the paper lacked several important details, and as a result this reviewer lost confidence in the efficacy of the proposed method.", "strength_and_weaknesses": "Strengths\n- Combining the denoising diffusion objective with masked reconstruction in order to improve the prediction of high-frequency features without incurring significant additional computation cost is a good idea. This reviewer would be curious to see how much this approach benefits other SSL methods independently of the CAN method. \n- The experimental infrastructure is impressive -- the experiments required a lot of hardware time to run. \n\nWeaknesses\n- In my opinion, the primary weakness of this paper lies with its comparison to baselines. It’s unclear why the authors' implementation of MAE appears to differ so significantly from the original method (e.g., different masking ratio,  different image augmentations, different learning rate). For example, the MAE paper explicitly mentions that their method does not use color jitter since it degrades performance, yet in this paper the authors apply color jitter during pretraining. \n- It seems odd to report only the linear probe accuracy in Figure 1, without the finetuning performance, when including that information seems like it would paint a different (and more complex) picture. Further, as mentioned in [1], linear probing performance does not necessarily correlate well with transfer learning performance (i.e., finetuning well on a downstream task of interest is usually more valuable than linear separability of the learned features). And in addition, Figure 5 in the MAE paper [2] shows a significant difference in the linear probe performance vs the finetuning performance when masking rate is changed. They observed that linear probe accuracy varied quite significantly, while the finetuning performance was more robust to choice of hyperparameters. This reviewer is open to the possibility that their understanding is incorrect, but in that case phrases in the reviewed paper such as “In experiments our default masking rate is m = 50% unless explicitly stated otherwise.” are misleading. \n- The authors do not provide the source code (though they promise they will release it). This would have been useful to clear up any misunderstandings about their implementation of the MAE.\n- On page 8, the authors claim that “CAN performs particularly well under JFT-300M pre-training: it improves over SimCLR and MAE baselines in all 7 cases, often by significant margins”. However, their results in Figure 5 do not support this claim. MAE appears to have outperformed CAN in 3 of the 7 cases when pre-trained on JFT-300M and finetuned on IN-Real, IN-Sketch, and IN-V2. \n- Table 2 and 3 seem to be a bit misleading, or at least the work is lacking some citations. While the comparison to exemplar methods of these SSL families is important, acknowledgement of the other methods is also important, especially if comparing to the “state of the art” on ImageNet linear probe. For example, [3] is an ICLR 2022 paper  that shows several methods with comparable FLOP and parameter counts to CAN, but that achieves better linear probe performance. More specifically, EsVIT with the Swin-B architecture achieved over 80% on the ImageNet linear probe task, and the EsViT with Swin-T has only 28 M parameters and achieved 78.1% linear top-1 acc on ImageNet, whereas the best reported method in Table 3 achieved 78.2%, and CAN achieved 76.2%.\n\n[1] Exploring Simple Siamese Representation Learning https://arxiv.org/pdf/2011.10566.pdf\n[2] Masked Autoencoders Are Scalable Vision Learners https://arxiv.org/pdf/2111.06377v2.pdf\n[3] EFFICIENT SELF-SUPERVISED VISION TRANSFORMERS FOR REPRESENTATION LEARNING https://arxiv.org/pdf/2106.09785.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Pdi4j8qX3IF", "reviewer": "Reviewer_JaaX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes CAN, a simple, efficient, and scalable method for self-supervised learning of visual representations, which combines contrastive learning, masked autoencoders, and noise prediction approach used in diffusion models. The learning mechanisms are complementary to one another. Extensive empirical studies on linear evaluation, finetuning, transfer learning, and robustness demonstrate that the proposed method achieves strong downstream performance.", "summary_of_the_review": "See Strength And Weaknesses", "strength_and_weaknesses": "Strength:\n+ Combining contrastive learning, masked autoencoders, and noise prediction is novel.\n+ Extensive empirical studies on linear evaluation, fine-tuning, transfer learning, and robustness demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n+ JFT-300M is not accessible to everyone. Thus, ImageNet-1K or ImageNet-22K is more suitable for researchers to conduct fair comparisons. It seems the performance of CAN is not superior to concurrent work CAE or even MAE in terms of fine-tuning accuracy.\n+ Lack of full ablation studies for each component.\n+ For the denoising part, how is MLP designed? Why is the target noise? How could you ensure the network could learn high-frequency information as claimed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "NsicvJXyyF", "reviewer": "Reviewer_sdad", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new approach called CAN that combines contrastive learning and masked autoencoders, and also adds denoising diffusion-inspired loss to learn visual representations. Experiments are performed over Imagenet where the model is compared and surpassing leading self-supervised approaches like MAE and SimCLR. The approach is also much more computationally efficient than SimCLR.", "summary_of_the_review": "That’s a great paper with a novel and simple idea that combines proven modern approaches and shows strong results across multiple empirical dimensions (accuracy, efficiency, few-shot learning, robustness under distribution shifts) compared to leading methods on an important task of self-supervised visual learning. I recommend acceptance.", "strength_and_weaknesses": "**Strengths**: \n* **Combining complementary approaches**: the paper combines contrastive learning with masked auto-encoding to achieve complementary benefits of global and local signals. In high-level it’s similar to BERT two losses at the sentence level and at the masked-word level. That’s a great simple idea that that paper shows to work well. The usage of both noise prediction and auto-encoding also intuitively seem to play well together, where the former captures high-frequency features while the latter captures lower ones. \n\n* **Model Technical details**: I also like the specific technical manner in which the ideas are combined and find it an conceptually appealing approach (like a form of function composition): two masks are produced and then the two partial views are compared using contrastive learning. If the representations are good, then two partial views of the same image should be close to each other compared to other images.\n  \n* **Novelty**: The ideas explored in the paper are novel and simple. It combines existing general methods (contrastive learning, masked autoencoding) in an original manner making them complement each other. See more details below.\n\n* **Simplicity**: The way the two approaches are combined is simple and elegant, and so given the good empirical results, more likely to be broadly adopted by the community. In such cases, simple ideas are preferable in my opinion than more complex combinations as is the case with other works that explore manners to combine these approaches. \n\n* **Extensive set of suitable experiments**: multiple experiments are presented over both ImageNet and larger-scale JFT datasets, and ablations are also explored. The contribution of each loss is being explored and it looks like each of them contributes to a comparable degree. The subject of efficiency is also explored in its own section, although extending it further with additional experiments could be useful. Finally, the paper also explores the model’s robustness under distribution shifts. Additional experiments in the appendix explores additional transfer learning experiments. \n\n* **Strong results and suitable baselines**:  strong results on the imagenet linear probing compared to leading approaches such as MAE and SimCLR. It also had a lot better computational efficiency than SimCLR.\n\n**Weaknesses**: \n\nI don’t identify fundamental weaknesses. See some suggestions for writing and presentation improvement below. Smaller improvements:\n* **Limitations and future directions discussion**: Would be good if the authors could discuss any limitations they recognize of the work and potential directions for next steps! \n* **Qualitative results and visualizations**: could be good to show any sort of qualitative results and some image samples, or empirical visualizations of the model. \n* **More baselines**: Comparing to additional baselines will also be useful to corroborate the paper findings. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject