Energy-Inspired Self-Supervised Pretraining for Vision Models
Authors: Ze Wang
Jiang Wang
Zicheng Liu
Qiang Qiu
Venue: ICLR 2023 notable top 25% (2023)

Abstract: Motivated by the fact that forward and backward passes of a deep network naturally form symmetric mappings between input and output representations, we introduce a simple yet effective self-supervised vision model pretraining framework inspired by energy-based models (EBMs). In the proposed framework, we model energy estimation and data restoration as the forward and backward passes of a single network without any auxiliary components, e.g., an extra decoder. For the forward pass, we fit a network to an energy function that assigns low energy scores to samples that belong to an unlabeled dataset, and high energy otherwise. For the backward pass, we restore data from corrupted versions iteratively using gradient-based optimization along the direction of energy minimization. In this way, we naturally fold the encoder-decoder architecture widely used in masked image modeling into the forward and backward passes of a single vision model. Our framework accepts a wide range of pretext tasks with different data corruption methods, and permits models to be pretrained from masked image modeling, patch sorting, and image restoration, including super-resolution, denoising, and colorization. We support our findings with extensive experiments, and show the proposed method delivers comparable and even better performance with remarkably fewer epochs of training compared to the state-of-the-art self-supervised vision model pretraining methods. Our findings shed light on further exploring self-supervised vision model pretraining and pretext tasks beyond masked image modeling. 

Published as a conference paper at ICLR 2023
ENERGY-INSPIRED SELF-SUPERVISED PRETRAINING FOR VISION MODELS
Ze Wang' Jiang Wang Zicheng Liu-
Qiang Qiu
Purdue University *Microsoft Corporation (zewang, sqiu/epurdue.edd {jiangwang, 21 alu aliujemic crwsoft.comm
ABSTRACT
Motivated by the fact that forward and backward passes of deep network naturally form symmetric mappings between input and output representations, we introduce a simple yet effective self-supervised vision model pretraining framework inspired by energy-based models (EBMs). In the proposed framework, we model energy estimation and data restoration as the forward and backward passes of single network without any auxiliary components, e.g., extra decoder. For the forward pass, we fit a network to an energy function that assigns low energy scores to samples that belong to an unlabeled dataset, and high energy otherwise. For the backward pass, we restore data from corrupted versions iteratively using gradient- based optimization along the direction of energy minimization. In this way, we naturally fold the encoder-decoder architecture widely used in masked image mod- eling into the forward and backward passes of single vision model. Thus, our framework now accepts wide range of pretext tasks with different data corruption methods, and permits models to be pretrained from masked image modeling, patch sorting, and image restoration, including super-resolution, denoising, and coloriza- tion. We support our findings with extensive experiments, and show the proposed method delivers comparable and even better performance with remarkably fewer epochs of training compared to the state-of off-hhe-rrt art ceffssuper vision model pretraining methods. Our findings shed light on further exploring self-supervised vision model pretraining and pretext tasks beyond masked image modeling.
INTRODUCTION
The recent rapid development of computation hardware and deep network architectures have paved the way for learning very large deep networks that match and even exceed human intelligence on addressing complex tasks (Brown et al. 2020 He et al.) 2017 Silver et al. 2016). However, as annotating data remains costly, leveraging unlabeled data facilitate the learning of very large models attracts increasing attention. Exploiting context information in the massive unlabeled data in natural language processing (NLP) stimulates Chen al. (020aa) to use the direct modeling of pixel sequences as the pre-text tasks of vision model pretraining. Recent self-supervised vision model pretraining through masked image modeling (MIM) (He et al., 2022 Wei al., 202 Xie et al. 2022) typically adopt an auto-encoder (AE) architecture, where the target vision model to be pretrained serves as encoder to encode an image with incomplete pixel information
latent representation. An auxiliary decoder is jointly trained to restore the missing information from the latent representation. Contrastive self- supervisedd learning methods (Chen al.) (2020b) usually require large training batch sizes to provide sufficient negative samples. Recent Siamese network based self-supervised lcarning methods (Grill et al., 2020 Chen & Hc 2202 Tian al. [202] He (ct al. 2020 Chen et al. 2021)) alleviate the huge batch challenge by deploying an momentum copy of the target model to facilitate the training and prevent trivial solutions. VICReg (Bardes et al.
2022) prevents feature collapsing by two explicit regularization terms. Barlow Twins (Zbontar et
2021 reduce the need of large batch size or Siamese networks by proposing a new objective based on cross-correlation matrix between features of different image augmentations.
In this paper, we make a further step towards the following question: Can we train a standard deep network do both representation encoding and masked prediction simaltaneously, so that no auxiliary components, heavy data augmentations, or modifications to the network structure are required?

---

Published as a conference paper at ICLR 2023
Hinted by the fact that the forward and the backward passes of a deep network naturally form symmetric mappings between input and output representations, we extend the recent progress on energy-based models (EBMs) (Xie et al. 20T6, Du & Mordatch| [019 Du et al. 2020b] Zhao et al., 2017) and introduce a model-agnostic self-supervised framework that pre-trains any deep vision models. Given an unlabeled dataset, we train the forward pass of the target vision model to perform discriminative recognition. Instead of instance- wisee classification as contrastive self-supervised learning, we train the target vision model to perform binary classification by fitting it to an energy function that assigns low energy values to positive samples from the dataset and high energy values otherwise. And we train the backward pass of the target vision model to perform conditional image restoration in masked image modeling methods, by restoring positive image samples from their corrupted versions through conducting traseent-based updating iteratively along the direction of energy minimization. Such conditional sampling schemes can produce samples with satisfying quality using as few as one gradient step, thus prevents the unaffordable cost of applying the standard implicit sampling of EBMs on high- figh-dimensional data. In this way, we naturally fold the incoderddecode architecture widely used in masked image modeling into the forward and backward passes of a single vision model, so that the structure tailored for discriminative tasks is fully preserved with no auxiliary components or heavy data augmentation needed. Therefore the obtained vision model can better preserve the representation discriminability and prevent knowledge loss redundancy. Moreover, after folding the corrupted data modeling (encoder) and the original data restoration (decoder) into a single network, the proposed framework now accepts broader range of pretext tasks to be exploited. Specifically, we demonstrate that beyond typical masked image modeling, the proposed framework can be easily extended to learning from patch sorting and learning from image restoration, e.g.. super-resolution and image colorization.
We demonstrate the effectiveness of the proposed method with extensive experiments on ImageNet- IK. It is easy notice that almost every parameter trained from the self-supervised training stage will be effectively used in the downstream fine-tuning. And we show that competitive performance can be achieved even with only 100 epochs of pretraining on a single 8-GPU machine.
RELATED WORK
Vision model pretraining, Pretraining language Transformers with masked language modeling (Kenton& & Toutanova 2019) has stimulated the research of using masked image modeling to pretrain vision models. BEIT (Bao et al. (02I)) trains the ViT model to predict the discrete visual tokens given the masked image patches. where the visual tokens are obtained through the latent code of discrete VAE (Ramesh et al.] (02T). iBoT (Zhou et al.) 2022) improves the tokenizer with online version obtained by a teacher network, and learns models through self-distillation. Masked auto-encoder (He et al.) 2022) adopts an asymmetric encoder- decoder architecture and shows that scalable vision learners can be obtained simply by reconstructing the missing pixels. (Wei et al., 2021) studies empirically self-supervised training through predicting the features, instead of the raw pixels of the masked images. Different forms of context information for model pretraining are also discussed by learning from predicting the relative position of image patches (Doersch et al. 2015), sorting sequential data (Noroozi & Favaro 2016), training denoising auto-encoders ( (Vincent et al. 2008), image colorization (Zhang et al. 2016), and image inpainting (Pathak et al. (016))
Similar to metric learning (Hinton) (202)). contrastive seff-supervieed learning methods learn visual representations by contrasting positive pairs of images against the negative pairs. (Wu et al. 2018
adopts coise-contrastive estimation to train networks to perform instance- classification for
feature learning. Recent methods construct positive pairs with data augmentation (Chen et al.) 2020b). )
and obtain pretrained models with high discriminability (Caron et | 2021). l). To relax the demand of large batch size for providing sufficient negative samples, (He et al.] 2020; Chen et al. 2020c) are proposed to exploit supervision of negative pairs from memory queues. And it is shown that self-supervised learning can even be performed without contrastive pairs (Grill et al., [2200] Chen &
superving
He 2001] Tian et al.] 2021) by establishing dual pair of Siamese networks to facilitate the training. (Donahue & Simonyan 2019) extends unsupervised learning with generative adversarial networks to
learning discriminative features.
Energy-based models. The proposed framework for vision model pre-training is inspired by the progress of energy-based models (LeCun et al. 2006). As a family of generative models, EBMs are
mainly studied to perform probabilistic modeling over data (Ngiam al. 2011 Qiu et al. 2019
Nijkamp al. 20200 Du & Mordatch] 2019 22019, Du et al. 2020b] /Zho et 2020 Xie et al., 2016
2017
2020 2021
Xiao et al. 2020
Arbel et al. 2021) 1), and conditional sampling (Du al. 2020a 2021
2

---

Published as a conference paper at ICLR 2023
Energy minimization
Energy minimization
Step 40
Step
Step
Step
Figure I: Typical EBM sampling demands long chains even with a mild resolution of 64 x 64 (left). Our conditional sampling with short chains obtain satisfactory results with few single gradient step at a standard resolution of 224 224 (right).
is shown in (Grathwohl et al. 2020 that a standard discriminative classifier can be interpreted as an EBM for the joint data-label distribution, which can then by exploited to learn from unlabeled data in an semi-supervised manner. Recently, the idea of EBMs is being applied to more applications including reasoning (Du et al. 2022). , latent space modeling of generative models (Pang et al. 2020), and anomaly detection (Wang et al. 2022 Dehaene al. 2020). To the best of our knowledge, we are the first to apply baaey model training to self-supervised vision model pre-training.
METHOD
In this section, introduce in details the proposed framework of energy- energy-inspired self self--uper vi- sion model pretraining. We first briefly review the backgrounds of energy-based models in Section We present the general process of the proposed pretraining framework, with straightforward example based on mask image modeling in Section 3.2 We then present how the proposed framework allows extensions to wide range of variants adopting different pretext tasks with examples of learning from image restoration (Section 3.3) and learning from sorting (Section 3.4).
3.1 BACKGROUNDS
Being mainly generative models, EBMs are usually trained to model a target distribution density function. EBM training is typically achieved by learning energy function that predicts the unnormalized density, named the energy score, for given data sample. Specifically, given data sample x E IRd, the energy function E (x) IRd - R, with as the learnable parameters, maps the sample its energy score, which expected be low for the in-distribution (positive) samples, and high for the out-of-distribution (negative) samples. The modeled data density pa pa(x) is expressed as:
Po exp(- -F (x))
(1)
where Zo fx exp(-EE is the partition funetion. Approximating target data distribution Paaaa (x) equals to minimizing the expected negative log-likelihood function over the data distribution, defined by the maximum likelihood loss function:
Lxa Exxpunn log Po Px)] = Ex-pes (x)(x) + log Zo).
(2)
As the computation of LML involves the intractable Za. the common practice is to represent the gradient of LML. as,
VoLm. I Vo Ex- )}
(3)
The objective in train the model LEg to effectively distinguish in-domain and out-of-domaal samples by decreasing the predicted energy of positive data samples x from the true data distribution and increasing the energy of negative samples x obtained through sampling from the model po-
Sampling from the modeled distribution equals to finding the samples with low energy scores. Parametrizing the energy function as deep neural network allows for continuous energy space to be learned from data, where sampling can be accomplished by randomly synthesizing a negative sample of high energy, and moving in the corresponding energy space along the direction of energy minimization. Inspired by MCMC based sample techniques such Langevin dynamics (Welling &
Teh 2011), common practice (Du & Mordatch 2019
Du al. 2020b resorts gradient-based

---

Published as a conference paper at ICLR 2023
Enagg prolittion and minimiztiion
Step
Applying the pro- posed framework to masked im- age modeling. The unlabeled image is corrupted with ran- dom patches, and the network is trained to recognize the cor- rupted sample as negative one with high energy, and recover the original image by updating the image iteratively along the direc- tion of energy minimization.
Figure 2:
Corruption
Ee
Rondom musking
Enengy prediction
Energ proditttion
Step
F
optimization for implicit sampling. Specifically, by performing N gradient steps, the approximated optimum xN can be obtained as
(4) where a is the step size of the gradient-based optimization. In practice, the noise term w is usually set to smaller empirical scale in the official implementation of (Du et al., 2020b) for faster sampling. x² is usually obtained by sampling from a predefined prior distribution such as Uniform. For more comprehensive formulation of implicit generation with energy-basee based models, please refer to (Xie et al. 2016 Du & Mordatch 20199] Du al. 2020b).
x - tw", w" ~ NN(0,20)) 20), n n= 1.
3.2 PROPOSED FRAMEWORK
We denote the deep vision model be pretrained as sb Our energy meeggyinspirred model can be constructed by simply appending a linear head / with single output dimension to the feature extractor, i.c., Ex(x) = hx)) with collectively denoting the parameters of both & and h. In a typical setting, the linear head h contains only hundreds of parameters, After the pretraining, the obtained vision model can be directly used as an image recognition model by only replacing the linear head h. The full preservation of network architecture with no auxiliary network components, e.g., a decoder, to be removed, better maintains the network discriminability and prevents potential feature redundancy.
As illustrated in Figure T even using low resolution, the typical implicit sampling of EBMs in (4) can take dozens or even hundreds of gradient steps produce an image sample of satisfying quality (Du Mordatch 2019; Zhao et al. 2020). Applying the standard EBM training to self-supervised pretraining introduces unaffordable cost. It is discussed in (Gao et al. 2021) that reformulation of the training objective based on recovery likelihood can stabilize the training of EBMs. this paper,
inspired by (Wang et al. 2022), we forgo the from-scratch sampling and train the network to perform conditional sampling, so as to restore partially corrupted data with explicit supervision. As visualized in Figure [ the costly noise-to-image sampling of EBMs is now replaced with conditional sampling, where chain of sampled data moving towards the low-energy region are obtained for each corrupted sample rapidly. In our case of self-supervised learning, doing so has two major advantages: Firstly, as being further discussed in Section 4.3 the proposed framework now allows the restoration of each sample to be completed with as few as two gradient optimization steps, and permits desirable speed for self-supervised training on large scale datasets. Moreover, such conditional sampling allows to replace the contrastive divergence (3) designed for unconditional sampling by explicit supervision with pixel values training according to our observations. The proposed framework imposes little restrictions to the image sample corruption methods deployed and permits a wide range of pretext tasks to be exploited. For the sake of discussion, we present in details one straightforward variant with masked image modeling walk through the training process, and illustrate other possible variants in later sections.
will discuss later, and such strong supervision alleviates the unstable EBMs
Masked image modeling. As visualized in Figure 2 given a batch of image samples fx; h we first corrupt each image using a predefined function + (-). In this example, 1 (-) denotes random image masking. After image masking, (x,) can be seen as a sample that is out of the target data distribution Pdara with the remaining pixels inferring the original contents of the image. With the target modeling a continuous energy function, we can perform online evaluation to the estimated energy function by examining how well moving the masked image in the modeled energy space
4

---

Published as a conference paper at ICLR 2023
along the energy minimization direction can restore the original data x, Specifically, resort the gradient based optimization (4) and perform --step image restoration with x = (x,). The loss of the restoration steps can then be expressed as:
1 KN EN MSE(X/ x.), where X - aV (SG(X ))
C
(5)
with SG denoting the stop gradient operation that blocks the gradient propagation across steps. We empirically observe that adding stop gradient operations between consecutive steps helps accelerate
the training speed and convergence. C here encourages original images to be restored from the negative images (corrupted versions and the sampled versions along the sampling chains of ( by gradient based updating along the direction of energy minimization, which equally encourages higher energy values for negative images, and can functionally replace the second term in (3). The supervision in (5) is similar the ones used in score matching and diffusion models (Vincent 201 I Ho et al. 2020). One major different is that all the inputs for the intermediate steps in our method are obtained by the previous step of restoration, instead of generated using the original images based on some noise schedulers.
Notably, as discussed in (Du & Mordatch 2019), standard EBM training with (3) using arbitrary energy model can cause sharp changes in gradients, and the stable training requires heavy tuning to the hyperparameters and techniques like spectral normalization to constrain the Lipschitz constant of the network. While in our framework, unstable training caused by sharp gradients is naturally prevented by the explicit supervision in (5), as faithfully restoring the original data requires the gradient in (5) to be bounded within certain range. We summarize the overall training steps of the We further provide Py PyTorch-styla pseudo code in Appendix Section to facilitate reproducing our results.
proposed framework in Algorithm [
Algorithm 1 Energy-based self self-supervisee vision model pretraining.
1: Given: A target network z/ to be pretrained, a large-scale unlabeled dataset (x,), and an image sample corruption function + (.).
2: Given: Step size and number of steps N for the gradient update of corrupted samples.
3: Initialize the target network &b and the linear head h.
4: repeat
Sample batch of images from the unlabeled dataset.
6: Corrupt each sample and initialize the conditional sampling chains as x² =1 (xx).
7: for Step n = 1: do
8:
Stop gradient x 9: Perform gradient update to the corrupted samples as in (5)
SG
10: end for
11: Compute the restoration error of each step using (5) and update up and with gradient optimization.
12:
until Converge
13:
Return $b.
3.3
BEYOND MASK IMAGE MODELING
Recent self-supervised vision model pretraining methods (Xie et al. 2022, He et al. 2022 Wei et al. 2021) invariably adopt masked image modeling as the pretext task. We argue that the encoder-decoder architectures used in these methods prevent them from being easily extended to other pretext tasks. In the auto-encoder based methods, the vision model be pretrained serves as the encoder, and is only exposed with the corrupted images during pretraining. Therefore, it is important to present part of the original image patches to the encoder, so that the encoder can learn from those intact patches network weights that transfer well in downstream finetuning. While in the proposed pretraining framework, both corrupted samples and original samples are exposed to the target vision model, in the forms of input and supervision, respectively. Specifically, by simply replacing the corruption establish wide range of pretext variants that learn vision models from, such
function + (-), as patch sorting, super-resolution denoising, and image colorization. Further details and results will be discussed in Section 4.1 With certain degrees of global image corruption, networks can be trained to infer possible content given the incomplete pixel information, and restore the missing information, such as detailed textures or color, by the patterns learned from the true data and stored in the network weights. With the restriction the corruption methods being lifted, our framework stimulates further discussions on the pretext tasks of vision model pretraining. Patch sorting is discussed next as an example of new pretext tasks.
5

---

Published as a conference paper at ICLR 2023
3.4 LEARNING FROM SORTING
Sorting the patches of an image according to the spatial position requires inferring the global content by integrating the local information contained in each patch, and sorting the order of the patches accordingly. Such process involves both local feature extraction and global semantic inference, therefore can be an useful pretext task of self-supervised training. However, restoring the patch orders in the image pixel space can be extremely challenging to learn. MP3 (Zhai et al.) 2022) extends mask image modeling to position predictions by dropping tokens in the values of the self-attention layers and predicting the corresponding position using the extracted features. Thanks to the absolute position embedding widely adopted in ViTs, we present interesting variant of learning from sorting that does not modify any intermeidate layers of ViTs.. Specifically, the feature extraction of a ViT tb can be expressed as:
(6) where Ør and op denote the stacked transformer layers and the patch embedding layer, respectively.
(x,)) (T (x; (r. g)) PE[p, ()pp-...-))
We use p and to index the image patch and Pł[p, g] is the corresponding position embedding. Adopting the simple non-learnable sin-cos function as the position embedding, we can shuffle the image patches by simply shuffling the position embedding, and train the target network to sort the patches by performing gradient-based optimization to the shuffled position embedding along the direction of energy minimization. Specifically, based on (6) and omitting the indexes, we define the new energy function parametrized by the target vision model as Eo(x;, PE)), and train the network using the following loss
N ΣΣ
K
Csort KN 1- 1=
MSE(PE PE), where PEC PE Ex, SG((EE
(7)
Learning from sorting corrupts only the position embedding, and allows the original image signal to be fully exposed to the network. And the network encouraged to infer the global structure of an image from the features of patches and sort the patches to form a semantically meaningful pattern.
Random edge masking in the patch embedding layer. Note that for most natural images, two neighboring patches may share nearly identical pixels at the edge rows or columns. While such information hard to be exploited by human for patch sorting, is can be easy for a network to learn such trivial sorting solution, and perform nearly perfect patch sorting without resorting to actual semanties. Such trivial solution can be easily avoided by random masking the weights in the patch embedding layer and preventing the patch embedding layer from learning only the edge pattern of image patches. We discuss the detailed implementation of the edge masking in Appendix Section AA.2 Such trivial solution can also be resolved by randomly masking tokens in vision transformers as in (Zhai et al. 2022).
4
EXPERIMENTS
With the standard ImageNet-IK, IK dataset, we show that the proposed EBM pretraining framework can help deep vision model to achieve competitive performance with as few as 200 epochs of training. We use ViT to conduct most of the experiments, And we further show in Appendix Section B.3 that as a model-agnostic framework, the proposed method can be seamlessly extended other architectures.
Training. We use Adam W (Loshchilov & Hutter 2019) as the optimizer for both sulf-super training and tuning. For all the self- pretraining experiments, we adopt only random cropping and random horizontal flipping as the data augmentation. We present comprehensive training details in Appendix Section Table Tablle A Most of the experimental settings follow (He et al., 2022). Unlike recent methods (Zhou et al. 2022, He et al. 2022). we do not perform exhaustive searches for the optimal hyperparameters such as learning rates. Training energy functions introduces a new hyperparameter a, which is the step size of the gradient optimization to the corrupted data. Thanks to the explicit supervision available in the proposed framework, we can set to be learnable, and jointly train with the network without the concern of training stability as in standard EBM training. If not otherwise specified, we adopt N = 2, i.e., two steps of gradient- based energy minimization in the pretraining stage for the best performance-efficinng trade-off.
4.1 SELF COMPARISONS
As discussed in Section 3 the proposed framework accepts a wide range of variants with different pretext tasks. To illustrate the fiexibility, present results with different variants including learning

---

Published as a conference paper at ICLR 2023
Table 1: Masked image modeling with different patterns and ratios of image masking. The result of MAE (He) et al., (022) with 400 epochs is based our reimple- mentation. The results of our methods are obtained by 100 epochs of pretraining. All results obtained with 100 epochs of finetuning. Baseline results are in gray. From scratch indicates the purely supervised baseline.
Table 2: Results obtained by different pretext tasks of learning from image restoration and patch sorting. Baseline results are in gray.
Methods
Accuracy
From scratch
76.6
AE SR 16x
77.1
Masking strategies
Accuracy
AE denoising
76.8
Denoising
79.2
From scratch
76.6
SR 8x
77.1
Random large
79.7
SR 14x
78.2
Random small
79.3
SR 16x
79.6
SR 24x
78.4
% of masking MAE
10%
30%
50%
70%
90%
SR 32x
76.3
78.3
Colorization
79.5
Gridded (16)
76.7
78.3
78.7
79.0
78.8
Gridded (24)
76.8
78.2
78.7
79.2
78.8
AE sorting
77.2
Gridded (32)
77.1
78.4
78.6
79.0
78.7
Patch soring
79.5
from masked image modeling, image restoration, and sorting. All results in this section are obtained by pretraining and finetuning ViT-S for 100 epochs the ImageNet-IK IK (Deng al. 2009) dataset.
Learning from masked image modeling. A straightforward way of implementing the proposed framework is to train the network to perform masked image modeling given incomplete pixel information. We present results obtained with different masking strategies and ratios of masking in T As visualized in Figure B in the experiments with gridded mask, we evenly divide an image into squared patches with the same size, and randomly mask out a portion of the patches. Note that in the Gridded (16) experiments, the patch partition in the image masking matches exactly with the patch partition in the ViT networks, therefore it is fair comparison against MAE (He et al. 2022). For the random masking experiments, we randomly place blank patches with the size and aspect ratio sampled from a particular range each image. In the Random small experiments, 75 blank patches with normalized sizes sampled from a Uniform distribution of (( 0.025). In the Random large experiments, we randomly place 25 blank patches with normalized sizes sampled from (00022, 0.05). For both experiments, the aspect ratio of each patch is sampled from (0.5, 2.0).
randomly place
Learning from image restoration. As discussed in Section 3.3 flexibility as the pretrained vision model is exposed with both true samples and artificial negative ones, thus when the input images are corrupted globally, our framework can still learn good models. To show this, we present in Table results obtained with learning from image restoration. Specifically, we train the network to learn from image resolutin,, denoising. and image colorization, where every pixel is corrupted with predefined function. Table [ SR denotes super- resolution. AE + SR 16 denotes a baseline experiment with auto-encoder architecture as in (He) al. 2022 In the s-time super-resolution (denoted as SR sx), the image are first downsampled using bicubic interpolation for times, and resized back to the original size using nearest- neighbrr interpolation. In the denoising experiments, we take a noise scheme inspired by diffusion models (Song et al. 2 et al. 2020) with 4 (x) / 7e, with (0, /) and 7 uniformly sampled as ~ (0.
framework enjoys higher
As shown in the quantitative results in Table and visualizations in Figure 3 with proper degrees of corruption, restoring the original images may require the network to infer the general content given the corrupted pixels, and recover the details using the knowledge learned from the true samples and stored in the network weights. For example, in the image colorization experiments, the pretrained vision model learns the common colors of different objects from the massive unlabeled data in self-super way. As visualized in Figure the vision model learns common knowledge such as stop signs are usually red, and the background of horse is usually green while manatees are marine mammals therefore the background is usually blue. Summarizing such knowledge requires the vision models to learn identifying objects first, therefore transferable network weights and feature representations can be obtained from pretraining. And shown in the 'Denoising' row of Figure 3 with strong noise injected to the input, the model is able to recover objects that are almost invisible. This finding potentially connects our pretraining method with genitive models (Song al. 2021a a Ho
7

---

Published as a conference paper at ICLR 2023
Gridded masking (16)
Giridded masking (32)
Kandom masking
Randon smaill maskiigg
SR 12x
16x
Denoising
Colorization
STOP
STOP
STOP
Corruptdd
Original
Restored
Corrupted
Original
Restored
Corrupted
Origiaa
Restored
Comupied
Original
Restored
Conditional sampling with masked image modeling with different masking strategies and learning from image restoration. The proposed framework accepts broader range of pretext tasks.
Figure 3:
Table Quantitative comparisons against the recent self- supervised model pretraining methods. * denotes results produced by our re-implementation. PT and FT denote pretraining and finetuning, respectively. All ImageNet results are evaluated the validation with single center crop 224 x 224 for each image. denotes the training involves external dataset other than ImageNet- 1K. For our results, we set e = 50 for ViT-L, e = 100 for ViT-B, and 200 for ViT-S. We use N for 200-epoch experiments and N = otherwise.
Table 4: ViT-S training efficiency with number of epochs. Results of MAE (He lt al.] (022) are obtained with our re- implementation.
Methods
Epochs
Accuracy
MAE
400 + 100
78.3
MAE
400+200
80.2
Ours
100 100
79.7
Ours
100
81.0
Methods
(PT FT)
ViT-S
VIT-B
VIT-I.
From scratch
300
79.6"
82.3
82.6
Table 5: Efficiency comparisons with
DINO
82.8
GPU-hours. t denotes numbers obtained by Zhou et al. 2022).
MoCo-V2 V3
300+150
83.2
84.1
BEiT!
800+100
83.2
85.2
MaskFeat
300+100
83.6
85.7
Methods
GPUs x H
Acc.
iBOT
600 200
81.4
MoCo-V3
128 24
83.2
iBOT
1600 + 100
83.8
MP3
100 (150) 100
83.0
83.6
BETT!
16 x 90
81.4
MSN
600 100
83.4
DINO
16 x 112
81.6
MAE
400 + 100
78.3"
83.1"
iBOT
16 x 193
82.0
MAE
1600 +
83.6
85.9
MAE (400+100)
x 72
83.1*
Ours Sorting
800 + 100
83.2
MAE (1600+100)
8 288
83.6
Ours Mixed
200 +
81.2
83.1
Ours
8 232
83.4
Ours Mixed
800 +
81.8
83.4
85.4
Song & Ermon 2019 Song al. 2021bb. We will further investigate the connections in future efforts. In our method, corrupted images and original image are exposed to the input layers of the vision model input and supervision, respectively. Therefore, compared the auto-encoder based baseline, which only receives corrupted images as input, the proposed framework demonstrates clearly better performance after finetuning.
al.
2020
8

---

Published as a conference paper at ICLR 2023
Learning from sorting. To prevent the trivial solution of sorting as discussed in Section 4
adopt regularization schemes that prevents the network from simply sorting the patches based on
the edge pixels only. Details of the regularizations can be found in Appendix Section A.2 For
fair comparison, we conduct a baseline experiment of adopting an auto-encoder network to directly
predict the position of each patch. We follow the details presented in MAE (He et al.)
2022) and
implement an asymmetric auto-encoder structure with a lightweight decoder. Note that the baseline implementation, there is no position embedding used in the encoder ViT, and we add back trainable position embedding initialized with full zeros in the finetuning stage for fair comparisons. The quantitative comparisons are in the bottom rows of T All numbers are obtained with the same settings. We apply the same regularization schemes to the baseline method prevent trivial solution. The proposed method learns better features. And the discrepancy between pretraining and finetuning caused by the position embedding in the encoder of the AE baseline may be an important reason of its worse performance.
4.2 QUANTITATIVE COMPARISONS AGAINST RECENT METHODS
In this section, we present quantitative comparisons against the recent self-supervised model pretrain- ing methods. We train our method using a mixture of pretext tasks that are uniformly sampled from image masking, super-resolution, denoising. and colorization. In Table 3 we compare our method against DINO Caron et al. 202T), M3 (Chen et al. (02T). MaskFeat (Wei et al. 202T), BEiT (Bao et al., 2021D, iOOT (Zhou al. 2022). MP3 (Zhai et 2022). Masked Siamese Networks (MSN) (Assran et al. 2022). and MAE (He et al. 2022). We train our model with a mixture of all the pretext tasks discussed above. We do this by randomly sample a correction method for each image in a batch. With only 200 epochs of pretraining, the proposed framework can achieve comparable or even better performance with the state-of-the-ar self-supervised pretraining methods, some of which adopt much more epochs and leverage external data for training.
4.3 EFFICIENCY DISCUSSIONS
In this section, we present discussions on the training efficiency of the proposed method. As discussed in Section 3.2 in the proposed framework, the network learns to model the density with samples from real data distribution and sampled negative. Therefore, compared to other masked image modeling based pretraining methods that learn from a relatively small part of the images in each iteration, the proposed framework delivers comparable performance with fewer epochs of training. Note that when using each training iteration in our approach involves N forward passes and N - backward passes with additional backward pass for the gradient of parameters. We present efficiency comparisons with number of epochs in Table [4] The proposed method can achieve better results with fewer epochs of training compared to MAE (He et al. 2022). We further present comparisons on training efficiency with GPU-hours in Table S The proposed method demonstrates a good performance-efficieee trade-off compared to the state-of-the-arr methods. We present in Appendix Figure A performance obtained with different N steps of updating.
5 CONCLUSION
We presented training and trained the vision model to perform conditional sampling initialized from corrupted samples by moving them along the direction of energy minimization. The bi-directional mappings between images and latent representations are modeled naturally by the forward and backward passes of a network, which fully preserve the discriminative structure of the target vision model and avoid auxiliary network components and sophisticated data augmentation to facilitate pretraining. We presented extensive experiments with different pretext tasks, including learning from masked image modeling, learning from image restoration, and learning from sorting. We hope our findings can shed light on further exploring the pretext tasks of self-supervised vision model pretraining.
self-supervised pretraining for vision models. We accelerated EBM
Limitation. While strong finetuning results are observed, our method does not directly provide features that are strongly linearly-separable, which is reflected by lower linear probing accuracy compared to contrastive learning based pretraining methods. This phenomena is also observed and discussed in (He et al. 2022). and may be attributed to the fact that both (He et al. 2022) and our method do not explicitly encourage linear separation of features in the pretraining stage as the contrastive learning based method do. And linear probing cannot faithfully validate strong but non-linear features. We present quantitative comparisons in Appendix Section B.4 and will keep improving the linear separation as direction of future effort.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. ICLR, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. arXiv preprint, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 0844042,00.,,224 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variune-iinarranc---vvaannn regularizaton for self-supervised learning. ICLR, 2022.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005.1416. 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervvid vision transformers. ICCV, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In JCML, 2020a.
Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICMI., 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2 2021.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders discriminators rather than generators. ICLR, 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
David Dehaene, Oriel Frigo, Sébastien Combrexelle, and Pierre Eline. Iterative energy-based projection on normal data manifold for anomaly localization. ICLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. NeurlPS, 2019.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. NeurIPS, 2019.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. NeurIPS, 1556637-6647, 2020a.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. ICML, 2020b.
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. NewrlPS, 34:15608-15620, 2021.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In ICML., pp. 5570-5582. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2023
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy based models by diffusion recovery likelihood. ICLR, 2021.
Will Grathwohl, Kuan-Chieh Wang. Jörn- Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. ICLR, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Berarro Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. NeurlPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 2002.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurUPS, 33: 6840-6851, 2020.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy-based learning. Predicting structured data, 1(0), 2006.
7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. CVPR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, ICLR, 2019.
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models, In ICML, 2011.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In AAAI, 2020.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016.
Bo Pang, Tian Han, Erik Nijkamp. Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. NeurlPS, 33:21994- 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
11

---

Published as a conference paper at ICLR 2023
Yixuan Qiu, Lingsong Zhang, and Xiao Wang. Unbiased contrastive divergence algorithm for training energy-based latent variable models. In ICLR, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint 2rXiv:2/22./209: 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwisser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 2016.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR. 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021b.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-superrised learning dynamics without contrastive pairs. ICML, 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 2011.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. ICML, 2008
Ze Wang, Yipin Zhou, Rui Wang, Tsung- Yu Lin, Ashish Shah, and Ser-Nam Lim. Few-shot fast-adaptive anomaly detection. In NeurlPS, 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:21 /2.09133, 2021.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In ICML. Citeseer, 2011.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. ICLR, 2020.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In ICMI., 2016.
Jianwen Xie, Song- Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by -pappal-temporal generative convnet. In CVPR, 2017.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Generative voxelnet: learning energy-based models for 3d shape synthesis and analysis. IEEE T-PAMI, 2020.
Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification. In CVPR, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. CVPR, 2022.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
12

---

Published as a conference paper at ICLR 2023
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny, Barlow twins: Sulf-supervised learning via redundancy reduction. In ICML, 2021.
Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Yitan Cheng, Walter Talbott, Chen Huang, Hanlin Goh, and Joshua Susskind. Position prediction as an effective pretraining strategy. ICML, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In ECCV, 2016.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. ICLR, 2017.
Yang Zhao, Jianwen Xie, and Ping Li. Learning energy-based generative models via coarse-to-fine expanding and sampling. In ICLR, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pp. 633-641, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Ibot: Image bert pre-training with online tokenizer. ICLR, 2022.
13

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A.1 DETAILS ON TRAINING
We present the training details for both self-supervised training and finetuning in Table A All experiments are implemented using PyTTrcc (Paszke et al. 2019). We use the default API for automatic mixed-precision training. The step size can either be initialized as 0.1 and trained along with all the parameters in the vision model in an end-to-end fashion, or predicted by linear head given the extracted features. In practice, impose positive constraint the value of a during training. Following standard practice, we use an image resolution of 224 x 224 in all experiments.
Configurations
Pretraining
Finetuning
optimizer
AdamW
AdamW le-3 Cosine decay 0.05
base learning rate learning rate schedular weight decay
le-4 Cosine decay 0.05
momentum of AdamW
B1 0.9, 0.95
B1 0.9, 02 0.999 0.75
layer-wise Ir decay (Clark al. 2020
batch size
256
1024
drop path (Huang ct al. 2016
0.1
augmentation
RandomResizedCrop
RandAug (9, 0.5) (Cubuk al. 2020
label smoothing (Szegdyy al. 2016
0.1
mixup (Zhang al. 2018
0.8
cutmix (Yun et al. 2019
1.0
Mix-preciston trainig
Table A: Training details for both self-super pretraining and finetuning.
A.2 LEARNING FROM SORTING
In the experiments of learning from sorting, adopt the 2D sin-cos position embedding following the implementation of MoCo-V3 (Chen et al.] 2021). The embedding for each position remains fixed during the self-supervised pretraining stage. In the finetuning stage, initialize the position embedding using the same 21) sin-cos function, and allow the embedding to be trainable along with all the parameters in VIT.
Regularization. To prevent the trivial solution of sorting the patches based on only the edge pixels of image patches as discussed Section 3.4, we adopt two regularization methods the pretraining stage. Firstly, we adopt the random edge masking as presented Section 3.4 Specifically, for each image patch, randomly set the values of the k out-most pixels all zeros. We observe that satisfactory performance can be achieved by simply setting the probabilities of k: = and k: = 2 to 50% and 50%, respectively. To further improve the robustness and training speed, in each iteration, we randomly dropout 50% of the image patches before the patch embedding enters the Transformer layers. This simple patch drop-out scheme significantly reduces chance that neighboring patches entering the Transformer layers simultaneously. and prevents the network from learning to sort the patches simply based on edge pixels of patches.
A.3 PSEUDO CODE IN PYTORCH STYLE
model = VisienMode1
initialize deep vision model with any architectures
head Linear (in, channels=model..oo dim, out channels=l, bias=False) initialize a simple linear head for energy score prediction
7 criterion = SmoothLlLoss (beta=1.0)
$ define loss function for image reconstruction
optimizer AdamN poodeel.paramtter..
head. pawameters
initial ie parameter opt optimize
14

---

Published as a conference paper at ICLR 2023
13
training lcop
14
For images image_loader:
15
images with shape [n, C, h w]
16
ieruuteed_imagess aaruptico_methine immaees)
17
loss
20
for
in num steps:
21
corrupted_images - corrupted_images.coo detach ()
22
# stop gradients between inner-loop steps.
23
encrgy_score = head (model (cerrupted
# energy score with shape [n, 1]
im_grad autograd (energy_score sum (), corrupted_irager compute the gradient of input pixels along the direction of energy maximization zation
27
28
29
corrupted_images - corrupted_images alpha im_grad
# gradient descent along the direction of energy minimization
32
lcss criterion pncrrupted_imageee images)
33
14
opt meimmize men. zer.zzr_aaa loss bawckarrd
35
36
opt mi welzze.sst set.ste()))
Listing 1: PyTorch-style pseudo code of the proposed pretraining framework.
B ADDITIONAL ANALYSIS
B. PERFORMANCE WITH DIFFERENT STEPS OF GRADIENT UPDATE
We present performance obtained with different N steps of gradient update the corrected samples. We use = 2 for the best eerfrrnanc--fficiencyy trade-off and the proposed framework can perform fairly well with few single step of gradient update to each corrupted sample.
Accuracy
79
78
Random large
Colorization
77
Sorting
steps
Figure A: Performance with different N. N = corresponds to using corrupted images as negative.
B.2
Loss CURVE
We plot the training curve for the mixed experiment with VIT-B in Figure B The potentially inaccurate energy estimation at early training stage does not hurt the overall training.
B.3 OTHER NETWORK ARCHITECTURES AND DOWNSTREAM TRANSFER
Different from models like MAE He et al. 2022 and SimMIM (Xie et al. 2022) that are specifically tailored for particular network architectures. our framework can be seamlessly applied any deep vision models without any customization or auxiliary network components beside the simple linear head h. To show this, we present results with convolution-basel ResNet (He al. 2016), Cts (Liu et al.] 2022) and TriinTTranffomer (Liu et al. 2021) in Table B We replace the batch normaliza- tion layers with group normalization layers in ResNet ensure the training stability. And validate
15

---

Published as a conference paper at ICLR 2023
Training loss curve
0.20
Loss
0.15
اll
0.10
2000
4000
6000
8000
10000
12000
14000
Iterations
Figure B: Training loss curve.
Table B: The proposed framework can be seam- lessly applied any deep vision models. FS, PT, and FT denote from-scratch training, pretraining, and finetuning, respectively.
Table C: mloU results with ADE20K semantic segmentation finetuning.
method
data
ViT-B
Supervised
ImageNet
47.4
Networks
FS 300E
PT 200E + FT 100E
MoCo-v3
ImageNet
47.3
ConvNeXt-T
82.1
82.7
BETT
ImageNet-DALL.E
47.1
Swin-T
81.3
82.2
MAE
ImageNet
48.1
ResNet-50
76.5
77.2
Ours
ImageNet
47.5
the effectiveness the downstream transfer, we finetune the pretrained network the ADE20K
(Zhou et al., (017)) semantic segmentation dataset, and present the results with mean interaction union (mloU) in Table The proposed framework generalizes well across architectures and downsteam tasks.
B.4 ADDITIONAL EVALUATIONS
We report in Table D additional evaluations ImageNet-I All numbers obtained with the ViT-B network. And we present experiments with non-linear MLP head (three layers with 768 channels and ReLU activation) with the pretrained feature extractor frozen. For both MAE He et al. (2022) and our method, we consistently observe that a three-layer MLP head cannot significantly improve the performance compared to linear probing.
Following the low-data regime discussions in (Chen et al. 2020b), we further present Table D results with semi-supervisel finetuning on ImageNet. Specifically, we finetune the entire networks with 1% 10% ImageNet training samples and report the top-1 accuracy on official validation set.
To further evaluate how well the learned features transfer to downstream tasks, following the discus- sions in (Chen al. 2020b), we present linear probing results with additional fine-grained natural image datasets in Table
B.5 ENERGY SCORE
we show the histograms of the scores estimated by a trained model. Step 0 in Figure C corresponds to the scores of the manually corrupted images, and Step corresponds to the scores of the images obtained by one-step recovery by our model given the manually corrupted images. The trained model assigns lowest scores to the real images. We further present the the energy score histograms of ImageNet and the validation set of Stanford Cars (Over 8K images) in Figure The energy estimation generalizes well to unseen images as the network assigns the same low scores to the images of the additional natural images.
In Figure C.
16

---

Published as a conference paper at ICLR 2023
Table Additional empirical evaluations. We report the top-1 accuracy for all experiments.
Models
VTT-B MAE
ViT-B Ours
Linear probing MLP head
67.8
66.5
68.2
67.2
1% semi superviie
48. 48
48.67
10% semi-supervised
72.73
72.58
Cars
52.13
52.58
Aircraft
58,43
58.67
Pots
85.64
85.08
Flowers
93.38
94.02
Histograms Energy Scores
Real
4000
Step
Step
Frequency
2000
1000
-0
-40
-30
20
-10
Energy score
Figure C: Histograms of energy scores. All scores obtained on the ImageNet validation set.
Histograms of Energy Scores
ImageNet
2000
Cars
Frequency
1500
1000
500
-50
40
-30
-20
-10
Energy score
Figure D: Energy score histograms of natural images.
17


Reviews:
[{"review_id": "xhbSDZKowKs", "reviewer": "Reviewer_qFyR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel representation learning method inspired by energy-based generative models, which can be combined with different restoration tasks as self-supervised surrogate tasks. Specifically, a neural network-based energy function is learned which assigns a high energy to corrupted images, and a low energy to clean images. The method is evaluated for different corruptions and performs competitively with MAE when fine-tuning on the full ImageNet data set, without using a decoder for pretraining. ", "summary_of_the_review": "The proposed method is interesting and novel, and turns many common restoration tasks into self-supervised learning objectives. The method shows promising results in the presented evaluations.\n\nThe paper could benefit from additional evaluations, such as transfer beyond the training data distribution, possibly in a low data regime. Furthermore, there are several aspects that could be discussed more and put into context of additional related work.\n\nI’m choosing my rating defensively, but I’ll be happy to raise it if the authors provide a convincing rebuttal.", "strength_and_weaknesses": "*Strengths:*\n- The approach seems to be novel and takes the direction direction of using generative models for representation learning, which was less successful than e.g. contrastive methods in the recent past\n- The method can be combined with several corruptions, which might lead to different representations which are amenable to different types of downstream tasks.\n- The method is similar to MAE in that it produces a dense output in image space, but does not require a decoder which is only used in pretraining and does not contribute to the representation.\n\n*Weaknesses:*\n- The paper only reports fine-tuning results on the full ImageNet data set (and ADE20k), while acknowledging that training a linear head on top of the frozen representation leads to weaker performance than contrastive methods, similar to MAE. To strengthen the paper, the authors could provide linear evaluation results or results based on an MLP head for reference and comparison with other methods, or - more sensibly - transfer results to different data sets potentially in a low data regime (see, e.g. Chen et al. 2020 for inspiration).\n- The authors highlight several times that their method needs fewer steps to reach a given quality than prior work, but they do not discuss that their method requires two forward passes, and three backward passes (two gradient computations w.r.t input, and one w.r.t. the parameters for the update step). I think this should be discussed explicitly (I do acknowledge the walltime evaluation in Table 5).\n- Sorting as a self-supervised task has been explored recently in [a]. I feel the paper should discuss [a] and compare their results to [a].\n\n[a] Zhai, Shuangfei, et al. \"Position Prediction as an Effective Pretraining Strategy.\" International Conference on Machine Learning. PMLR, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mj-cVl89yb", "reviewer": "Reviewer_nBFT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework inspired by energy-based models for self-supervised pretraining. An image is first degraded by a randomly selected image corruption, and the training objective is defined as the MSE between the original image and the restored image derived by one-step gradient descent along the energy direction from the restored image from the previous sampling step. Empirical results on ViT-type structures show that the method is competitive with MAE and computationally more efficient. The proposed method can also be applied to other structures and downstream tasks. ", "summary_of_the_review": "Overall speaking, the paper proposes an interesting idea inspired by EBMs for self-supervised pretraining. The framework can handle various pretraining tasks and is agnostic to model architecture. Empirical results are promising. More principled justification of the training objective is desired. ", "strength_and_weaknesses": "### Strength:\n\n- Given the close connection between EBMs and discriminative models, it is natural to consider applying it the self-supervised learning. The proposed method enjoys the advantage of simplicity (no auxiliary networks) and efficiency (no large amount of negative samples, faster convergence). \n-  Extensive experiments have been conducted to validate the effectiveness of the proposed method. \n\n### Weakness:\n\n- The statement that sampling from EBMs equals to argmin the energy function or Eqn. 4 is wrong. Think about a simple case where the target distribution is a 1D standard Gaussian. Then samples derived by iteratively applying Eqn. 4 will form a Dirac distribution at 0 instead of the Gaussian. \n- A more principled derivation of Eqn. 5 is desired. For example, Eqn. 3 is derived by maximizing likelihood of the EBM. Does Eqn. 5 correspond to something similar but with a modified model assumption? In practice, have you ever tried to use the loss function in Eqn. 3 instead of the one in Eqn. 5, keep everything else the same and compare the results? \n- The paper states that one potential reason that the method performs better than MAE is that the model is exposed to both the original images and corrupted images. It is contrary to the statement by MAE that corrupted images are away from the data manifold so it is better to let the model be exposed to only the observed part of the images. More discussion on why the statement in this paper is true is preferred. \n- In the experiment of patch sorting (Table 2), it is not quite fair to train the AE baseline with position embeddings removed in pretraining and added back during fine-tuning. A fairer setting would be using randomly shuffled position embeddings in pretraining.  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_OU6HyI3SD", "reviewer": "Reviewer_WmV7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is working on pre-training of vision models. They propose to do energy-based sampling, but with MSE loss to the original sample as supervision. Several techniques are used to improve the training efficiency, such as doing energy-based sampling starting from randomly masked images, instead of random noise. The method is evaluated on different tasks including masking, super-resolution, colorization, and sorting. Experiments on several datasets have shown better performance than some baselines.", "summary_of_the_review": "This paper provides a simple idea with promising experimental results; It will be helpful for the community as a baseline and inspire more investigations on similar models such as diffusion models.\n\n-------------------------------------------\nI think my concerns have been addressed by the author's response. Specifically, since the generative perspective is not the major point of this paper, with some claims being toned down and adjusted a bit, I think the submission is good for acceptance and I've raised my score to 8. I think releasing the code (other than the pseudo-code alone) and adding some analysis on the generative modeling side will make the submission more helpful for the whole community.", "strength_and_weaknesses": "Strength\n- This paper proposes a very simple baseline. \n- Compared to maximum likelihood learning in EBMs, the MSE loss used here provides explicit supervision, which may reduce the instability in EBM training and potentially improve the training efficiency. Different from EBMs that are doing general generative modeling, the proposed method is doing more specific tasks on visually pretraining. Such a focus shift makes the modifications proposed to be reasonable. Though losing some nice properties of EBMs, such as flexible sampling starting from random noise, the proposed method can still serve as a strong visual pretraining model. \n- From a visual pre-training perspective, combining the encoder and decoder may reduce the model complexity, yielding a more compact model with shared weights.\n\nWeakness\n- From a theoretical perspective, the contributions are limited. The overall idea is heavily based on EBMs. But this paper is doing it with a different initialization and using MSE loss as direct supervision. The MSE loss term has also been used in EBM learning in some work as a regularization term to stabilize the training.\n- There is a lack of theoretical analysis. Most of the illustrations on EBMs are only intuitive, e.g., 'by gradient based updating along the direction of energy minimization, which equally encourages higher energy values for negative images, and can functionally replace the second term in (3).' From the generative modeling perspective, this loss function may or may not lead to convergence on the target distribution we are trying to model. Does the learned network have a smooth energy landscape?\n- I failed to understand the experiments in Figure A. If $N=0$, how does the model obtain something different from the input? Also, the accuracy does not have large improvements even with more steps.\n- In terms of efficiency, the number of epochs may not be a good metric due to the varying number of Langevin sampling steps used.\n- Eq. 4 is missing the noise term. It may be better to include the original Langevin Dynamics equation here and indicate the noise term is dropped in practice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "YB449Z-E4A6", "reviewer": "Reviewer_74n4", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new series of pretext-tasks for self-supervised learning by using an energy based approach. The pretext-task is defined with a denoising score matching criterion aiming for the reconstruction of a corrupted image. The authors try several corruption schemes and show that their method perform is competitive with masked autoencoders and several other SSL methods without the need for an explicit decoder.", "summary_of_the_review": "I really love the idea behind the paper and if the experimental setup was a bit stronger and more diverse, I would advise strongly for acceptance. However, for now, I have some concerns, especially around the potential use or not of training examples as conditioning in Figure 3 as well as for the results in linear probing which seem to have been voluntary hidden by the authors.", "strength_and_weaknesses": "Strength:\n- This paper demonstrate that we can learn a model that is competitive with MAE without the need to use an explicit decoder. I really like that the authors try learn the decoder in an implicit way using score matching.\n- It's also very impressive and I am really surprised that you only need 2 or 3 steps for sampling.\n\nWeaknesses:\n- **Lack of diversity in the experiments** I would have strongly appreciated the addition of the linear probing results in the paper. You mentioned lower linear probing accuracy in the limitations, but it's not clear what does that mean if you don't present any numbers. For instance [1] demonstrate that MAE can achieve 73,5% linear probing accuracy which is already better than most of SSL methods. So even if it's not as good as the performances in finetuning, it's still a competitive and respectable score (in comparison with the current literature). So does your method perform as well as MAE in linear probing (around 70%) or does it fail completely (like you only get 5% accuracy on ImageNet). Following this first point, the main issue I have with this paper is that you only use finetuning as evaluation criteria. However, anyone can also fine-tune from random weights. If your baseline in grey in table 1 and table 2, the \"From Scratch\" is a training from random initialized weights, it mean that the benefice of pre-training is only 2 or 3%, which is not really convincing.  So I would expect to see additional evaluation metrics like linear probing or/and KNN (even if it's not as good as MAE, it's ok but one need to be sure that the representation learned is actually \"meaningful\"). In addition of linear probing, you can also trained a small MLP on top of the frozen VIT backbone representation (it should be able to retrieve the information even if it's not easily linearly separable). \n\n- **Lack of details in the experimental setup** The reconstruction in Figure 3 seems great, but are you using images from the training set or the validation set ? When you mentioned your baseline \"From scratch\" in Table 1 or 2, what is the training scenario of this baseline, are you also using 200 epochs or 100 epochs ? For the denoising experiments, are you still using only 2 steps ? Can you add on Figure A, the denoising experiment ? (I would expect the number of denoising steps to vary depending on which gamma parameter you are sampling from the uniform, did you try with fixed value of gamma ?) The main reason about why I am asking this, is that in the diffusion model literature one major issue is the number of sampling steps to get good image generation. When reading the paper, one could think that you were successfully able to solve the diffusion model number of sampling steps issues which seem to be difficult to believe. Are you using images of size 224x224 ? Overall, there is a lack of detail and precision in the experimental section. It's not clear why you have 7 corruption schemes in Figure 3 but you have only 3 curves in Figure A.\n\n- **Missing important references**: Your training criterion corresponds to the denoising score matching loss [2] so this reference should at least been cited. I would have appreciated to see MSN [3] in table 3 and 5 since it's a SSL join embedding method that use masking and that is very efficient.\n\n- **Overclaim** The authors suggest that \"The proposed method can be seamlessly applied to ay deep vision models\" . However, in the experimental setup, they present only results with transformers inspired vision models. What about the classic Resnet50 ? Either you specify that the proposed method can be applied on any deep transformer vision model, and present experiments with vision transformers OR you present also results with traditional architecture like Resnet/Desnet.\n\n- **Miscellaneous** Small typo, p9 \"the masked images.Different\"\n\n[1] Masked autoencoders are scalable vision learners, He et al, 2022\n\n[2] A Connection Between Score Matching and Denoising Autoencoders, Vincent et al\n\n[3] Masked Siamese Networks for Label-Efficient Learning, Assran et al", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FQFzcwDZhd", "reviewer": "Reviewer_TAyT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel self-supervised pretraining framework for vision models. Inspired by energy-based models (EBMs), the paper proposes training a vision model by sampling corrupted image samples and moving them along the direction of energy minimization using gradient descent, with corrupted samples having high energy and vice-versa. Comprehensive experiments were conducted using a broad range of pretext tasks with different data-corruption methods. ", "summary_of_the_review": "Borderline reject. The paper definitely proposes a novel idea, however, I think a bit more analysis and results are required. If the author(s) are able to address the above comments, the decision will be reconsidered.", "strength_and_weaknesses": "Pros\n\t- The proposed method is simple and seems easy to implement with the provided details.\n\t- Strong benchmark performance improvements in the provided results on Imagenet-1K.\n\nCons\n\t- The paper was a bit difficult to read. In several places, prior knowledge about concepts involved in training energy based models was assumed from the reader. It would be helpful if more information is provided about forward and backward passes while training energy based models like the Boltzmann machines, sampling strategies, etc.\nAll the results were presented only on one particular dataset. Since the paper is on self-supervised pretraining of vision models, it would be really helpful if results on other vision datasets/tasks are also compared to the state-of-the-art results in the respective tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DBJ-9aTgDa", "reviewer": "Reviewer_A5RJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a new self-supervised learning (SSL) framework for vision models.\n\nThe framework leverages an energy-based model (EBM): an energy $E_{\\theta}$ is parameterized by a deep neural network which should learn to assign high energy to out-of-domain samples and low energy to in-domain samples. To train an EBM, one typically needs to sample from the learned distribution by doing gradient descent w.r.t $E_{\\theta}$. Hence, according to the authors, training an EBM in the context of SSL is unaffordable.\n\nTo avoid this issue, the authors propose to perform conditional sampling, e.g. restoring partially corrupted data. By doing so, few gradient steps are required for sampling (2 in the paper). Then, the energy is updated by minimizing the MSE loss between the true image and successive reconstruction steps in the pixel space. The proposed framework allows to use various pretext tasks for image corruption such as masking, patch sorting, colorization, denoising and superresolution.\n\nThe authors then proceed to evaluate the proposed method by pre-training and evaluating on ImageNet1k, and finetuning on ADE20k, demonstrating on par results with other baselines while being more efficient both in terms of epochs and than most baselines. The energy is usually modeled with a ViT but ConvNeXt and SwinTransformer are also used.", "summary_of_the_review": "This work proposes an appealing method for learning visual representations: it seems simple and provides both a generative model and a representation. Moreover, it demonstrates results on ImageNet that are on-par with many competitive methods with seemingly more efficiency in terms of epochs and FLOPs while working not only with ViTs but also with ConvNeXt. However, this comes with some flaws: for example, some parts of the paper lack clarity, the experimental analysis could study other downstream tasks, linear probing, and the generative capability of the model. I believe these problems could be adressed in the rebuttal.\n\n________\n\nAfter rebuttal: my concerns (lack of clarity on design choices, lack of discussion around the energy function), other tasks were answered and I updated my score to 8", "strength_and_weaknesses": "Strengths:\n- The proposed method seems effective: competitive results are achieved on ImageNet while using less pre-training epochs.\n- As opposed to joint embeddings method, the proposed framework does not rely as much on data augmentations.\n- The authors provides good results for other architectures than ViT (ConvNeXt and SwinTransformer).\n- A lot of different pretext tasks are evaluated.\n\nWeaknesses:\n- Linear evaluation: the authors acknowledge that their model does not perform as well with linear probing as with fine-tuning but it seems that no numbers were provided. In my opinion it can be discussed whether linear probing is an essential feature for a SSL model, but it would be nice to have the number to better assess the proposed method.\n- It would be useful to have at least one more downstream task evaluation to better assess the method.\n- I understand that the outcome of the proposed method is both a discriminative and a generative model. If so, it would be useful to study the generative properties of the learned model.\n\nQuestions:\n- How to ensure / do we have in practice that (5) further minimizes the energy for the final reconstruction steps than for the first steps?\n- Somehow related, I understand that the $E_{\\theta}$ is trained to minimize (5), meaning it is never explicitly trained to maximize the energy of out-of-domain samples?\n- In the light of the last two questions, it may be interesting to provide an experimental study of the energy function to study if its behavior aligns with the motivation for EBMs.\n- After the $E_{\\theta} is trained, is is possible to sample \"original\" images from the low energy regions?\n- Many SSL methods use a projector in between the pretext task (minimizing the energy) and the representation (your $\\psi$), which is beneficial. Have you considered using a more sophisticated head than the linear $h$? (I do not request this experiment). Intuitively, it may improve the linear separability of the representation if this is not aligned with obtaining the energy linearly from the representation. Another quick experiment would be to try linear probing on an intermediate representation obtained before the last layer of the ViT.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%