Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces a new self-supervised learning (SSL) framework for vision models.

The framework leverages an energy-based model (EBM): an energy $E_{\theta}$ is parameterized by a deep neural network which should learn to assign high energy to out-of-domain samples and low energy to in-domain samples. To train an EBM, one typically needs to sample from the learned distribution by doing gradient descent w.r.t $E_{\theta}$. Hence, according to the authors, training an EBM in the context of SSL is unaffordable.

To avoid this issue, the authors propose to perform conditional sampling, e.g. restoring partially corrupted data. By doing so, few gradient steps are required for sampling (2 in the paper). Then, the energy is updated by minimizing the MSE loss between the true image and successive reconstruction steps in the pixel space. The proposed framework allows to use various pretext tasks for image corruption such as masking, patch sorting, colorization, denoising and superresolution.

The authors then proceed to evaluate the proposed method by pre-training and evaluating on ImageNet1k, and finetuning on ADE20k, demonstrating on par results with other baselines while being more efficient both in terms of epochs and than most baselines. The energy is usually modeled with a ViT but ConvNeXt and SwinTransformer are also used.
strength_and_weaknesses: Strengths:
- The proposed method seems effective: competitive results are achieved on ImageNet while using less pre-training epochs.
- As opposed to joint embeddings method, the proposed framework does not rely as much on data augmentations.
- The authors provides good results for other architectures than ViT (ConvNeXt and SwinTransformer).
- A lot of different pretext tasks are evaluated.

Weaknesses:
- Linear evaluation: the authors acknowledge that their model does not perform as well with linear probing as with fine-tuning but it seems that no numbers were provided. In my opinion it can be discussed whether linear probing is an essential feature for a SSL model, but it would be nice to have the number to better assess the proposed method.
- It would be useful to have at least one more downstream task evaluation to better assess the method.
- I understand that the outcome of the proposed method is both a discriminative and a generative model. If so, it would be useful to study the generative properties of the learned model.

Questions:
- How to ensure / do we have in practice that (5) further minimizes the energy for the final reconstruction steps than for the first steps?
- Somehow related, I understand that the $E_{\theta}$ is trained to minimize (5), meaning it is never explicitly trained to maximize the energy of out-of-domain samples?
- In the light of the last two questions, it may be interesting to provide an experimental study of the energy function to study if its behavior aligns with the motivation for EBMs.
- After the $E_{\theta} is trained, is is possible to sample "original" images from the low energy regions?
- Many SSL methods use a projector in between the pretext task (minimizing the energy) and the representation (your $\psi$), which is beneficial. Have you considered using a more sophisticated head than the linear $h$? (I do not request this experiment). Intuitively, it may improve the linear separability of the representation if this is not aligned with obtaining the energy linearly from the representation. Another quick experiment would be to try linear probing on an intermediate representation obtained before the last layer of the ViT.
clarity,_quality,_novelty_and_reproducibility: Clarity:
- The paper is globally easy to understand but some parts are confusing:
- In Table 1, is "from scratch" the supervised baseline? In Table 3, what does "Mixed" refer to?

Quality:
- The introduction relies on misconceptions or small errors: it seems it refers to methods such as BYOL or SimSiam as contrastive methods whereas they are usually coined as non-contrastive (the fact that they are truly non-contrastive can be discussed but the adopted formulation is confusing). More importantly, it is exaggerated to say that the batch size is an important problem in contrastive methods: SimCLR for example can provide good results with a batch size of 1024, generally requiring 8 GPUs "only". See, e.g., [1]. Plus, non-contrastive methods such as BarlowTwins or VICReg do not require the momentum copy.

Novelty:
- To the best of my knowledge, obtaining such results on ImageNet with an EBM is new.

Reproducibility:
- No code is provided, while important hyper-parameters are given in Appendix.

[1] Garrido et al., On the duality between contrastive and non-contrastive self-supervised learning (2022).
summary_of_the_review: This work proposes an appealing method for learning visual representations: it seems simple and provides both a generative model and a representation. Moreover, it demonstrates results on ImageNet that are on-par with many competitive methods with seemingly more efficiency in terms of epochs and FLOPs while working not only with ViTs but also with ConvNeXt. However, this comes with some flaws: for example, some parts of the paper lack clarity, the experimental analysis could study other downstream tasks, linear probing, and the generative capability of the model. I believe these problems could be adressed in the rebuttal.

________

After rebuttal: my concerns (lack of clarity on design choices, lack of discussion around the energy function), other tasks were answered and I updated my score to 8
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework inspired by energy-based models for self-supervised pretraining. An image is first degraded by a randomly selected image corruption, and the training objective is defined as the MSE between the original image and the restored image derived by one-step gradient descent along the energy direction from the restored image from the previous sampling step. Empirical results on ViT-type structures show that the method is competitive with MAE and computationally more efficient. The proposed method can also be applied to other structures and downstream tasks. 
strength_and_weaknesses: ### Strength:

- Given the close connection between EBMs and discriminative models, it is natural to consider applying it the self-supervised learning. The proposed method enjoys the advantage of simplicity (no auxiliary networks) and efficiency (no large amount of negative samples, faster convergence). 
-  Extensive experiments have been conducted to validate the effectiveness of the proposed method. 

### Weakness:

- The statement that sampling from EBMs equals to argmin the energy function or Eqn. 4 is wrong. Think about a simple case where the target distribution is a 1D standard Gaussian. Then samples derived by iteratively applying Eqn. 4 will form a Dirac distribution at 0 instead of the Gaussian. 
- A more principled derivation of Eqn. 5 is desired. For example, Eqn. 3 is derived by maximizing likelihood of the EBM. Does Eqn. 5 correspond to something similar but with a modified model assumption? In practice, have you ever tried to use the loss function in Eqn. 3 instead of the one in Eqn. 5, keep everything else the same and compare the results? 
- The paper states that one potential reason that the method performs better than MAE is that the model is exposed to both the original images and corrupted images. It is contrary to the statement by MAE that corrupted images are away from the data manifold so it is better to let the model be exposed to only the observed part of the images. More discussion on why the statement in this paper is true is preferred. 
- In the experiment of patch sorting (Table 2), it is not quite fair to train the AE baseline with position embeddings removed in pretraining and added back during fine-tuning. A fairer setting would be using randomly shuffled position embeddings in pretraining.  


clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written and easy to follow. 
Quality: overall good. some statements in the paper are wrong or require further justification. 
Novelty: the combination of EBMs and pretraining in the proposed way is novel. The results are interesting enough to the community.
Reproducibility: pseudo-code and implementation details are provided. 
summary_of_the_review: Overall speaking, the paper proposes an interesting idea inspired by EBMs for self-supervised pretraining. The framework can handle various pretraining tasks and is agnostic to model architecture. Empirical results are promising. More principled justification of the training objective is desired.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper introduces a new series of pretext-tasks for self-supervised learning by using an energy based approach. The pretext-task is defined with a denoising score matching criterion aiming for the reconstruction of a corrupted image. The authors try several corruption schemes and show that their method perform is competitive with masked autoencoders and several other SSL methods without the need for an explicit decoder.
strength_and_weaknesses: Strength:
- This paper demonstrate that we can learn a model that is competitive with MAE without the need to use an explicit decoder. I really like that the authors try learn the decoder in an implicit way using score matching.
- It's also very impressive and I am really surprised that you only need 2 or 3 steps for sampling.

Weaknesses:
- **Lack of diversity in the experiments** I would have strongly appreciated the addition of the linear probing results in the paper. You mentioned lower linear probing accuracy in the limitations, but it's not clear what does that mean if you don't present any numbers. For instance [1] demonstrate that MAE can achieve 73,5% linear probing accuracy which is already better than most of SSL methods. So even if it's not as good as the performances in finetuning, it's still a competitive and respectable score (in comparison with the current literature). So does your method perform as well as MAE in linear probing (around 70%) or does it fail completely (like you only get 5% accuracy on ImageNet). Following this first point, the main issue I have with this paper is that you only use finetuning as evaluation criteria. However, anyone can also fine-tune from random weights. If your baseline in grey in table 1 and table 2, the "From Scratch" is a training from random initialized weights, it mean that the benefice of pre-training is only 2 or 3%, which is not really convincing.  So I would expect to see additional evaluation metrics like linear probing or/and KNN (even if it's not as good as MAE, it's ok but one need to be sure that the representation learned is actually "meaningful"). In addition of linear probing, you can also trained a small MLP on top of the frozen VIT backbone representation (it should be able to retrieve the information even if it's not easily linearly separable). 

- **Lack of details in the experimental setup** The reconstruction in Figure 3 seems great, but are you using images from the training set or the validation set ? When you mentioned your baseline "From scratch" in Table 1 or 2, what is the training scenario of this baseline, are you also using 200 epochs or 100 epochs ? For the denoising experiments, are you still using only 2 steps ? Can you add on Figure A, the denoising experiment ? (I would expect the number of denoising steps to vary depending on which gamma parameter you are sampling from the uniform, did you try with fixed value of gamma ?) The main reason about why I am asking this, is that in the diffusion model literature one major issue is the number of sampling steps to get good image generation. When reading the paper, one could think that you were successfully able to solve the diffusion model number of sampling steps issues which seem to be difficult to believe. Are you using images of size 224x224 ? Overall, there is a lack of detail and precision in the experimental section. It's not clear why you have 7 corruption schemes in Figure 3 but you have only 3 curves in Figure A.

- **Missing important references**: Your training criterion corresponds to the denoising score matching loss [2] so this reference should at least been cited. I would have appreciated to see MSN [3] in table 3 and 5 since it's a SSL join embedding method that use masking and that is very efficient.

- **Overclaim** The authors suggest that "The proposed method can be seamlessly applied to ay deep vision models" . However, in the experimental setup, they present only results with transformers inspired vision models. What about the classic Resnet50 ? Either you specify that the proposed method can be applied on any deep transformer vision model, and present experiments with vision transformers OR you present also results with traditional architecture like Resnet/Desnet.

- **Miscellaneous** Small typo, p9 "the masked images.Different"

[1] Masked autoencoders are scalable vision learners, He et al, 2022

[2] A Connection Between Score Matching and Denoising Autoencoders, Vincent et al

[3] Masked Siamese Networks for Label-Efficient Learning, Assran et al
clarity,_quality,_novelty_and_reproducibility: The idea is very easy to understand, the authors did a great job in explaining it and also I really appreciate the pytorch pseudo code available in the appendix. I don't think that there is a real novelty in this work (in the fundamental meaning) however the authors were successfully able to put the different existing pieces together which I think was something important to do. 
summary_of_the_review: I really love the idea behind the paper and if the experimental setup was a bit stronger and more diverse, I would advise strongly for acceptance. However, for now, I have some concerns, especially around the potential use or not of training examples as conditioning in Figure 3 as well as for the results in linear probing which seem to have been voluntary hidden by the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is working on pre-training of vision models. They propose to do energy-based sampling, but with MSE loss to the original sample as supervision. Several techniques are used to improve the training efficiency, such as doing energy-based sampling starting from randomly masked images, instead of random noise. The method is evaluated on different tasks including masking, super-resolution, colorization, and sorting. Experiments on several datasets have shown better performance than some baselines.
strength_and_weaknesses: Strength
- This paper proposes a very simple baseline. 
- Compared to maximum likelihood learning in EBMs, the MSE loss used here provides explicit supervision, which may reduce the instability in EBM training and potentially improve the training efficiency. Different from EBMs that are doing general generative modeling, the proposed method is doing more specific tasks on visually pretraining. Such a focus shift makes the modifications proposed to be reasonable. Though losing some nice properties of EBMs, such as flexible sampling starting from random noise, the proposed method can still serve as a strong visual pretraining model. 
- From a visual pre-training perspective, combining the encoder and decoder may reduce the model complexity, yielding a more compact model with shared weights.

Weakness
- From a theoretical perspective, the contributions are limited. The overall idea is heavily based on EBMs. But this paper is doing it with a different initialization and using MSE loss as direct supervision. The MSE loss term has also been used in EBM learning in some work as a regularization term to stabilize the training.
- There is a lack of theoretical analysis. Most of the illustrations on EBMs are only intuitive, e.g., 'by gradient based updating along the direction of energy minimization, which equally encourages higher energy values for negative images, and can functionally replace the second term in (3).' From the generative modeling perspective, this loss function may or may not lead to convergence on the target distribution we are trying to model. Does the learned network have a smooth energy landscape?
- I failed to understand the experiments in Figure A. If $N=0$, how does the model obtain something different from the input? Also, the accuracy does not have large improvements even with more steps.
- In terms of efficiency, the number of epochs may not be a good metric due to the varying number of Langevin sampling steps used.
- Eq. 4 is missing the noise term. It may be better to include the original Langevin Dynamics equation here and indicate the noise term is dropped in practice.
clarity,_quality,_novelty_and_reproducibility: The writing is overall clear and the evaluations are of good quality.
Hyper-parameters are provided, but only with pseudo-code. There are some implementation details such as how the step size $\alpha$ is learned is not covered in the paper.
summary_of_the_review: This paper provides a simple idea with promising experimental results; It will be helpful for the community as a baseline and inspire more investigations on similar models such as diffusion models.

-------------------------------------------
I think my concerns have been addressed by the author's response. Specifically, since the generative perspective is not the major point of this paper, with some claims being toned down and adjusted a bit, I think the submission is good for acceptance and I've raised my score to 8. I think releasing the code (other than the pseudo-code alone) and adding some analysis on the generative modeling side will make the submission more helpful for the whole community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel self-supervised pretraining framework for vision models. Inspired by energy-based models (EBMs), the paper proposes training a vision model by sampling corrupted image samples and moving them along the direction of energy minimization using gradient descent, with corrupted samples having high energy and vice-versa. Comprehensive experiments were conducted using a broad range of pretext tasks with different data-corruption methods. 
strength_and_weaknesses: Pros
	- The proposed method is simple and seems easy to implement with the provided details.
	- Strong benchmark performance improvements in the provided results on Imagenet-1K.

Cons
	- The paper was a bit difficult to read. In several places, prior knowledge about concepts involved in training energy based models was assumed from the reader. It would be helpful if more information is provided about forward and backward passes while training energy based models like the Boltzmann machines, sampling strategies, etc.
All the results were presented only on one particular dataset. Since the paper is on self-supervised pretraining of vision models, it would be really helpful if results on other vision datasets/tasks are also compared to the state-of-the-art results in the respective tasks.
clarity,_quality,_novelty_and_reproducibility: The paper was a bit difficult to read. In several places, prior knowledge about concepts involved in training energy based models was assumed from the reader. The paper proposes a novel idea of using ideas from energy based models for self supervised training.
summary_of_the_review: Borderline reject. The paper definitely proposes a novel idea, however, I think a bit more analysis and results are required. If the author(s) are able to address the above comments, the decision will be reconsidered.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a novel representation learning method inspired by energy-based generative models, which can be combined with different restoration tasks as self-supervised surrogate tasks. Specifically, a neural network-based energy function is learned which assigns a high energy to corrupted images, and a low energy to clean images. The method is evaluated for different corruptions and performs competitively with MAE when fine-tuning on the full ImageNet data set, without using a decoder for pretraining. 
strength_and_weaknesses: *Strengths:*
- The approach seems to be novel and takes the direction direction of using generative models for representation learning, which was less successful than e.g. contrastive methods in the recent past
- The method can be combined with several corruptions, which might lead to different representations which are amenable to different types of downstream tasks.
- The method is similar to MAE in that it produces a dense output in image space, but does not require a decoder which is only used in pretraining and does not contribute to the representation.

*Weaknesses:*
- The paper only reports fine-tuning results on the full ImageNet data set (and ADE20k), while acknowledging that training a linear head on top of the frozen representation leads to weaker performance than contrastive methods, similar to MAE. To strengthen the paper, the authors could provide linear evaluation results or results based on an MLP head for reference and comparison with other methods, or - more sensibly - transfer results to different data sets potentially in a low data regime (see, e.g. Chen et al. 2020 for inspiration).
- The authors highlight several times that their method needs fewer steps to reach a given quality than prior work, but they do not discuss that their method requires two forward passes, and three backward passes (two gradient computations w.r.t input, and one w.r.t. the parameters for the update step). I think this should be discussed explicitly (I do acknowledge the walltime evaluation in Table 5).
- Sorting as a self-supervised task has been explored recently in [a]. I feel the paper should discuss [a] and compare their results to [a].

[a] Zhai, Shuangfei, et al. "Position Prediction as an Effective Pretraining Strategy." International Conference on Machine Learning. PMLR, 2022.

clarity,_quality,_novelty_and_reproducibility: Overall the paper is well written and easy to follow. To my knowledge, the approach is new. 
Regarding the model formulation, it would be informative to work out the precise connection of the training objective (5) to the EBM objective (1) if there is one.

In terms of related work, it would probably be worth mentioning iGPT [b] and BigBiGAN [c] as two recent generative modeling based representation learning approaches.

Some details on the experiments are missing, for example I could not find any mention of the image resolution used to train the method. The paper mentions a patch size of 24, which does not divide 224, which is a commonly used training resolution.

Typos:
- P5: “Note that…” instead of “Noted that…”; “...actual semantics.” instead of “...actual semantic.”
- p9: “...images. Different…” instead of “...images.Different…”

[b] Chen, Mark, et al. "Generative pretraining from pixels." International conference on machine learning. PMLR, 2020.

[c] Donahue, Jeff, and Karen Simonyan. "Large scale adversarial representation learning." Advances in neural information processing systems 32 (2019).

summary_of_the_review: The proposed method is interesting and novel, and turns many common restoration tasks into self-supervised learning objectives. The method shows promising results in the presented evaluations.

The paper could benefit from additional evaluations, such as transfer beyond the training data distribution, possibly in a low data regime. Furthermore, there are several aspects that could be discussed more and put into context of additional related work.

I’m choosing my rating defensively, but I’ll be happy to raise it if the authors provide a convincing rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a novel framework for self-supervised visual feature learning. The framework is general and encompasses ideas from previous work (masking, colorisation). The core idea is to use conditional sampling to implement an energy-based model. 
The paper shows good performance, with many architectures, and using many pretext tasks.
It has been noted that the paper lacks linear evaluation, as well as additional downstream evaluations. These concerns have been partially addressed during the rebuttal.
Overall, this submission is in good shape for a publication at ICLR. It is timely, implements an interesting framework while drawing links with previous work. I recommend accepting this paper as an Spotlight.
justification_for_why_not_higher_score: While presenting a novel framework, this paper draws a lot from previous work. The obtained performance allows to match the other work's accuracy when finetuning the model. The linear evaluation still lags behind whole-image contrastive algorithms. For this paper to get an oral, I think the framework should unlock something that was not possible before.
justification_for_why_not_lower_score: The paper is in strong shape, the experimental evaluation is thorough and the claims supported by empirical evidence.
Decision: Accept: notable-top-25%