Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Using labelled real data, an EBM is trained in the latent space of a pretrained GAN, rather than directly in pixel space. An ODE solver is used to sample from the EBM rather than Langevin Dynamics, which is shown to be more robust to hyperparameter settings. Using properties of the EBM, conditional generation compositional editing can be demonstrated, including in zero-shot scenarios.
main_review: The advantages of this approach are that it operates in latent space rather than pixel space, so training and inference is fast. Also, only the EBM portion is trained and synthesis uses a pretrained GAN, so it can combine the compositionality properties of EBMs with the image quality of recent GANs.

A few points of further clarification:

In the energy update function Eqn 1, samples $x$ are drawn from both the real data and the model, however in line 160 it is mentioned that the classifier is trained in the w space of StyleGAN, so it seems like the w-space is used as input to the classifier f, which does not naturally exist for real images. How are the real images provided as inputs to the classifier with respect to Eqn 1? 

In the CIFAR10 experiment, the StyleGAN-ada unconditional model FID is reported as 2.92, and conditional as 2.42, whereas in Table 1 the reported FID is slightly higher than that of the pretrained models. What causes this discrepancy, since the same model is used as the image generator?

Overall, the writing is clear and compositional results seems convincing. 
limitations_and_societal_impact: Limitations and broader impact are addressed.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies conditional learning by training an EBM in the latent space of a pre-trained top-down generator such as StyleGANs. The EBM is a joint distribution of data and attributes together, and sampling from it is formulated as solving an ordinary differential equation. Experimental results show that the method outperforms the state-of-the-art in both conditional sampling and sequential editing. The contribution of the paper lies on obtaining state-of-the-art performance of conditional learning by combining existing technologies, such as StyleGAN, latent space energy-based learning, and score-based generative model via stochastic differential equations. 
main_review: This paper addresses an important problem in computer vision, which is learning conditional generative models for controlled image generation. The paper combines the existing state-of-the-art techniques, including StyleGAN, energy-based model, and score-based generative models, to achieve the state-of-the-art performance of conditional generation. The paper is well-written, well-organized, and easy to follow.   
limitations_and_societal_impact: Despite the state-of-the-art results, the paper has the following limitations.

(1)	Insufficient novelty. (i) The performance of the proposed methods, i.e., the high quality of the generated images, comes from the pretrained generator of the StyleGANs, and the sampling of the EBM relies on the score-based generative model. Therefore, the results are not surprising because the paper just combines all existing state-of-the-art works together, and such a contribution is very straightforward. Even though the proposed pipeline to train such a conditional model is useful for the application, but the novelty is insufficient for the conference.  (ii) As to the latent space energy-based modeling, the EBM built on the data and attributes actually the posterior distribution of the top-down generator (a latent variable model). Sampling the latent space given the data via Langevin dynamics has been well studied in [1][2][3][4], in which the generator is trained by MLE where sampling the latent space conditioned on the data is required. This step is called MCMC inference. The energy function of the posterior distribution is the equation (6) in the current paper.  That means the latent space energy-based modeling is not new. 

[1] Alternating back-propagation for generator network. AAAI-17.

[2]  Learning dynamic generator model by alternating back-propagation through time. AAAI-19  

[3] Cooperative learning of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI 2020)

[4] Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning. (arXiv 2019 or PAMI 2021)

(2) Insufficient references. (i) The paper lacks some discussion of related works [1][2][3][4], which also samples the latent space of the generator by MCMC or uses the same form of latent space EBM as the current manuscript. (ii) missing pioneering works of ConvNet-EBM. The current successful deep energy-based learning framework was first proposed in Xie et al. 16 [5], where deep neural network is proposed to parameterize the energy function, and MLE with Langevin dynamics is used to train the parameters. The current paper does not cite those published prior arts before 2019, such as [5][6][7][8]. This might lead to negative ethical issues. 

[5] A Theory of Generative ConvNet (ICML 2016)

[6] Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet (CVPR 2017)

[7] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. (CVPR 2018)

[8] Learning generative ConvNets via multigrid modeling and sampling. (CVPR 2018)

(3)	Insufficient comparison. The paper might need to compare with [4], which is also related to controlled image generation. 

The limitations of the current paper are the insufficient novelty and insufficient references, thus I suggest a rejection.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper considers the task of obtaining additional control over a pre-trained generative model without re-training it. Specifically, it aims to generate conditional samples from an unconditional StyleGAN by only training a classifier with respect to the conditioning information. To achieve this, it considers the conditional distribution of StyleGAN latent codes. By Bayes Theorem, this conditional distribution is proportional to the unconditional distribution of the latent codes (which has a known standard normal distribution) times the distribution of conditioning information given the corresponding sample. The latter is modeled as a Gibbs distribution with an energy derived from the classifier. Two approaches to sample from this unnormalized distribution are evaluated: (i) Langevin sampling and (ii) Performing gradient descent on the classifier-derived energy. Experiments demonstrate good performance of this approach with significant improvements over StyleFlow for combinations of conditionings which are not in the training data.
main_review: **Strengths**
- The paper is well motivated by the desire to reuse powerful generative models without having to retrain them, which is quite costly. In particular, it is difficult to anticipate what needs regarding control over a generative model are desired by users and it is therefore necessary to develop techniques that enable customized controls in a computationally efficient post-hoc manner.
- The approach is simple and shows good performance. Based on this it could serve as an important baseline for face editing works and enjoy wide adaption.
- Besides good quality of the synthesized results, it achieves the goal of providing flexible control with little computational requirements as demonstrated by comparatively short training and inference times.
- Especially in the case of rare combinations of conditioning information, the approach demonstrates improved performance over StyleFlow. In general, the formulation of controlled synthesis as a composition of different energies 
provides a flexible control mechanism.

**Weaknesses**
- Both the idea that classifiers can be interpreted as (part of) an EBM (Grathwohl, Will et al. “Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One.” 2020) as well as the idea that optimizing latent codes of generators instead of pixels improves the quality when optimizing an energy derived from classification networks (Nguyen, Anh M et al. “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.” 2016) are known. Similarly, the ODE formulation follows [38] (although I'm not really sure why that derivation is necessary to formulate Eq. (10); see points below) and the composition of energies follows [8]. Thus, the main contribution is the evaluation of these techniques in a very specific setting.
- The paper makes the very restrictive assumption that the latent distribution follows a standard normal distribution. Thus, it assumes that the most difficult part regarding generative modeling is already solved. Moreover, it considers only relatively easy datasets with CIFAR-10 and FFHQ. To quote from Upchurch, P. et al. “Deep Feature Interpolation for Image Content Changes.” 2017: "Generative models can be much more powerful than linear interpolation, but the current problems (in particular, face attribute editing) which are used to showcase generative approaches are too simple." Here, the situation is even more severe because it makes the problem even simpler by relying on StyleGAN, such that one is directly working with a normally distributed, relatively low-dimensional code. Note that for example Nguyen 2016 does not make assumptions regarding normally distributed codes and considers more complex datasets such as ImageNet and MIT Scenes.
- While it is always interesting to see connections between different approaches and works, I fail to see how the extensive discussion in Sec. 2 of energy based models, langevin sampling and stochastic differential equations is necessary to arrive at Eq. (10). The reverse ODE to Eq. (10) describes gradient flow on the latent code with respect to a classification/regression network. Optimizing inputs with respect to such a network is a very natural idea that has been considered many times in different contexts in the literature (like Nguyen 2016 and Upchurch 2017 but also Goetschalckx, Lore et al. “GANalyze: Toward Visual Definitions of Cognitive Image Properties.” 2019 and similar works on finding interpretable directions in the latent space of GANs, encoding images into this latent space or performing superresolution as in Menon, Sachit et al. “PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models.” 2020). This discussion might be useful if it would unify previous approaches but in its current form it only seems to present a simple approach in an unnecessarily complex way.
- Regarding the derivation of Eq. (10): For $f(x,t)=-\frac{1}{2}\beta(t)x$ and $g(t)=\sqrt{\beta(t)}$ (defined in l.137), the reverse SDE according to Eq. (6) in [38] should read $dx = -\frac{1}{2}\beta(t) \[ x + 2 \nabla_x \log p_t(x) \]dt + \sqrt{\beta(t)} d\bar{w}$. Eq. (7) in the paper under review does not contain the factor 2 on the score $\nabla_x \log p_t(x)$. Because of this, $z$ and $\nabla_z \log p_t(z)$ in Eq. (9) cancel under the assumption that $p_t$ is standard normal. However, with the factor of 2, these terms would not cancel and Eq. (10) would include an additional term $\frac{1}{2}\beta(t) z dt$. Intuitively this also makes more sense to me since the latter would account for the fact that the marginal distribution of $z$ is standard normal. In this case, the resulting (reverse) ODE would also directly correspond to the gradient flow on the energy in Eq. (6). So which version is correct and which one was used in the implementation?
- One differences to other approaches that optimize latent codes is the use of the ODE solver [5] to solve (the reverse) of the gradient flow in Eq. (10) instead of using a simple Euler discretization commonly used for gradient descent. However, there is no ablation regarding this choice. Is it important to use [5] or does a simple Euler discretization work just as well? Similarly, the choice of the time-dependent diffusion coefficient beta is never discussed. What is used for beta? What is the effect of it?

**Comments and Suggestions**
- I think the paper demonstrates precisely the fact that face attribute editing with the help of pre-trained generative models is too simple: The final "model" that is proposed by this work is to optimize latent codes with respect to a desired output from a classification or regression model. Given the good results obtained by this approach is a good indication that this field (face attribute editing) should reconsider its baselines. Thus, I think the paper could become a valuable contribution by pointing this out and presenting its simple and strong approach that can serve as a baseline for future works. I think it is very important to have simple and strong approaches but unfortunately, in its current form, the paper seems to try to hide the simplicity of its approach.
- Why not formulate directly the distribution of x/z conditioned on c instead of talking about the joint distribution most of the time when the goal is to sample the former?
- I think the results from the ablation regarding which space to train the classifier on in A.4 are quite interesting and important. This choice seems to have a significant effect on the performance, and I guess it is also the reason for the short training time? How does training time on w space compare to training time on image space?

**Rating**
- The paper presents a simple approach to utilize a pretrained, unconditional GAN for conditional sampling and demonstrates good results. While it could serve as a simple and strong baseline for face attribute editing, in its current form the paper presents a different message regarding a "novel formulation of a joint EBM". Given that the model performs gradient descent on the latent code of StyleGAN this seems like an overclaim and I fail to see the novelty and contribution. The paper must reconsider to what purpose it draws the connection to energy based models. In its current form, this connection is not very natural and rather seems to, unnecessarily, hide a simple approach in a complex derivation.
- Thus, in my preliminary rating I do not recommend accepting the paper, but I would like to hear why the authors think that the connection to energy based models is important (especially with respect to Eq. (10)) and how this gives the paper more significance than directly stating that a simple baseline which performs gradient descent with respect to a classifer in the latent space of StyleGAN outperforms recently proposed and more complex approaches like StyleFlow.
limitations_and_societal_impact: Limitations regarding the need for powerful pre-trained generative models and the availability of labels to train the classifier are addressed and general potential impacts of generative models which are inherited by the presented approach are mentioned.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an effective and efficient approach to conditional image generation using GANs. Two main advances are described 1) The use of EBM in latent space that don’t require the retraining of image generators. This allows for very efficient training since the generator can be held fixed. 2) The use of an ODE solver for sampling that is less sensitive to hyperparameters than traditional LD solvers.

Results are shown on CIFAR-10 and FFHQ datasets. This includes results on sequential generation, compositional generation and zero-shot generation.

main_review: Overall this paper is borderline. The improvements in efficiency and the novelty of the approach push it slightly over the top, but the limitations in the experimental results keeps the paper from being a clear accept.

Positives:

++ EBM in latent space leads to very efficient training.

\+ Results show good disentanglement of attributes.

\+ Both ODE and LD solutions provide good results.

\+ Results are impressive with respect to StyleFlow

\+ Paper is well-written and easy to follow.

Negatives:

\- Inference time is longer than other approaches, but still not excessive.

\- The experimental set-up was varied from StyleGan paper, so hard to compare results.

\- It is unclear how inference time scales with the number of conditioned attributes.

\- No experiments on other high-res datasets except FFHQ.

\- Results on FFHQ only compare against StyleFlow. The StyleFlow paper offers comparisons to three other methods. This might be okay if the experimental set-up was the same, but it is not.

\- Results for sequential generation only use a single ordering. Are similar results found with different orderings? Be good to show a figure similar to Fig. 7 in StyleFlow paper.

Other comments:

+ The compositions of energy functions section (Line 239) doesn’t add much to the paper. It’d be better to use this space to compare against more approaches.

+ Results missing in several figures. Figure 2a left, Figure 4b left

+ Since you’re generating 1024x1024 images, it’d be good to show high-res crops (zoom-ins) showing the details of the images.

limitations_and_societal_impact: Limitations and societal impact are well-written. It would be good to cover in societal impact the implications of conditional GANs vs. unconditioned GANs. I.e., conditioned GANs may be more useful for nefarious applications.
review_title: null

Meta-review: The paper proposes an approach for controllable image generation based on applying an energy-based model in the latent space of a pre-trained unconditional generative model. The reviews are mixed: three reviewers are in favor of acceptance, while one recommends rejection. After reading the reviews, the rebuttal, the discussion, and the paper itself, below are some key points.

Pros:
1) The method is sensible and simple
2) The method has been evaluated thoroughly and shown to work well compared to the relevant baselines
3) Quite thorough analysis experiments evaluating different design decisions

Cons:
1) Limited novelty - the work mainly combines existing methods and is related to many prior papers
2) High-res experimental results only on one dataset

Overall, the proposed method is not overwhelmingly new, but it is simple and effective. The paper is well written and easy to understand, and the experimental evaluation is fairly thorough. I thus recommend acceptance at this point, but I urge the authors to take the reviewers' comments into account and adjust the final version of the paper accordingly.
Decision: Accept (Poster)