Controllable and Compositional Generation with Latent-Space Energy-Based Models
Authors: Weili Nie
Arash Vahdat
Anima Anandkumar
Venue: NeurIPS 2021 Poster (2021)
Keywords: Controllable generation
compositional generation
image synthesis
energy-based models
deep generative models
neural ODEs

Abstract: Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications, but it still remains as a great challenge. In particular, the compositional ability to generate novel concept combinations is out of reach for most current models. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. To make them scalable to high-resolution image generation, we introduce an EBM in the latent space of a pre-trained generative model such as StyleGAN. We propose a novel EBM formulation representing the joint distribution of data and attributes together, and we show how sampling from it is formulated as solving an ordinary differential equation (ODE). Given a pre-trained generator, all we need for controllable generation is to train an attribute classifier. Sampling with ODEs is done efficiently in the latent space and is robust to hyperparameters. Thus, our method is simple, fast to train, and efficient to sample. Experimental results show that our method outperforms the state-of-the-art in both conditional sampling and sequential editing. In compositional generation, our method excels at zero-shot generation of unseen attribute combinations. Also, by composing energy functions with logical operators, this work is the first to achieve such compositionality in generating photo-realistic images of resolution 1024x1024.


Controllable and Compositional Generation with Latent-Space Energy-Based Models
Weili Nie NVIDIA wmie@nvidia.cmm com
Arash Vahdat NVIDIA avahdat@nvidia.com
Anima Anandkumar Caltech, NVIDIA anima@caltech, edu
Abstract
Controllable generation is one of the key requirements for successful adoption deep generative models in real-world applications, but still remains as a great challenge. In particular, the compositional ability to generate novel concept combinations is out of reach for most current models. this work, we use energy- based models (EBMs) to handle compositional generation over set of attributes. To make them scalable to high-resolution image generation, we introduce an EBM in the latent space of a pre-trained generative model such StyleGAN. We propose a novel EBM formulation representing the joint distribution of data and attributes together, and show how sampling from it is formulated solving ordinary differential equation (ODE). Given a pre-trained generator, all we need for controllable generation is done efficiently in the latent space and is robust to hyperparameters. Thus, our method simple, fast train, and efficient to sample. Experimental results show that our method outperforms the state-of-tte offte--f-the-arrr in both conditional sampling and sequential editing. In compositional generation, our method excels zero-shot generation of unseen attribute combinations. Also, by composing energy functions with logical operators, this work is the first to achieve such compositionality generating photo-realistic images of resolution 1024x 1024.
train an attribute classifier. Sampling with ODBs
Introduction
Deep generative lcarning has made tremendous progress in image synthesis. High-quality and photorealistic images can now be generated with generative adversarial networks (GANs) [17
28 229]. score-based models [3 [5I), and variational autoencoders (VAEs) 32 54) However,
key requirement for the success of these generative models in many real-world applications the controllability of the generation process. Controllable generation, in particular the compositional ability to generate novel concept combinations, still remains great challenge to current methods.
A common approach for achieving controllable generation is to train conditional model [36 31 where the conditional information, such as semantic attributes, specified during training to guide the generation. There are two major drawbacks with this approach: (i) Since the set of attributes is fixed and pre-defined, it is difficult to introduce new attributes to an existing model. This, in turn, limits the model's 's compositional ability for unseen attributes and their novel combinations [2]. (ii) Training conditional generative models from scratch for new attributes is computationally expensive, and it can be challenging to reach to the same image quality of the uncontrollable counterparts [10
10 41 38,
Recent works have attempted to overcome these issues by first training an unconditional generator, This is often achieved
and then converting it to conditional model with a small cost (26] 20 by discovering semantically meaningful directions in the latent space of the unconditional model. This way, one would pay the most computational cost only once for training the unconditional model. However, these approaches often struggle with compositional generation, in particular with rare
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

combination of the attributes and the introduction of new attributes. An appealing solution to the compositionality problem is to use energy-based models (EBMs) for controllable generation [22 18
13]. This is due to the fact that energy functions representing different semantics can be combined together to form compositional image generators. However, existing approaches train EBMs directly in the pixel space, making them slow to sample and difficult to generate high-resolution images [3]
In this paper, we leverage the compositionaliiy of EBMs and the generative power of state-of-the-ar pre-trained models such as StyleGANs (28] 29 to achieve high-quality controllable and compositional generation. Particularly, we propose an EBM that addresses the problem of controlling an existing generative model instead of generating images directly with EBMs or improving the sampling quality.
Our main contributions
summarized follows:
We propose novel formulation of joint EBM in the latent space of pre-trained generative models for controllable generation. In particular, the EBM training reduces to training a classifier only, and it can be applied to any latent-variable generative model.
Based on our EBM formulation, we propose new sampling method that relies on ordinary different equation (ODE) solvers. The sample quality robust to hyperparameters.
We show that our method is fast train and efficient sample, and it outperforms previous methods in both conditional sampling and sequential editing.
We show strong compositionality with zero-shot generation on unseen attribute combinaions, and with logical compositions of energy functions in generating high-quality images.
Specifically, we build an EBM in the joint space of data and attributes where the marginal data distribution is denoted by an implicit distribution (e.g., a pre-trained GAN generator), and the conditional distribution of attributes, given data, is represented by an attribute classifier. Using the reparameterization trick, we show that our EBM formulation becomes an energy function in the latent space where the latent distribution is known (i.e., a standard Gaussian). This way, we only need to train the classifier in the data space and do the sampling in the latent space. Because our method only requires training a classifier to add controllability, it is conceptually simple and fast-to-train.
For sampling, most existing EBMs rely on the Langevin dynamics (LD) sampling which is often computationally expensive and sensitive to the choice of sampling parameters. Instead, we build on the recent score-based model (51). and define the sampling process in the latent space using the corresponding probability flow ODEs, induced by the reverse diffusion process. Thus, we rely on an ODE solver for sampling from our EBM formulation in the latent space. With its adaptive step size, the ODE sampling in the latent space is efficient and robust to the sampling parameters.
In experiments, we show our method achieves higher image quality and better controllability in both the conditional sampling and sequential editing, compared to various baselines, including StyleFlow I1 and JEM [18. For the training time, we show our method is 25x faster than StyleFlow on the FFHQ data. For the inference time, our sampling is at least 49x and 876x faster than EBMs [18] 12] and score-based models [5T] in the pixel space, respectively, on CIFAR-10. 10. More importantly. our method excels on zero-shot generation with unseen attribute combinations where StyleFlow almost completely fails. By composing energy functions with logical operators [12] our method is the first to show such compositionality in generating photo-realistic images of resolution 1024x 1024.
2
Method
In this section, we will first give an overview of EBMs and then propose our new EBM formulation in the latent space, based on which, we introduce a new sampling method through the ODE solver.
2.1
Easeg models
represent data E IRd by learning an unnormalized probability distribution (e( xx (x) where Ee(x) _ + R is the energy function, parameterized by a neural network. To train an EBM on data distribution Pdata with maximum likelihood estimation (MLE), we can estimate the gradient of the data -olleeiidoood L(0) E Exp [log Po
EBMs 34
(z) as 22]:
VL(0) = E Enp- ( Ea [x) E Epp ( Ex)]
(I)
To sample r from py for training and inference, Langevin dynamics (LD) [5 is applied as follows,
? V. Ex(e)) + nI)
(z()), Dt+1 =
(2)
2

---

where Po is the initial distribution, 1) is the step size. When 17 - 0 and + oo, 24 is guaranteed to follow the probability (o( under some regularity conditions (50]
2.2 Modelling joint EBM in the latent space
To formulate our EBM for controllable generation, we consider the following setting: We have the data z E X' C R and its attribute code с fcr.-.. , C. E C C R", which is n-dimensional vector with each entry c; representing either a discrete mllelass attribute where & E {0, ... m. - 1}, or a continuous attribute where 6 E IR. The goal is to learn a generative model and semantically control its generated samples by manipulating its conditioning attribute code c. To begin with, we define the joint generative model on both the data and attribute as:
(a c) == Ps (p))p((())) x Pa (x))e (x)) -Fo(ola)
where (2) an implicit distribution defined by a pre-trained generator g (such as GANs) in the form g(z) with the latent variable E Z. And, (o(c(ar is conditional distribution on attributes given x, modeled by a conditional energy function Eg(cl1z).
In this paper, we assume that the generator g is fixed and we only train the energy function Eg(c We also assume the attributes are conditionally independent, i.c., Ee(clz) = Σ Eo(c(lz). Since, our goal is to preserve the image generation quality, we design the joint in Eq. (3) such that the
marginal data distribution (2) from our joint satisfies (=(x)= Py This is obtained with an energy function that is normalized up to constant. Thus, we define:
(z) [c]] + log E., exp (f((x;))cc) if c discrete, 1 2n (q: fi(x;))
Fo(cylar) =
cj is continuous
where [z; 0) is the output of multi-class classifier mapping from X' to RT if the i-th attribute is discrete or a regression network mapping from to I if it is continuous. Without loss of generality, we will always call fg (2:; f) a classifier. Here, o² is hyperparameter to adjust the continuous energy. In practice, we set 2 0.01 after normalizing all the continuous attributes to [0, 1].
We cannot use Eq.
3 for sampling, since (2) is defined implicitly in GAN generators. However, in Appendix A. we show that using the reparameterization trick G 58, sampling x from po (z, c)
reparameterized to sampling 2 from (2 c) x e Fe(clo(=)) p(s, and then transferring the samples to the data space using generator x g(z). ). In most generative models, the prior distribution p(z) is a standard Gaussian. Thus, the joint becomes po(2, c) xe Ee(s,e) with the energy function:
Eo(z,c) = Σ Ea(cila()) + 1
(5)
where each conditional energy function (s(c)lo(2)) is given by Eq. (4). For sampling, we can run LD of Eq. (2) in the z-space using the energy function above. At the end of a chain, we pass the final samples the generator g to get the x samples. Since our latent-space EBM is formulated for controllable generation, we term it as LAtent Controllable EBM (LACE) throughout this work.
Inspecting the joint energy Fz c) in Eq. 5 shows that when the unconditional generator g is fixed, the only trainable component is the classifier f (a;; 0) that represents (o Therefore, unlike the previous joint EBMs [18] that train both pz( and po(c( our method only needs to train a classifier for the conditional (c( and does not require sampling with LD in the x-space during training. That is, for controllable generation, we can train the classifier only in the z-space and do the LD sampling in the z-space. This fact makes our method conceptually simple and easy to train.
Note that the considered r--pace for training the classifier does not have to be the pixel space. Instead, we can choose any intermediate layer of the pre-trained generator as the x r--ppace Take StyleGAN2 29 as an example, if we train the classifier in the w-space, then g(z) in Eq. (5) actually corresponds to the mapping network of StyleGAN2, and once we obtain the w samples via the LD sampler, we can pass them to the synthesis network of StyleGAN2 to get the final images. In other words, since data r is a deterministic function of z, the distribution of attributes is uniquely defined by z. We have the freedom to define Pe(c]22) by applying the classifier on top of z or any representation extracted from z including the generator g(2) or its intermediate representations such as w-space in StyleGAN2.
2.3 Sampling through an ODE solver
Sampling with LD requires hand-crafted tricks to speed up and stabilize its convergence Even in the latent space, our experiments show that LD tends to be sensitive to its hyperparameters. In
3

---

this section, we introduce another sampling method, called ODE sampler, that relies on an ODE solver. Our method with the LD and ODE sampler is termed as LACE-LD and LACE-ODE, respectively.
Our idea is inspired by the prior work [51], which shows that controllable generation can be achieved by solving conditional reverse-time SDE Specifically, if consider Variance Preserving (VP) {((()) + Vf(fddw. where w standard Wiener process and Zig Pdata is a data sample. The scalar time-variant diffusion coefficient ((t) has
SDE [51], the forward SDE is defined as dx
a linear form B(t) Suiin + (Smax - (mia) )t, where [0. T7. Then, the conditional sampling from
(o(e)c) is equivalent to solving the following reverse VP-SDE:
dr ((t) + 2V. log c)), + VB(t)dū
(6)
where w is a standard reverse Wiener process when time flows backwards from T to O, and pt (z c) is the join data and attribute distribution at time t. Song et al. [51] show that there exist corre- sponding ODE for the SDE in Eq. (6) which can be used for sampling. Appendix [A.2] using the reparameterization trick, we show that the ODE in the latent space for our model takes a simple form:
1 B(1) Σ V= dz =
2==
with negative time increments from 7 to O. To generate sample x conditioned on all we need to draw =T) ~ N(,, I) and solve the ODE in Eq. (7) using black-box solver. The final =(0) samples are transferred to the data space through the generator Appendix(A.2] [A.2] leads to the simple ODE is that the latent 2 in most generative models follows a standard Gaussian distribution, and diffusing it with VP-SDE will not change its distribution, i.e.,
g(=()).. Our main insight in
Pe (O),
Remarks There are some key observations from our ODE formulation in Eq.
(i) Connections to gradient flows. Similar to 45 3 that build connections between SDE/ODE and gradient flows, our ODE sampler can be seen as a gradient flow that refines a latent 2 from a random noise to a vector conditioned on an attribute vector. While (3] relied on the Maruyama solution of the SDE, we convert our generative SDE to an ODE. Adaptive discretization with a higher order method (c.g.. Runge- Kutta (77) is often preferred when solving ODEs. But our ODE formulation also works well with the first-order Euler method, and its performance lies in-between
LACE-LD and LACE-ODE (See Appendix (1.11.3)
(ii) Advantages of ODE in the latent space. If the SDE/ODE is built the pixel space as (51. requires 1) estimating the score function Va a log (x()),, and 2) training a time-variant classifier pe Pe(clz(()), both of which make its training and inference challenging. Instead, our ODE sampler is much simpler: we do not need to train any score function. We only train a time-invariant classifier. Compared with the LD sampler that uses a fixed step size and is sensitive to many hyperparameters (e.g., step size, noise scale and number of steps), ODE sampler is adaptive in step sizes and only needs to tune the tolerances, making it more efficient and robust to hyperparameters.
3 Experiments
In this section, we show the effectiveness of our method in conditional sampling, sequential editing and compositional generation, and also perform an ablation study on the sampling method.
Experimental setting We use StyleGAN-ADA 27 as the pre-trained model for experiments on CIFAR-10 33 and StyleGAN2 (29 for experiments on FFHQ (28). We train the classifier in the w-space, where our method works best (see ablation studies in Appendix (..5). To train the latent classifier in the w-space, we first generate (image, w) pairs from StyleGAN, and then label each w latent by annotating its paired image with an image classifier (see data preparation in Appendix AA.))
the 'dopri5' solver 77] with the tolerances of (Ie-3, Ic-3), and we set T 1, Amin = 0.1 and 3max = 20. For LACE-LD, the step size and the standard deviation a of c are chosen separately for faster training (18). where the number of steps N = 100, step size 1) = 0.01 and standard deviation a = 0.01. For metrics, we use (i) conditional accuracy (ACC) to measure the controllability, where we generate images using randomly sampled attribute codes, and pass them to a pre-trained image classifier to predict the attribute codes, and (ii) FID to measure the generation quality and diversity 21.. See Appendix
For LACE-ODE,
A.3 A.3 for more details.

---

Table 1: Comparison of our method and baselines for conditional sampling CIFAR-I0. For notations, Train training time, Infer inference time (m: minute, 8: second), which refer to the single GPU time for gen- erating a batch of 64 images, 7 is the LD step size, and
. is the number of predictor steps in the PC sampler.
Methods
Train
Infer
FIDJ, ACC1
Cond-FBM
VE-SDE
JEM [88]
2160m
135s
52.35 0.645
Cond-EBM 12
2280m
24.5s
41.72 0.792
VP-SDE [5
52800m
438s
19.13
0.643
VE-SDE 51
52800m
448s
2.97
0.662
JaM (7/0.1)
21m
0.63s
8.75
0.950
Latent-JEM (p=0.011
21m
0.63s
5.65
0.821
LACE-PC (N=1000
4m
0.84s
2.99
0.747
Latent-JEM 0.1
Ours
LACE-PC (N=200)
4m
1.86s
2.94
0.722
Figure Conditionally generated images of
LACE-LD
4m
0.68s
4.30
0.939
method (LACE-ODE)) and baselines on the plane class
LACE-ODE
4m
0.50s
6.63
0.972
of CIFAR-10. See Appendix A.6 for more results.
3.1 Conditional sampling
Baselines For comparison, we consider a set of baselines: StyleFlow JEM 18. Conditional EBM (Cond-EBM) [12], and SDEs [5]]. We also propose Latent-JEM, a variant of JEM modelled in the latent space, and LACE-PC, which replaces the ODE sampler with the Fredictor-Corrector (PC) sampler that solves the reverse SDE (Eq. 6 [5I]. See the Appendix A.4 for more details.
The results of our method against baselines on CIFAR-10 are shown in Table T and F Our method requires only four minutes to train, and it takes less than one second to sample batch 64 images on a single NVIDIA V100 GPU, which significantly outperforms previous BBMs (at least 49x faster) and score-based models (at least 876 faster) in the pixel space. Latent-JEM that works in the latent space is also slower training as it performs the LD for each parameter update.
CIFAR-10
For the conditional sampling performance, LACE- ODE and LACE-LD largely outperform all the baselines in precisely controlling the generation while maintaining the relatively high image quality. In particular, LACE-PC performs similarly to VE-SDE: they can achieve better image quality but have a problem with precisely controlling the generation (ACC<0.75). Besides, we observe that EBMs in the latent space achieve much better overall performance than EBMs in the pixel space.
To test on FFHQ, we use 10k (w, c) pairs created by for training, where is sampled from the w-space of StyleGAN2. and e the attribute code. Unless otherwise specified, we use truncation vi 0.7 for our method. Following the evaluation protocol from [. we use 1k generated samples from StyleGAN2 to compute the FID. Note that given the small sample size, FID values tend to be high. For the reference, the original unconditional StyleGAN2 with 1k samples has FID 20.87.
FFHQ
The results of comparing method with baselines shown in Table and Figure 2 where condition on glasses and pander_smile, agee respectively. For the training time, our method only takes minutes, which is around 25x faster than StyleFlow, and 5x faster than Latent-JEM Our inference time increases with the number of attributes to control. For instance, LACE-ODE needs similar inference time with StyleFlow (0.68s vs. 0.61s) on glasses, but more inference time than StyleFlow (4.81s Is vs. 0.61s) on mander_smile, age. In practice, we could adjust the ODE tolerances, allowing for trade-offs between inference time and overall performance. Besides, we can also optimize the network to further reduce the inference time (see results in Appendix A.7).
For the controllability, both LACE-ODE and LACE-LD outperform the baselines by large margin. Also, the generation quality of our method is on par with our proposed baselines, and much better than the prior work StyleFlow. For instance, LACE-ODE has much lower FID than StyleFlow (24.52 vs. 43.55). Latent-JEM achieves better FID but always has worse controllability. These quantitative results could be verified by the visual samples in Figure 2 where our method achieves high-quality controllable generation. However, StyleFlow has difficulty with (i) the full controllability in different cases, and (ii) the lack of image diversity specifically when conditioning on more attributes.
3.2 Sequential editing
In sequential editing, we semantically edit the images by changing attribute each time without affecting other attributes and face identity. Given a sequence of attributes (c,,... , ...). ). we adapt our method for sequential editing by relying the compositionality of energy functions. We define the
5

---

(a) glasses=1 (Left: StyleFlow, Right: Ours)
(b) gender- fender-female, female,
smile-1, femelee 55 (Left: StyleFlow, Righr: Ours)
Figure 2: Conditionally generated images of our method (LACE-ODE) and StyleFlow [1] on the glasses and gender_ seceorsssmlle_eeeeee age of FFHQ. respectively. Visually, method achieves higher image generation quality, more diversity and better controllability than StyleFlow.
Table 2: Comparison between method and baselines for conditional sampling the glasses and waweeerssslee__eeeeeeeeee
of FFHQ. respectively. For notations, Train training time, Infer inference time (m: minute, s: second), which refer the single GPU time for generating a batch of images, 17 the LD step size, and N is the number of predictor steps in the PC sampler.
glasses
gender_smile www.smmleea.... age
Methods
Train
Inter
FIDI
ACC₂f
Infer
FID.
ACCxe1
ACC,T
ACC.T
StyleFlow 1
50m
0.61s
42.08
0.899
0.61s
43.88
0.718
0.870
0.874
Latent-JEM (rf=0.1)
15m
0.69s
22.83
0.765
0.93s
22.74
0.878
0,953
0.845
Latent-JEn (y=00.01)
15m
0.69s
21.58
0.750
0.93s
21.98
0.755
0.946
0.831
LACE-PC (N=)00)
2m
1.29s
21.48
0.943
2.65s
24.31
0.951
0.922
0.896
LACE-PC (N=200)
2m
2.20s
21.38
0.925
4.62s
23.86
0.949
0.914
0.894
LACE-LD
2m
1.15s
20.92
0.998
2.40s
22.97
0.955
0.960
0.913
LACE-ODE
2m
0.68s
20.93
0.998
4.81s
24.52
0.969
0.982
0.914
joint energy function of the i-th edit as
F (z, (e))) == Fo(z, (col)-1) + pd( 1-1 + +7 Σ j>f ((1(g(());0),
where the joint energy function Ez, {c:) is from Eq. 5 and fz(:0) is the classifier output for the j-th attribute, d(z, 1) ll - - 2 + +|2 2 prevents from moving too far from the previous 2;-1, and the last term penalizes a for changing other attributes, with d(-.-) defined as the squared L2 norm. Note that in the i-th edit, we use - the new initial point of the sampling for faster convergence. By default, we set = 0.04 and y = 0.01.
We compare our method against StyleFlow M. the state-of-the-att in sequential editing, with two additional metrics that quantify the disentanglement of the edits: (i) the face identity loss (ID) [ 42). which calculates the distance between the image embeddings before and after editing to measure the identity preservation. (ii) the disentangled edit strength (DES), defined DES E DES, and DES, = Ep)(a)) A - max Ay]]. where A 1 ACCCo
ACC,-ACCo. denotes the normalized ACC
improvement, with ACC, and ACC,, being the ACC score i-th attribute after and before the i-th edit, respectively. Intuitively, the maximum DES is achieved when each edit precisely control the considered attribute only but leave other attributes unchanged, leading to good disentanglement.
Table 3 shows the quantitative results, where the truncation coefficient v = 0.5, and we apply the subset selection strategy as proposed in StyleFlow 1 to alleviate the background change and the reweighting of energy functions (see Appendix A.3 for details) for better disentanglement, Our method outperforms StyleFlow in terms of disentanglement (DES), identity preservation (ID), image quality (FID) and controllability (ACC). This is confirmed by the qualitative results in Figure 3 where

---

Table Comparison between our method StyleFlow sequential editing on FFHQ, where we edit cach attribute in the sequence of [yau, smile, age, glasses]. Note that here we only show the final performance after all edits, and the results of every individual edit and ablation studies are deferred to Appendix A.8
All
Methods
DESt
ID1
FID,
ACC,T
ACC,T
ACC,
ACCaT
StyleFlow O
0.569
0.549
44.13
0.947
0.773
0.817
0.876
LACE-ODE
0.735
0.501
27.94
0,938
0.956
0.881
0.997
Original
*yow-left
+smile=1
5ggo-55
*glasees1
MALEPIAIS
Original
*yaw-froot
+smile=0
28ge-28
*glassos
Figure 3: Sequentially editing images with our method (LACF-ODE) and StyleFlow ① on FFHQ with a sequence of Iyaw, smile, age, glasses]. Our method can successfully perform each edit while less affecting the other attributes. On the contrary, StyleFllow may unintentionally modify/lose glasses, largely change the f'ace identity, or have smiling face when is not supposed to.
StyleFlow usually has the following issues: (i) changing unedited attributes, such as accidentally modifying or losing glasses and changing face identities, and (ii) having incomplete edits, for example, the smiling face still appears after setting smile=t (last row in Figure3). On the contrary, our method suffers less from the above problems. Moreover, as the sequential editing in our method is defined by simply composing energy functions, it can incorporate novel attributes a plug-and-play way.
3.3 Compositional generation
Zero-shot generation The goal is to generate novel images conditioned on unseen combinations of attributes that are not present in the training data, which used for evaluating a model's composition- ality in controllable generation. We compare our method against StyleFlow on zero-shot generation in Figure 4 where we condition on (a) (beard=1, smile=0, glasses=1, age=15) and (b) igender=female, smile=0, glasses=1, age=10), respectively. We can see our method still performs well zero-shot generation while StyleFlow suffers from a severe deterioration of image quality and diversity, and almost completely fails in the controllability. Quantitatively, the ACC scores our method are much larger than StyleFlow, as we have 0.935 vs. 0.679 (smile), and 0.982 vs. 0.694 (glasses) in the setting (a), and 0.906 vs. 0.408 (smile), and 0.939 vs. 0.536 (glasses) in the setting (b). These results clearly demonstrate the strong compositionality of our method.
Compositions energy functions EBMs have shown great potential concept compositionality with various ways of composing energy functions [2] Our method can inherit their compositionality.
while achieving high image quality. Inspired by [12. we also consider composing energy functions
with three logical operators: conjunction (AND), disjunction (OR) and negation (NOT). In particular,
7

---

(a) beard-1, smile=0, 0mile=0,glasses-1 glasses=1, age-15 (Left: StyleFlow, Right: Ours)
(b) gender=female, smile=0, Alasses==1,ge== (Left: StyleFlow, Right: Ours)
Figure 4: Zero-shot conditional sampling results of our method and StyleFlow [ FFHQ. where of the above two attribute combinations seen in the training set. Our method performs well zero-shot generation while StyleFlow almost fails by either generating low low-quality images or missing the conditioning information.
given two attributes {ci, c2}, we can define the joint energy function of each logical operator as
E(2, fer AND ea}) = E(cilo(z)) E(calg(=)) + 1/21|=11
E(z, (ei OR e2)) == log (e F(c1lo(=)) + F(r2 ((ralo(=))) + 1/211=112
E(z, {er AND (sar ea]]) = E(ca]g(=)) aE(czlg(=)) + //
where the AND operator actually boils down to the conditional sampling with multiple attributes, and a,8 > 0 are tunable hyperparameters to balance the importance of different energy functions. For more complex compositionality, we can recursively apply these logical operators.
the concept compositionality of our method for glasses and yaw where we set 3 20, and = min( [[(..2]] 1). We can see the generated images not only precisely follow the rule of the given logical operators, but are also sufficiently diverse to cover all possible logical cases. For instance, given (glasses=1 OR yaw=front), some images have glasses regardless of the yaw while other images satisfy yaw=front regardless of the glasses. Our method also works well for the recursive combinations of logical operators, as shown in the bottom-right of Figure 6 To the best of our knowledge, our method is the first to show such strong compositionality when controllably generating photo-realistic images of resolution 1024 x 1024.
0.1
3.4 Ablation study on the ODE and LD sampler
1.0
Here, we carefully examine the ODE and LD samplers with a grid search large range of hyperparameter settings (see Appendix A1 for details). Figure 5 shows the ACC and FID trade-off of the two samplers on CIFAR-10, where each dot in the figure refers to the result of particular hyperparameter setting. After grid research, there are 81 and 104 hyperparam- eter settings for the ODE and LD sampler, respectively.
0.9
0.8
ODE
We can see that (i) the best ACC-FID scores (on the top-left of Figure 5 of the two samplers heavily overlap, implying that they perform equally well in their best tuned hyperpa- rameter settings. (ii) With the concentrated ACC-FID scores for the ODE sampler on the top-left, we can see that the ODE sampler is more stable and less sensitive to the choice of hyperparameters than the LDD sampler. (iii) When focusing
0.3
0
10
20
30
40
50
60
70
80
FID
Figure 5: The ACC-FID distributions for the ODE denotes cach hyperparameter setting.
sampler, where each
on the left with ACC > 95% and FID < 10, the ODE sampler needs much smaller (less than

---

glasses-1 AND yaw-frout
glasses-1 OR yaw-Front
glasses=1 AND (NOT yaw=front)
(glasses=1 AND yaw-froot) OR (glasses-O (NOT yaw-front)
Figure 6: Compositions of energy functions in method with different logical operators: conjunction (AND),
disjunction (OR), negation (NOT), and their recursive combinations FFHQ of resolution 1024x 1024.
white
breed-calico eat
agenyoung
breed-british british shorthair cal
yaw-left, gender-male
snireolor-ginger ago-young
yaw right heard-0, haircolomblack
breed-maine hnircolor grey, mood-grumpy
(a) MetFaces of resolution 1024 1024
(b) AFHQ-Cat Cats resolution 512x512
Figure 7: Conditional sampling our method (LACE-ODE)) on (a) MetFaces resolution 1024 1024 and (b) AFHQ-Cats of resolution 512x512, respectively. The generated images are conditioned on either a single attribute or combination of multiple attributes.
21%) number of minimum average steps than the LD sampler (41.8 vs. 200). It shows that given the similar performance, the ODE sampler tends to be more efficient due to its adaptivity in step sizes.
3.5 Results other high-resolution images
MetFaces [27] By applying the image classifier pre-trained the FFHQ data label the painting faces data (i.e., MctFaces), we train latent classifier in the w-space of StyleGAN-ADA on MetFaces. Accordingly, our method can effectively control the generation of painting faces with one or multiple attributes, as shown in Figure 7(a). It demonstrates the robustness of our method to the classification noise and its generalization ability regarding the domain gap between photos and paintings.
Since the original AFHQ- Cats dataset does not contain the ground-trutt attributes (i.e., breed, haircolor, and age, etc.), we have to rely on extra annotators for efficient labeling. Inspired by (15), apply the CL.IP [43 annotate the AFHQ-Cats data by designing proper prompts that contain the controlling attributes. Similarly, we then train latent classifier in the w-space of StyleGAN-ADA AFHQ-Cats apply our method for controllable generation. As shown in Figure 7b), we can effectively control the generation of cat images based on the CLIP annotations.
AFHQ-Cats [11]
4 Related work
Conditional generative models Significant efforts for conditional sampling and image editing have been invested in conditional generative models. In these models, various conditional information can be used to guide the generation process, such class labels [36, (32,, attribute codes (9] 38).
source images 100, text descriptions [4), and semantic maps 41). Training these models costly, in particular for generating high-resolution images. Additionally, adding modifying the conditional information requires re-training the whole network. On the contrary, our method is based on pre- trained unconditional models, and it requires little effort to introduce new conditional information.

---

Latent code manipulation for image editing An alternative approach to conditional GANs is to manipulate the latent code of pre-trained unconditional models, in particular GANs. Some works explore linear manipulations of latent code [26 20 46 while others consider more complicated nonlinear manipulations [7 16 42 Similar to our work, these methods can discover the directions that often correspond to meaningful semantic edits, and need not train conditional generative models from scratch. However, they tend to have issue with compositionality. For instance, StyleFlow [1 specifies fixed length of attributes during training, limiting its ability to generalize to new attributes and novel combinations (Section 3.3). Notably, [67 share the similar idea of optimizing latent variables of generators with classifier. But with the intuition of using generative models as a powerful prior, their classifiers are defined in the pixel space only, making their sampling more challenging. Instead, our method is formulated in a principled way from the EBM perspective (that unifies these methods) and can be defined in various spaces (pixel / w / 2) of a generator.
EBMs and score-based models EBMs [34] and score-based models [48] have been widely used for image generation (59
49 5T, due to their flexibility in probabilistic generative modeling. Recently, many works have applied them for controllable and compositional generation. [18] proposes the joint EBM and shows its ability in class-conditional generation. [12 explores the compositionality of conditional EBMss in controllable generation. [51] proposes a unified framework of score- based models from the SDE perspective, and performs controllable generation by using both a conditional reverse-time SDE and its corresponding ODE. Unlike our method that works in the latent space, they all formulate energy functions or score functions in the pixel space, making them both more challenging to train and orders of magnitude slower to sample for high-resolution images.
Also, many other works have built EBMs or score-based models in the latent space of generative models, most of which, however, focus on improving generation quality with EBMs as structural prior
56 55 58 53 More similar to our work, (0) 40 considers using the conditional EBMs as latent prior (conditioning on labels or attributes) of generator for controllable generation. key difference that they require inferring latent variables from data or sampling from the EBMs using LD during training while we do not need to. This advantage distinguishes our method from these prior methods regarding training speed and inference efficiency.
5 Discussions and limitations
We proposed novel formulation of joint EBM in the latent space of pre- pre-trained generative models for controllable generation. Based on our formulation, all we need for controllability is to train an attribute classifier, and sampling is done in the latent space. Moreover, and adaptive sampling method by formulating it as solving an ODE. Our experimental results showed that our method is fast to train and efficient to sample, and it outperforms state-of-the-art techniques in both conditional sampling and sequential editing. With our strong performance in the zero-shot generation with unseen attribute combinations and compositions of energy functions with logical operators, our method also demonstrated compositionality in generating high-quality images.
proposed more stable
One limitation of this work is that the generation quality is limited by the generative power of the underlying pre-trainee trained generator. Since our EBM can be applied to any latent-variabll model including GANs and VAEs, it is interesting to apply our method to different generative models based on downstream tasks. Another limitation is that training the attribute classifier requires the availability of attribute labels. We believe that advanced semi-supervised and self-supervise techniques 47 can improve the classifier training while reducing the dependency on labels.
6 Broader impact
The method presented in this work enables high-quality controllable image synthesis, built upon pre-trained generative models (c.g.. StyleGAN2 (29). Technically, it inherits the compositionality of
EBMs and overcomes their difficulty in generating nigh-resolution images. Since our method does not train any conditional generator from scratch, it significantly reduces the computational cost of training generators for new conditioning attributes.
On the application side, it shares with other image synthesis tools similar potential benefits and risks, which have been discussed extensively in [ [2). Our method does not produce new images but only guides the generation process of existing generators. Thus, inherits potential biases from the pre-trained generators, e.g. StyleGAN2 trained on the FFHQ dataset 28 On the other hand, by providing a semantic control of image generation with strong compositionaliiy, our method can be used to discover and proactively reduce unknown biases in existing generators.
10

---

References
[1] Abdal, R.. Zhu, P., Mitra, N. J., and Wonka, P. (2021). Styleflow: Attribute-onditioned exploration of images using conditional continuous normalizing flows. ACM Truns. Graph.
[2] Aneja, J., Schwing, A., Kautz, J., and Vahdat, A. (2021). A contrastive learning approach for training variational autoencoder priors. Neural Information Processing Systems (NeurlPS).
[3] Ansari, A. Ang, L.,, and Soh, H. (2020). Refining deep generative models via discriminator gradient flow. arXiv preprint ar Xiv::
[4] Arbel, M., Zhou, L., and Gretton, (2020). Generalized energy based models. In International Conference on Learning Representations.
[5] Bailey, J. (2020). tools of generative art, from flash to neural networks. Art America.
[6] Che, T., Zhang, R., Sohl-Dickstein Larochelle, H., Paull, L., Cao. Y., and Bengio. (2020). Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003, 06060.
[7] Chen, R. T. Q. Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2018). Neural ordinary diffcrential equations. Advances in Neural Information Processing Systems.
[8] Chen, T., Korablith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International conference muchine learning. PMLR.
[9] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I,, and Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference Neural Information Processing Systems.
[10] Choi, Y., Choi, M., Kim, M., Ha, J.,W., Kim, S., and Choo, J. (2018). Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition.
[11] Choi, Uh Y., Yoo. J., and Ha, J.-W. (2020). Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition.
[12] Du, Y., S., and Mordatch, (2020a). Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems.
[13] Du, Y., Li, S., Tenenbaum, J., and Mordatch, (2020b). Improved contrastive divergence training of energy based models. arXiv preprint 2rXiv:2012.01366
[14] Du, Y. and Mordatch, I. (2019). Implicit generation and generalization in energy- based models. arXiv preprint arXiv:1903.08633
[15] Gabbay, A., Cohen, N., and Hoshen, Y. (2021). An image is worth more than thousand words: Towards disentanglement in the wild. arXiv preprint arXiv:2106.1550
[16] Goetschalckx, I.,, Andonian, A., Oliva, A., and Isola, P. (2019). Ganalyze: Toward visual definitions of cognitive image properties. In Proceedings of the IEEF/CVF International Conference Computer Vision.
[17] Goodfellow, I., Pouget-Abadic J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial ncts. In Advances in neural information processing systems (NeurIPS).
[18] Grathwohl, W., Wang, K.-C., Jacobsen. J.-H., Duvenaud, D.,, Norouzi, M., and Swersky, K. (2020). Your classifier secretly an energy based model and you should treat like one. In Internal Conference of Learning Representations.
[19] Han, T., Lu, Y., Zhu, S.-C., and Wu, Y. N. (2017). Alternating back-propagation for generator network. In Proceedings of the AAAI Conference on Artificial Intelligence.
[20] Härkönen, E.,, Hcrtzmann, A., Lehtinen, J., and Paris, S. (2020). Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems.
[21] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems.

---

[22] Hinton, G. (1999). Products of experts. In Proceedings of the Ninth International Conference on Artificial Neural Networks.
[23] Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in neural information processing systems (NeurIPS).
[24] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andrectto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint / 04861.
[25] Huang, G., Liu, Z.., Van Der Maaten, L.., and Weinberger, K. (2017). Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition.
[26] Jahanian, A., Chai, L.,, and Isola, (2019). On the" steerability" of generative adversarial networks. In International Conference on Learning Representations
[27] Karras, T., Aittala, M.. Hellsten. J., Laine. S., Lehtinen, J., and Aila, (2020a). Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems.
[28] Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[29] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020b), Analyzing and improving the image quality of StyleGAN. In roceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition.
[30] Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:/412.698//
[31] Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M. (2014), Semi sumi-super learning with deep generative models. In Advances in neural information processing systems.
[32] Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes, arXiv preprint urXiv:1312.6114.
[33] Krizhevsky, A., and Hinton, G. (2009). Learning multiple layers of features from tiny images. (Technical Report) University of Toronto.
[34] LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. (2006). A tutorial on energy-based lcarning. Predicting structured data, 1(0).
[35] Liu, 7., Luo, P., Wang, X., and Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV).
[36] Mirza, M. and Osindero, S. (2014), Conditional generative adversarial nets. 411.4:1411.1784.
arXiv preprint
[37] Nguyen, A.. Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in nearal information processing systems.
[38] Nie, W., Karras, T., Garg, A., Debnath, S., Patney, A., Patel, A., and Anandkumar, A. (2020). Semi- supervised stylegan for disentanglement learning. In International Conference on Machine Learning. PMLR.
[39] Øksendal, B. (2003). Stochastic differential equations. In Stochastic differential equations, pages 65-84. Springer.
[40] Pang, B., Han, T., Nijkamp, F., 7hu, S.-C., and Wu, Y. N. (2020). learning latent space energy-hased prior model. Advances in Neural Information Processing Systems.
[41] Park, T., Liu, M.-Y., Wang, T.-C., and Zhu, J.-Y. (2019). Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[42] Patashnik, O.. Wu, Z., Shechtman. E., Cohen-Or. D.,, and Lischinski. (2021). Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
[43] Radford, A., Kim, J. W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. et al. (2021). Learning transferable visual models from natural language supervision. arXiv preprint AXXv:2103.00022
12

---

[44] Reed, S., Akata, Z.. Yan, X.. Logeswaran, L.,, Schiele, B., and Lee, H. (2016). Generative adversarial text to image synthesis. In International Conference Machine Learning. PMLR.
[45] Santambrogio, F. (2017). (Euclidean, metric, and Wasserstein] gradient flows: overview. Bulletin of Mathematical Sciences, Scienccs, 7(1):87-154.
[46] Shen, Y. and Zhou, B. (2021). Closed-form factorization of latent semantics in gans. In CVPR.
[47] Sohn, K., Berthelot, D., Li, C.-I.., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., and Raffel,
C. (2020). Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXi::2001.06833
[48] Song. Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems.
[49] Song. Y. and Ermon, S. (2020). Improved techniques for training score-based generative models. In Advances in neural information processing systems.
[50] Song, and Kingma, D. P. (2021). How to train your energy-based models. arXiv preprint 2rXiv:2101.0222
[51] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021). Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations.
[52] Vaccari, C. and Chadwick, A. (2020). Deepfakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news. Social Media+ Society.
[53] Vahdat, A., Andriyash, E., and Macready, W. G. (2018a). DVAE#: Discrete variational autocncoders with relaxed Boltzmann priors. In Neural Information Processing Systems (NeurIPS).
[54] Vahdat, Kautz, J. (2020). NVAE: A deep hierarchical variational autoencoder. Neural Information Processing Systems (NeurIPS).
[55] Vahdat, A., Kreis, K., and Kautz, (2021). Score-based generative modeling latent space, In Neural Information Processing Systems (NeurIPS).
[56] Vahdat, A., Macready, W. G., Bian, Z., Khoshaman, and Andriyash, E. (2018b). DVAE++ Discrete
variational autoencoders with overlapping transformations. In International Conference on Machine Learning (ICML).
[57] Welling, M. and Teh, (2011). Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML).
[58] Xiao, Z., Kreis, K., Kautz, J., Vahdat, A. (2020). Vaebm: A symbiosis between variational autoencoders and energy based models. In International Conference on Learning Representations.
energy-based
[59] Xic, Lu, Y., Zhu, S.-C., and Wu, Y. (2016). A theory of generative convnet. In International Conference on Machine Learning. PMLR.
[60] Xie, J., Zheng, 7.,, Fang, X., 7hu, S.-C., and Wu, N. (2021). Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[61] Zagoruyko, S. and Komodakis, (2016). Wide residual networks. In British Machine Vision Conference.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section T
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
13

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
2.2 and Section 2.3
(b) Did you include complete proofs of all theoretical results? [Yes] We defer the proofs to Appendix A.]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes] We include the code to reproduce our results in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Section 3 and Appendix A.3 for more details.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report the averaged results in Section 3 and the corresponding results with error bars are reported in the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix [A.3]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 3 and Appendix A.4 where we cite the datasets, pre-trained generative models, and the baseline methods we have used.
(b) Did you mention the license of the assets? [Yes] We mention the license of the assets in the submitted code.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 'r using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [No] We only use publicly available datasets (CIFAR-10, FFHQ, MetFaces and AFHQ-Cats), which are standard benchmarks the generative modeling community.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yoVS-SuTWl2", "reviewer": "Reviewer_QDS5", "summary": "Using labelled real data, an EBM is trained in the latent space of a pretrained GAN, rather than directly in pixel space. An ODE solver is used to sample from the EBM rather than Langevin Dynamics, which is shown to be more robust to hyperparameter settings. Using properties of the EBM, conditional generation compositional editing can be demonstrated, including in zero-shot scenarios.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The advantages of this approach are that it operates in latent space rather than pixel space, so training and inference is fast. Also, only the EBM portion is trained and synthesis uses a pretrained GAN, so it can combine the compositionality properties of EBMs with the image quality of recent GANs.\n\nA few points of further clarification:\n\nIn the energy update function Eqn 1, samples $x$ are drawn from both the real data and the model, however in line 160 it is mentioned that the classifier is trained in the w space of StyleGAN, so it seems like the w-space is used as input to the classifier f, which does not naturally exist for real images. How are the real images provided as inputs to the classifier with respect to Eqn 1? \n\nIn the CIFAR10 experiment, the StyleGAN-ada unconditional model FID is reported as 2.92, and conditional as 2.42, whereas in Table 1 the reported FID is slightly higher than that of the pretrained models. What causes this discrepancy, since the same model is used as the image generator?\n\nOverall, the writing is clear and compositional results seems convincing. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and broader impact are addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "nQjfQilGl2o", "reviewer": "Reviewer_vR7r", "summary": "The paper considers the task of obtaining additional control over a pre-trained generative model without re-training it. Specifically, it aims to generate conditional samples from an unconditional StyleGAN by only training a classifier with respect to the conditioning information. To achieve this, it considers the conditional distribution of StyleGAN latent codes. By Bayes Theorem, this conditional distribution is proportional to the unconditional distribution of the latent codes (which has a known standard normal distribution) times the distribution of conditioning information given the corresponding sample. The latter is modeled as a Gibbs distribution with an energy derived from the classifier. Two approaches to sample from this unnormalized distribution are evaluated: (i) Langevin sampling and (ii) Performing gradient descent on the classifier-derived energy. Experiments demonstrate good performance of this approach with significant improvements over StyleFlow for combinations of conditionings which are not in the training data.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper is well motivated by the desire to reuse powerful generative models without having to retrain them, which is quite costly. In particular, it is difficult to anticipate what needs regarding control over a generative model are desired by users and it is therefore necessary to develop techniques that enable customized controls in a computationally efficient post-hoc manner.\n- The approach is simple and shows good performance. Based on this it could serve as an important baseline for face editing works and enjoy wide adaption.\n- Besides good quality of the synthesized results, it achieves the goal of providing flexible control with little computational requirements as demonstrated by comparatively short training and inference times.\n- Especially in the case of rare combinations of conditioning information, the approach demonstrates improved performance over StyleFlow. In general, the formulation of controlled synthesis as a composition of different energies \nprovides a flexible control mechanism.\n\n**Weaknesses**\n- Both the idea that classifiers can be interpreted as (part of) an EBM (Grathwohl, Will et al. “Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One.” 2020) as well as the idea that optimizing latent codes of generators instead of pixels improves the quality when optimizing an energy derived from classification networks (Nguyen, Anh M et al. “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.” 2016) are known. Similarly, the ODE formulation follows [38] (although I'm not really sure why that derivation is necessary to formulate Eq. (10); see points below) and the composition of energies follows [8]. Thus, the main contribution is the evaluation of these techniques in a very specific setting.\n- The paper makes the very restrictive assumption that the latent distribution follows a standard normal distribution. Thus, it assumes that the most difficult part regarding generative modeling is already solved. Moreover, it considers only relatively easy datasets with CIFAR-10 and FFHQ. To quote from Upchurch, P. et al. “Deep Feature Interpolation for Image Content Changes.” 2017: \"Generative models can be much more powerful than linear interpolation, but the current problems (in particular, face attribute editing) which are used to showcase generative approaches are too simple.\" Here, the situation is even more severe because it makes the problem even simpler by relying on StyleGAN, such that one is directly working with a normally distributed, relatively low-dimensional code. Note that for example Nguyen 2016 does not make assumptions regarding normally distributed codes and considers more complex datasets such as ImageNet and MIT Scenes.\n- While it is always interesting to see connections between different approaches and works, I fail to see how the extensive discussion in Sec. 2 of energy based models, langevin sampling and stochastic differential equations is necessary to arrive at Eq. (10). The reverse ODE to Eq. (10) describes gradient flow on the latent code with respect to a classification/regression network. Optimizing inputs with respect to such a network is a very natural idea that has been considered many times in different contexts in the literature (like Nguyen 2016 and Upchurch 2017 but also Goetschalckx, Lore et al. “GANalyze: Toward Visual Definitions of Cognitive Image Properties.” 2019 and similar works on finding interpretable directions in the latent space of GANs, encoding images into this latent space or performing superresolution as in Menon, Sachit et al. “PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models.” 2020). This discussion might be useful if it would unify previous approaches but in its current form it only seems to present a simple approach in an unnecessarily complex way.\n- Regarding the derivation of Eq. (10): For $f(x,t)=-\\frac{1}{2}\\beta(t)x$ and $g(t)=\\sqrt{\\beta(t)}$ (defined in l.137), the reverse SDE according to Eq. (6) in [38] should read $dx = -\\frac{1}{2}\\beta(t) \\[ x + 2 \\nabla_x \\log p_t(x) \\]dt + \\sqrt{\\beta(t)} d\\bar{w}$. Eq. (7) in the paper under review does not contain the factor 2 on the score $\\nabla_x \\log p_t(x)$. Because of this, $z$ and $\\nabla_z \\log p_t(z)$ in Eq. (9) cancel under the assumption that $p_t$ is standard normal. However, with the factor of 2, these terms would not cancel and Eq. (10) would include an additional term $\\frac{1}{2}\\beta(t) z dt$. Intuitively this also makes more sense to me since the latter would account for the fact that the marginal distribution of $z$ is standard normal. In this case, the resulting (reverse) ODE would also directly correspond to the gradient flow on the energy in Eq. (6). So which version is correct and which one was used in the implementation?\n- One differences to other approaches that optimize latent codes is the use of the ODE solver [5] to solve (the reverse) of the gradient flow in Eq. (10) instead of using a simple Euler discretization commonly used for gradient descent. However, there is no ablation regarding this choice. Is it important to use [5] or does a simple Euler discretization work just as well? Similarly, the choice of the time-dependent diffusion coefficient beta is never discussed. What is used for beta? What is the effect of it?\n\n**Comments and Suggestions**\n- I think the paper demonstrates precisely the fact that face attribute editing with the help of pre-trained generative models is too simple: The final \"model\" that is proposed by this work is to optimize latent codes with respect to a desired output from a classification or regression model. Given the good results obtained by this approach is a good indication that this field (face attribute editing) should reconsider its baselines. Thus, I think the paper could become a valuable contribution by pointing this out and presenting its simple and strong approach that can serve as a baseline for future works. I think it is very important to have simple and strong approaches but unfortunately, in its current form, the paper seems to try to hide the simplicity of its approach.\n- Why not formulate directly the distribution of x/z conditioned on c instead of talking about the joint distribution most of the time when the goal is to sample the former?\n- I think the results from the ablation regarding which space to train the classifier on in A.4 are quite interesting and important. This choice seems to have a significant effect on the performance, and I guess it is also the reason for the short training time? How does training time on w space compare to training time on image space?\n\n**Rating**\n- The paper presents a simple approach to utilize a pretrained, unconditional GAN for conditional sampling and demonstrates good results. While it could serve as a simple and strong baseline for face attribute editing, in its current form the paper presents a different message regarding a \"novel formulation of a joint EBM\". Given that the model performs gradient descent on the latent code of StyleGAN this seems like an overclaim and I fail to see the novelty and contribution. The paper must reconsider to what purpose it draws the connection to energy based models. In its current form, this connection is not very natural and rather seems to, unnecessarily, hide a simple approach in a complex derivation.\n- Thus, in my preliminary rating I do not recommend accepting the paper, but I would like to hear why the authors think that the connection to energy based models is important (especially with respect to Eq. (10)) and how this gives the paper more significance than directly stating that a simple baseline which performs gradient descent with respect to a classifer in the latent space of StyleGAN outperforms recently proposed and more complex approaches like StyleFlow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations regarding the need for powerful pre-trained generative models and the availability of labels to train the classifier are addressed and general potential impacts of generative models which are inherited by the presented approach are mentioned.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}, {"review_id": "5wx8RN2EHO1", "reviewer": "Reviewer_K3yB", "summary": "The paper proposes an effective and efficient approach to conditional image generation using GANs. Two main advances are described 1) The use of EBM in latent space that don’t require the retraining of image generators. This allows for very efficient training since the generator can be held fixed. 2) The use of an ODE solver for sampling that is less sensitive to hyperparameters than traditional LD solvers.\n\nResults are shown on CIFAR-10 and FFHQ datasets. This includes results on sequential generation, compositional generation and zero-shot generation.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is borderline. The improvements in efficiency and the novelty of the approach push it slightly over the top, but the limitations in the experimental results keeps the paper from being a clear accept.\n\nPositives:\n\n++ EBM in latent space leads to very efficient training.\n\n\\+ Results show good disentanglement of attributes.\n\n\\+ Both ODE and LD solutions provide good results.\n\n\\+ Results are impressive with respect to StyleFlow\n\n\\+ Paper is well-written and easy to follow.\n\nNegatives:\n\n\\- Inference time is longer than other approaches, but still not excessive.\n\n\\- The experimental set-up was varied from StyleGan paper, so hard to compare results.\n\n\\- It is unclear how inference time scales with the number of conditioned attributes.\n\n\\- No experiments on other high-res datasets except FFHQ.\n\n\\- Results on FFHQ only compare against StyleFlow. The StyleFlow paper offers comparisons to three other methods. This might be okay if the experimental set-up was the same, but it is not.\n\n\\- Results for sequential generation only use a single ordering. Are similar results found with different orderings? Be good to show a figure similar to Fig. 7 in StyleFlow paper.\n\nOther comments:\n\n+ The compositions of energy functions section (Line 239) doesn’t add much to the paper. It’d be better to use this space to compare against more approaches.\n\n+ Results missing in several figures. Figure 2a left, Figure 4b left\n\n+ Since you’re generating 1024x1024 images, it’d be good to show high-res crops (zoom-ins) showing the details of the images.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are well-written. It would be good to cover in societal impact the implications of conditional GANs vs. unconditioned GANs. I.e., conditioned GANs may be more useful for nefarious applications.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "3sCIAPViZaR", "reviewer": "Reviewer_joed", "summary": "This paper studies conditional learning by training an EBM in the latent space of a pre-trained top-down generator such as StyleGANs. The EBM is a joint distribution of data and attributes together, and sampling from it is formulated as solving an ordinary differential equation. Experimental results show that the method outperforms the state-of-the-art in both conditional sampling and sequential editing. The contribution of the paper lies on obtaining state-of-the-art performance of conditional learning by combining existing technologies, such as StyleGAN, latent space energy-based learning, and score-based generative model via stochastic differential equations. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important problem in computer vision, which is learning conditional generative models for controlled image generation. The paper combines the existing state-of-the-art techniques, including StyleGAN, energy-based model, and score-based generative models, to achieve the state-of-the-art performance of conditional generation. The paper is well-written, well-organized, and easy to follow.   ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Despite the state-of-the-art results, the paper has the following limitations.\n\n(1)\tInsufficient novelty. (i) The performance of the proposed methods, i.e., the high quality of the generated images, comes from the pretrained generator of the StyleGANs, and the sampling of the EBM relies on the score-based generative model. Therefore, the results are not surprising because the paper just combines all existing state-of-the-art works together, and such a contribution is very straightforward. Even though the proposed pipeline to train such a conditional model is useful for the application, but the novelty is insufficient for the conference.  (ii) As to the latent space energy-based modeling, the EBM built on the data and attributes actually the posterior distribution of the top-down generator (a latent variable model). Sampling the latent space given the data via Langevin dynamics has been well studied in [1][2][3][4], in which the generator is trained by MLE where sampling the latent space conditioned on the data is required. This step is called MCMC inference. The energy function of the posterior distribution is the equation (6) in the current paper.  That means the latent space energy-based modeling is not new. \n\n[1] Alternating back-propagation for generator network. AAAI-17.\n\n[2]  Learning dynamic generator model by alternating back-propagation through time. AAAI-19  \n\n[3] Cooperative learning of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI 2020)\n\n[4] Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning. (arXiv 2019 or PAMI 2021)\n\n(2) Insufficient references. (i) The paper lacks some discussion of related works [1][2][3][4], which also samples the latent space of the generator by MCMC or uses the same form of latent space EBM as the current manuscript. (ii) missing pioneering works of ConvNet-EBM. The current successful deep energy-based learning framework was first proposed in Xie et al. 16 [5], where deep neural network is proposed to parameterize the energy function, and MLE with Langevin dynamics is used to train the parameters. The current paper does not cite those published prior arts before 2019, such as [5][6][7][8]. This might lead to negative ethical issues. \n\n[5] A Theory of Generative ConvNet (ICML 2016)\n\n[6] Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet (CVPR 2017)\n\n[7] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. (CVPR 2018)\n\n[8] Learning generative ConvNets via multigrid modeling and sampling. (CVPR 2018)\n\n(3)\tInsufficient comparison. The paper might need to compare with [4], which is also related to controlled image generation. \n\nThe limitations of the current paper are the insufficient novelty and insufficient references, thus I suggest a rejection.  \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for controllable image generation based on applying an energy-based model in the latent space of a pre-trained unconditional generative model. The reviews are mixed: three reviewers are in favor of acceptance, while one recommends rejection. After reading the reviews, the rebuttal, the discussion, and the paper itself, below are some key points.

Pros:
1) The method is sensible and simple
2) The method has been evaluated thoroughly and shown to work well compared to the relevant baselines
3) Quite thorough analysis experiments evaluating different design decisions

Cons:
1) Limited novelty - the work mainly combines existing methods and is related to many prior papers
2) High-res experimental results only on one dataset

Overall, the proposed method is not overwhelmingly new, but it is simple and effective. The paper is well written and easy to understand, and the experimental evaluation is fairly thorough. I thus recommend acceptance at this point, but I urge the authors to take the reviewers' comments into account and adjust the final version of the paper accordingly.
Decision: Accept (Poster)