Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discrimination
Authors: Shelly Golan
Roy Ganz
Michael Elad
Venue: NeurIPS 2024 poster (2024)
Keywords: Adversarial Training; Robustness; Energy-Based Models; Classification;
Primary area: generative_models

Abstract: The recently introduced Consistency models pose an efficient alternative to diffusion algorithms, enabling rapid and good quality image synthesis. These methods overcome the slowness of diffusion models by directly mapping noise to data, while maintaining a (relatively) simpler training. Consistency models enable a fast one- or few-step generation, but they typically fall somewhat short in sample quality when compared to their diffusion origins. 
In this work we propose a novel and highly effective technique for post-processing Consistency-based generated images, enhancing their perceptual quality. Our approach utilizes a joint classifier-discriminator model, in which both portions are trained adversarially. While the classifier aims to grade an image based on its assignment to a designated class, the discriminator portion of the very same network leverages the softmax values to assess the proximity of the input image to the targeted data manifold, thereby serving as an Energy-based Model. By employing example-specific projected gradient iterations under the guidance of this joint machine, we refine synthesized images and achieve an improved FID scores on the ImageNet 64x64 dataset for both Consistency-Training and Consistency-Distillation techniques.

Enhancing Consistency-Based Image Generation via Addersariallyy-Trinee Classification and Energy-Based Discrimination
Shelly Golan Technion
Roy Ganz
Technion
shelly. .ow an@cs. technion. ac .c... i1
ganz@campus technion. ac .il
Michael Elad
Technion
eladêcs. wwrcchiion.a.... i1
Abstract
The recently introduced Consistency models pose an efficient alternative to diffu- sion algorithms, enabling rapid and good quality image synthesis. These methods overcome the slowness of diffusion models by directly mapping noise to data, while maintaining (relatively) simpler training. Consistency models enable a fast one- or few-step generation, but they typically fall somewhat short in sample quality when compared to their diffusion origins. In this work we propose novel and highly effeective technique for post-processing Consistency-bast images, enhancing their perceptual quality. Our approach utilizes a joint classifier- discriminator model, in which both portions are trained adversarially. While the classifier aims to grade an image based on its assignment to a designated class, the discriminator portion of the very same network leverages the softmax values to assess the proximity of the input image to the targeted data manifold, thereby serving as Energy-based Model. By employing example-specifie projected gradient iterations under the guidance this joint machine, we refine synthesized images and achieve improved FID scores the ImageNet 64x64 dataset for both Consistency-Trant Training and Maasissency-Distillaate techniques.
generated
Introduction
Diffusion models, also known score-based generative models, 13. set new benchmarks in multiple fields, the prime of which image generation aspect of these models is the iterative sampling process that gradually eliminates noise from an initial random vector, this way carving fair samples from a prior distribution. By controlling the depth of such processes, these methods offer flexible balance between computational effort and sample quality. However, compared to single-step generative models like GANs (9) the iterative generation procedure of diffusion models typically requires 10-2000 times more compute for sample generation (2), (28). [33, causing slow inference and limited real-time applications. Addressing this shortcoming, Consistency models (26) have been developed to expedite the sampling process, offering a faster alternative with one or two-step data generation. Unfortunately, their reliance on an intricate distillation process tends to limit the ultimate quality of the generated images.
27
28
29 have central
Recent work that aims to remedy this performance gap integrates Consistency Models 26 with Generative Adversarial Networks, so as to tackle common GAN issues like mode collapse and unstable training process, thereby enhancing the diversity and stability of generated images 220
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Adversarial Consistency Training (ACT) 17 further refines this approach by incorporating discriminator within the Consistency training framework, improving the generative model's quality by fine-tuning the image generation process for more realistic outputs. These methods offer new ways to overcome limitations in image fidelity, but require long and computationally expensive training.
In this work we propose a novel and highly effective technique for post- post-proeessing Consistency-basd generated images, enhancing their perceptual quality. Our approach is based on joint classifier- discriminator neural network, in which both functionalities are trained adversarially.
The adversarial approach is critical in training process, as we will further explain in section We rely on past research that has shown that deep neural networks can be easily deceived by adversarial attacks [TO], [8]. 30). Therefore, numerous strategies have been developed to make these networks more robust, with adversarial training becoming one of the most favored approaches [21]. [10]
During the analysis of such classifiers, notable phenomenon was discovered: perceptually aligned
gradients [31, [5] This trait implies that adjustments to an image, guided by the robust classifier,
result in changes that are visually consistent with the intended classification.
Our method utilizes the above phenomenon and iteratively modifies the images created by Consistency-mont models using joint robust clasifier-dierrmminato While the classifier aims to maxi- mize the conditional probability of a target class, the discriminator portion of the very same network uses the softmax values to maximize the energy difference between synthesized and observed data, functioning as an Energy-basee based Model 19
In order to refine synthesized images, we employ example-specific projected gradient iterations under the guidance of this joint machine. The process is controlled by an image-dependent early-stopping mechanism, that allows us to selectively refine images that diverge from the observed data distribution, ensuring that those already close to it remain unaffected. To summarize, our contributions are as follows:
1. We show significant quantitative and qualitative improvements in the perceptual quality of synthesized images for both munsistency-Trainin and Consistency-distilllttoo, with an FID boost of 27.48% and 20.96%, respectively.
2. We demonstrate a remarkable capability of adversarial robust classifiers: Serving as a joint classifier- disssfier-dicrrimintoor informed of both synthesized and observed data distributions. This fusion leads to enhanced effectiveness of projected projected-gradient steps, leading to an additional improvement of 11. 11.2% in FID results, over a classifier boosting alone [6].
3. While this work focuses on boosting Consistency based image generation, the proposed approach is general and applicable to any other generative model. While this is beyond the scope of the current work, we do present supporting preliminary such results.
2 Background
2.1 Diffusion Models and Consistency
generate data (e.g., images) by starting from a plain random vector and manipulating it progressively by sequential denoising steps and noise perturbations. More specifically, let (o( denote the Probability Density Function (PDF) of true images. Diffusion models introduce a forward path, in which Po (x( is gradually diffused over the time interval 0 < t S T to a canonical Gaussian, Pr (x) AO, /). This is formulated via Stochastic Differential Equation (SDE) that describes a continuous noise-mixing process,
Diffusion models 29
da, u(xe, t)dt (I)
Here T > 0 is fixed constant, -) and (.) are the drift and diffusion coefficients respectively, and [w,) | denotes the standard Brownian motion. We denote the distribution of I, as px)x). where (1 is (f)-blurred version of (z) due the additive noise in Z'.
Diffusion models are all about designing the reverse process from T to l 0, which noise is filtered gradually to produce high quality images that serve as fair samples from po(z). While the original diffusion techniques [3] offer an SDE-based reverse path, the more recent and more effective alternatives (c.g. DDIM (55 remove the intermediate stochasticity and offer ODE-based

---

reverse process of the form
dr Ulzz, t)dt 2
logpi (ma) v dt.
(2)
Notice the appearance of the Score-Function Vx, 1z.lopp(a in this flow, which is approximated by an image denoiser.
In practice, Diffusion Models are an Euler-Maruyama discretization [16 of the above equation, resulting with chain of iterations that starts with a random canonical Gaussian noise 2T, and gradually moves 2y towards zo by steps of denoising that follow the above equation. The resulting Fo can be considered as fair sample from (z).. assuming that the denoiser well-trained.
Consistency models [26) are heavily inspired by the theory described above, aiming to preserve the end-to-end behavior that takes from ær to Zo. but while expediting the process to single (or few) step. Given a solution trajectory {ære bejo.TT of the above reverse equation, the Consistency function the trained network f: (x'r, L) + xg. This function should have the property of self-consistency: For two temporal points 4, E [0. T] and their corresponding stages in the same trajectory, 2 and T.. the outputs should be the same f t) = f(z,, = Io. thus explaining the name "Consistency". Such models can be trained in two main ways (26). The first method (CD) involves distilling knowledge from a pre-trained diffusion model, allowing it to learn the distribution directly from the more computationally intensive yet perfomerf diffusion process. The second method (CT) trains the function .( directly via a tailored loss, without the need for pre-existing diffusion model.
2.2 Boosting Synthesis via Robust Classifier (BIGROC)
Consider a goal of refining generated images, so as to improve their perceptual quality. BIGROC 6 offers such a post-processing mechanism, by leveraging a robust classifier.
Let fo Rdd > I be classifier that maps images in Rd to scores for C' classes, also known as logits, and denote it's ct output by (z).. These logits aim to have probabilistic interpretation, being related the estimated posterior probability po(c)x)). Starting with an arbitrary image x and taking gradient steps V fzfī(z) in order to maximize (x), one would expect this process to produce an altered image that better resembles the targeted class e. However, when using plain classifier, even if it is very well performing, its gradients behave as a perceptually meaningless noise that is unrelated to the image content. This implies that, while the resulting image would tend to classified to c, its visual appearance would be hardly changed from the original image x. This phenomenon is well-known, giving rise to the wide topic of adversarial attacks on neural networks (21]. 19].
Past research has shown that, as opposed to the above, robust classifiers produce Perceptually Aligned Gradients (PAG) 3I & 77. These gradients are trait of adversarially trained models, their content being consistent with the human visual perception [31) [5] This implies that when used within the gradient steps mentioned above that aim to pull towards class c, robust models are expected to yield meaningful features aligned with the target class, thus modifying æ in perceptually convincing way.
Harnessing the above knowledge, the BIGROC algorithm 66 proposes an iterative process for boosting the perceptual quality synthesized images, assuming that they are classifiable via fe(z). This is obtained by modifying any generated image x so as to maximize the posterior probability of given target class è, i.e, Po (c)x), where Po is modeled by an adversarially trained classifier and è stands for the estimated class the image belongs to. The outcome of this process is modified version of æ that is more aligned with the class from human point of view, thus having a better perceptual quality.
While there are various techniques for generating adversarial examples, the one used in BIGROC is Targeted Projected Gradient Descent 21 a deterministic iterative process that operates as described in Algorithm [ I1 in this algorithm stands for the projection operator that ensures that the resulting image is s-close to the starting image, and is the loss function (c.g., cross-entropy) that defines how the distance between the classifier output and the target label computed. For simplicity, we can omit the loss and use the gradient of the softmax output of the classifier directly,
8,41 11, (8c Vaf6(x 8.)).
(3)
3

---

We would like to draw the readers attention to the fact that BIGROC 6 is totally unaware of the generated images' distribution, which aims improve. This brings naturally towards the proposed strategy in this paper.
Algorithm Targeted Projected Gradient Descent (PGD)
Input:
Robust Classifier fa(x),
Input image and
Algorithm's parameters: radius E, step-size no. of iterations T, and loss function t.
Set dg < 0.
Set é - (x) or get an external assignment for the class.
for t from to T do
51+1 II. (òg aV&e(folx + 6,),
end for
Output: Tade + x. + dr
2.3 Energy-Based Models
The core idea behind Energy-Based Models (EBM) 19 is turning an arbitrary function into probability distribution. Consider a learned energy function Ez(x) R2 - R, where x R is an input image, and it's output is a scalar value in the range (-o0, too). This neural network induces PDF (r) that has a Gibbs form:
po(x) expp exp(-EE(xx))
Za
Clearly, this function assigns probability greater than zero any input. Za is known as the Partition Function normalization factor that ensures that the density integrates to The negative sign exponent implies that low energy values are assigned to highly probable images, while high energy refers to poor looking images.
EBMs are family of algorithms that aim to learn Eg Ex(x) such that the resulting (=(x) aligns with the probability density function of the given training set. Given learned EBM, image synthesis can be obtained by iteratively climbing the function (x)) using Stochastic Gradient Langevin Dynamics (SGLD) 32.. PGD, as described above, offers an appealing deterministic alternative to SGLD, by leveraging loss-function and an additional & e-ball constraint in it's computation.
Energy-based models and image classifiers can be shown have common grounds M. as detailed
in Appendix A Assuming we have a classifier fo(x): RR + RC that maps images to scores for C classes, the probability of the th label is represented using the Softmax function,
exp(f5(x)) Ee exp(fā(x))
Pa (c(c)))
(5)
The normalizing denominator in the above equation can be turned into candidate energy function for an EBM,
C
Σ exp((t(x))
Es(x) -log
(6)
In words, the exponent of this energy is the sum of the logits' exponentials. And so, while the classifier's s assignment decision, po(cla) is independent of the induced energy, Ex(x), this value can be interpreted as the entity that defines the PDF (o(x)) that aims to match given image distribution. In this work we leverage this possibility, and modify it, as discussed hereafter.
3 Proposed Approach
In this work we propose method to enhance the perceptual quality of images generated by Consistency-base based techniques (26). Our approach utilizes robust joint classifierdiscriminate to

---

guide synthesized images to better resemble real images. In section 3.1 we explore this method's s training objectives and technical details. The post-processing procedure, described in section 3.2 uses the guidance of the joint model in an iterative algorithm to refine the realism and visual appeal of synthesized images. In this process, the synthesized images, originating from the generated distribution pa, are modified to align with the real images distribution Pdota-
3.1 Training a Joint Classffer-DDicrinnnati
Our method aims to leverage the Perceptually Aligned Gradients (PAG) property [31. and further extend and improve refinement capabilities of models that possess this trait by introducing the model with the distribution of both po and Pdata- To this end, we empower the classifier model by incorporating a robust discriminator within it, so as to improve guidance performance significantly with this joint model. This dual-training approach allows the proposed model better differentiate and understand the unique features of both real and synthetic images.
3.1.1 Proposed Training objective
While Energy- Baargy--asedddd Models (EBMs) 19]
may align perfectly with our task of improving image quality by modifying out-of-distribution samples to align with the real data manifold, several chal- lenges impact their practical application. These include the sensitivity of EBM training the choice of the energy function used, leading often to training instability and poor generalization performance. EBMs necessarily rely on computationally complex sampling techniques (e.g. MCMC), which add to their overall complexity and instability. More on these matters is brought in Appendix B
In this work we propose joint clasiffer-diseriinnator model that overcomes many of the EBM problems mentioned above. Our model aims to guide generated samples to be more realistic and emphasize visual features aligned with the target class. To do so, propose revised interpretation of the adversarial training objective and a modified view of the energy that serves the EBM part of our model. More specifically, the loss function proposed contains adversarial variants of both Cross-Entropp and Binary-Cross-Be Cross--ntropy components, each targeting specific aspects of the training as explained further below.
Discrimination loss: The Binary-Cross-Entroy (BCE) loss we suggest is formulated as follows:
(0) Exppsar BBC (Ø, c)] + E
where we define (C for an image from class via
(BCE(L,C)
[f (1 I ps-
I ~ Pdata
(8)
This loss admits values in the range [0, oo). For real images, the value (x)) should tend to be high, while for generated ones it should strive to low values. As such, this ingredient behaves as energy function that serves our diseriminator.
In these equations, à represents the adversarially perturbed version of x, a is the sigmoid function, So classifier and e the class to which the sample belongs. In this setting, derived from Pdata serves as an adversarial example designed to deceive the model by pushing it to be "less real". Conversely, derived from x ~ Pe represents synthetic image that has been created by the Consistency model, and was adjusted to appear more realistic. As we show in our ablation study, these adversarial examples are essential the training process.
Note that in both attacks, the energy function of the classifier, as given in Equation 6 is nos the one used for the PGD computation. Rather, we use only the logit magnitude as energy proxy. Similarly to EBMs, we suggest assigning high probability values to real images and low probability values fake ones. Further discussion about the relation between the suggested adversarial training and EBMs can be found in Appendix C
Classification loss: The commonly used Cross-Entron loss is formulated follows:
(cx(0) Encepata [ c)] + En-po (fan( ()),
(9)
where lo c) for an image from class e is defined by soft-max operation,
exp(ff (x))
(x = -log E E c (x))
(10)
5

---

which leads to values in the range [0, oc). 0 for high classification probability and large values for indecisive classification. Here, are the udversarially perturbed samples as described in the paragraph above, in the context of the BCE loss This loss ensures that the classifier can accurately classify real images to their correct classes, despite any potential adversarial modifications, thereby reinforeing the model's resilience to adversarial attacks and maintaining high classification accuracy on real data. Similarly to the BCE loss, this term calculates the CE loss for both the true and the generated images, maintaining the model's ability to classify generated images correctly.
= (C) + LECE The overall loss is simply an addition of the above two expressions, = This formulation ensures that the model learns to distinguish effectively between real and generated images, as well as classifying them correctly, enhancing its ability deal with udversarially altered images and to generalize well across different types of image manipulations. More importantly, due to the adversarial training, the learned model has meaningful gradients (i.e. PAG (II, 5 which are about to play a vital role in our inference process of pushing Consistency generated images to their improved versions. As our training strategy considers both classification and diserimination losses, these gradients carry both improvements geared by class adjustments and a migration from Pe to Pdato.
In our training process, as described above, we initialize the joint model with a robust RN50 network (1]] while replacing its fully connected head with randomly initialized one. As a consequence, we expect this network to inherit it's original classification robustness, which proves to be valuable for the overall image improvement obtained hereafter. Our training stands on the shoulders of this classifier, we update the model parameters with the loss described above using only attacked (real and synthesized) images.
3.1.2 Gradual Adversarial Training
When computing the adversarial attacks for the above losses, we operate differently on real and synthesized images. A real image is modified by the PGD algorithm by maximizing the BBC (z, c) loss, aiming to push it away from Pdtta. This is achieved by ascent directions that require a negative value of in Algorithm| I In contrast, attacks the synthesized images aim to minimize L (z, c) with a positive a in the PGD algorithm.
Referring to the later attacks on the synthesized images, x PGD applies T steps of gradient descent that explore a local area with small around the starting point of each input sample. We suggest to modify the activation of the PGD attack to gradual form, using a large value of e, as explained below. In forming the attacks mentioned above, our method aims to traverse the entire spectrum between fake and real samples, systematically eliminating any miss-classffication of fake images as reals ones by the model.
We propose a strategy of gradually increasing the number of attack steps T throughout the training process. By extending the number of steps and initializing to be relatively large, the PGD attack can move further from the initial starting points, eliminating closer misleading examples first, and then proceeding to more challenging adversarial examples. Moreover, as the model becomes more adept at countering simpler attacks, incrementally increasing the complexity of the attacks ensure that the training continues to challenge the model effectively.
To further improve our training process and ensure the model does not mistakenly classify high- quality but still fake samples as genuine ones, we implement an early stopping mechanism based on the model's 's probability outputs. This mechanism halts the PGD process when the model assigns a higher probability to perturbed fake image being real than it does to a perturbed real image in the same batch. This mechanism prevents the model from believing that fake images are more "real" than the real ones, preserving its' reliability and discriminative power.
3.2 Sampling
As move to inference, aim to harness the PAG obtained by the adversarial training of the joint model, in order to boost the perceptual quality of Consistency generated images. We propose an
One might claim that a different attack should be practiced here, so as to accommodate the classification task. Our experiments show that, while this option can be practiced, it is expected to mildly improve the results, while consuming substantial computational efforts.

---

iterative algorithm designed to adjust images Pa. by maximizing the probability that belongs to class c and simultaneously minimizing the gap between po and Pdota- The choice of the loss function, f, is critical in this part, as will be shown in the ablation study. To utilize the gradients of the joint model we start by proposing the following expression that we minimize via the PGD:
f(=, (c c) (BCE(Z.C) c).
(11)
Notice the marked difference between the above and the loss used for the training. This function operates only on incoming synthesized images I and while knowing their designated class c. The rational in this expression has two pieces: A desire to minimize ( e) so as to align the image with its target class, just as practiced by BIGROC. In addition, (2, c) should strive for a high value, so to make the image as realistic as possible, moving closer Pdata- We remind the reader that LCC (x, c) here refers to synthesized images (see Equation 7 and thus a high value corresponds to high perceptual quality.
In practice, our tests indicate that the above loss tends to stray unstably, as (BC (x, c) admits too high values, diverting from good quality outcomes. A remedy to this effect is the following alternative loss:
(xx,c) (aE c) (LECE(I,C) lo)²
(12)
Here we aim to push (BCE(T, c) towards a reference value to, set in the training process to be
to (BCrpp
(13)
In words, Eo the mean value of the BCE loss for real images. Minimizing the distance between (acw(2 e) and to pushes the modified image be more "real", while also making sure that it does not admit a value above Eo. which represents an exaggerated quality.
The overall algorithm for boosting images Po is described in Algorithm
Algorithm 2 Boosting Images via PGD Guided by Joint
Input:
Robust Joint
fo(z),
Input image æ of target class c, and
Algorithm's parameters: Radius e, Step-size a, No. of iterations T. and loss function
Output: Thoosted +
An alternative approach is to sample using Stochastic Gradient Langevin Dynamics, similarly to M as described in Algorithm 3 To further investigate the properties of SGLD, we explored replacing PGD with SGLD during both inference and training. During training, we found that optimizing with SGLD lacks the robustness achieved with PGD and suffers from convergence difficulties. However, during inference, SGLD surpasses PGD, as shown in Table The benefit of using SGLD during inference can be attributed to its ability to better explore multiple modes compared to methods like PGD due it's stochastic nature.
Algorithm 3 Boosting Images via SGLD Guided by a Joint Classifeer-Discriiindd Discriminator
Input:
Robust Joint ClassifierDDiscriiine
fe(z),
* Input image of target class c, and
Algorithm's 's parameters: Step-size a, No. of iterations T, noise level
for / from to T do
It Tt-1 - a Vz. ff( .o. I)..
end for
Output: IT.
4 Experiments
In this section, we present evaluation of the proposed boosting method. We begin by discussing the results obtained by our method compared to BIGROC (.. Subsequently, we conduct an ablation study to further analyze various components of the model, described in Section 3
7

---

4.1 Experimental Settings
During training, we use the pre-trained robust ResNet-50 (RN50) model [21] as a backbone, with randomly initialized classification head. We configure the parameters of the Projected Gradient Descent (PGD) attack for fake imgges with an of 10.0 and a step size of 0.1, increasing the number of PGD steps every 1,000 training iterations to enhance adversarial robustness over time. For real images, the PGD attack is set with an 1.0, a step size of 0.25 and steps. The model trained on four NVIDIA GeForce RTX 3090 GPUs. The same configuration applies to the training of the Wide-ResNett50- (Wide-RN) model. In the sampling process, we apply the loss function outlined in Equation 12 and the step size is set 0.1. The amount of steps is adjusted according to the generative model we enhance; further technical details can be found in Appendix
Experimental Results
We analyze the perceptual-quality boosting performance using Fréchet Inception Distance 12 Inception Score (23), Precision and Recall. [TJmmarizse our results on the ImageNet 64x64 dataset 3. Initially,
present the original FID and IS results as reported in the Consistency Models paper 266). We then apply our boosting algorithm using both a robust classifier and our joint model. The results demonstrate that our proposed approach offers greater benefits compared to refining with robust classifier alone (i.e., BIGROC (6). Notably, the proposed training of our joint model leads significantly improved results. This boost in image quality is observed across both Consistency Training (CT) and Consistency Distillation (CD) methods, and when using either one or two generative steps. In terms of efficiency, one PGD step requires only 0.02 seconds, while single Consistency (CT or CD) inference step takes 0.55 seconds, making 25 PGD steps comparable in cost to Consistency inference. This suggests that our approach could be interpreted as providing a continuum trade-off between complexity and perceptual quality. For comparison,
[7] used a
GAN-based approach to improve CT synthesis with adversarial training, achieving FID of 10.6, which slightly outperforms BIGROC but falls short of our joint model's performance boost. Note that their training much extensive than ours, and much heavier in computational complexity, due to the size of the CT network and the adversarial loss being used.
Table 1: Quantitative comparison of image generation performance ImageNet 64x64 using Consistency Models and our proposed boosting methods with robust classifiers and joint models.
Boosting Method
Inference Steps
FID (J)
IS (t)
Precision (t)
Recall (f)
Consistency Training
13.00
28.83
0.71
0.41
RN50 Robust Classifier
10.71
36.38
0.70
0.47
Wide-RN Non-Robust Joint
12.30
35.62
0.64
0.46
RN50 Robust Joint
9.60
50.80
0.76
0.39
Wide-RN Robust Joint
8.83
55.67
0.73
0.45
Consistency Training
11.12
27.28
0.68
0.54
RN50 Robust Classifier
9.65
33.55
0.68
0.54
Wide-RN Non-Robust Joint
11.28
33.16
0.63
0.54
RN50 Robust Joint
8.51
39.40
0.74
0.49
Wide-RY RN Robust Joint
7.98
46.33
0.71
0.52
Consistency Distillation
6.20
39.87
0.67
0.63
RN50 Robust Classifier
5.55
44.65
0.68
0.61
Wide-RN RN Non-Robust Joint
6.33
45.55
0.64
0.60
RN50 Robust Joint
4.95
51.98
0.72
0.58
Wide-RN Robust Joint
4.84
58.73
0.70
0.60
Consistency Distillation
4.69
42.28
0.68
0.63
RN50 Robust Classifier
4.20
46.78
0.69
0.63
Wide-RN RN Non-Robust Joint
4.66
46.37
0.68
0.63
RN50 Robust Joint
3.84
51.12
0.72
0.60
Wide-RN Robust Joint
3.78
55.13
0.71
0.61

---

4.3
Ablation Study
As explained in Section 3.2 the choice of the loss function in Algorithm is crucial. The results in illustrate that utilizing only Cross-Entropy loss produces results that are similar to those achieved using BIGROC [ as expected. The joint model demonstrates substantial discriminative power, which further enhances FID scores when Binary Cross-Entropy loss is applied. In exploring the integration of these two components, we experimented with the loss function proposed equation Our findings indicate that this approach improves upon Cross- Ensso-y loss, though it does not consistently surpass Binary Cross-Entropy loss. Consequently, we recommend adopting the loss 12 which shows an additional improvement in FID by approximately 0.1.
Table 2
I
outlined in Equation
Table 2: FID and IS results with varying loss functions as described in section 3.2
Method
Inference Steps
Loss Function
FID (c)
IS (f)
Consistency Training
13.00
28. 28.83
tCE {BC ICE LBCC
10.78
38.97
9.75
44.19
9.71
43.38
Our loss
9.60
50.80
Consistency Training
2
11.10
27.28
lCE LBCE
10.26
35,89
8.59
39.38
IcE - LDE
8.62
39.02
Our loss
8.51
39.40
Consistency Distillation
6.20
39.87
lcE
5.51
47.54
5,00
48.97
IcE LBCE
5.10
48.63
Our loss
4.95
51.98
Consistency Distillation
4.69
42.28
fce 4.17
47.81
ERCE Ice LBCE
3.89
48.72
4.97
48.35
Our loss
3.84
51.12
We also compare sampling methods - Projected Gradient Descent (PGD) and Stochastic Gradient Langevin Dynamics (SGLD) as discussed in Section 3.2 and presented in Table Our experiments reveal that SGLD offers advantages over PGD in certain contexts, particularly in terms of sampling diversity and fidelity.
Table 3: Quantitative comparison of our proposed boosting methods using Wide-Reshi with PGD and SGLD sampling techniques applied to Consistency Models.
Boosting Method
Inference Steps
FID (+)
IS(t)
Precision (t)
Recall (t)
Consistency Training
13.00
28.83
0.71
0.41
Wide-RN+PGD
8.83
55.67
0.73
0.45
Wide-RN+SGLL.
8.71
59.76
0.73
0.46
Consistency Training
2
11.12
27.28
0.68
0.54
Wide-RN+PGD
7.98
46.33
0.71
0.52
Wide-RN+SGLD
7.64
46.83
0.71
0.53
Consistency Distillation
6.20
39.87
0.67
0.63
Wide-RN+PPID
4.84
58.73
0.70
0.60
Wide-RN+SGI.D
4.65
59.00
0.71
0.60
Consistency Distillation
2
4.69
42.28
0.68
0.63
3.78
55.13
0.71
0.61
Wide-RN+SGLD
3.58
56.46
0.72
0.61

---

To assess the generalization capability of our joint model. we evaluated it on images generated by BigGAN and ADM-G at resolutions of 128x128 and 256x256 pixels (see Table Despite being trained exclusively on images from Consistency Models, our joint model demonstrates strong generalization across different generative models and resolutions. This suggests that our model can enhance images from variety of generative sources without requiring retraining or fine-tuning for each specific case.
Table 4: Quantitative comparison for various generative models different resolutions (64x64, 128x128, and 256x256), before and after applying our proposed boosting method using Wide-ResNet
with SGLD.
Boosting Method
FID (1)
IS (f)
Precision (1)
Recall (f)
Adm 64x64
2.61
46.78
0.73
0.63
Wide-RN- -
2.08
58.41
0.75
0.60
BigGAN 64x64
4.06
44.94
0.79
0.48
Wide-RN+SGLLAS
3.67
62.79
0.81
0.45
Iddpm 64x64
2.92
45.62
0.73
0.62
Wide-RN+SGLD
2.25
62.26
0.76
0.59
Adm-G 128x128
2.97
141.47
0.78
0.59
WGGe-RN+SGIID
2.34
271.95
0.80
0.58
BigGAN 128x128
6.02
145.83
0.86
0.34
Wide-RN+-GGLI
5.68
236.88
0.86
0.34
Adm-G 256x256
4.58
186.83
0.81
0.52
Wide-RN+ RN-DGLD
3.17
301.06
0.83
0.53
BigGAN 256x256
7.03
202.64
0.87
0.27
Wide-RN SSLDD
6.16
275.25
0.88
0.28
Conclusion
This work presents novel technique for leveraging the perceptually aligned gradients phenomenon for refining synthesized images. Our approach enhances the perceptual quality of images generated by Consistency-baad based techniques using a robust joint classifier-die disisferrddserrminaoo model. The dual functionality of our model, which serves as both classification and energy-based discrimination, leads significant improvements in image fidelity.
Limitations: In this work we have used the RN50 and Wide-RN 50-2 architectures for the joint
model, commonly employed in robust classification tasks. We note that this architectures relatively simple and small, compared to the more complex ones proposed in recent years. As consequence, our study is unavoidably constrained by the capabilities of the chosen models. Another limitation in our work is the reliance of our training on Consistency generated images only. Expanding the training-set include images from additional generative models could further improve the overall boosting effect that we document in this paper.
10

---

References
[1] Gunjan Aggarwal, Abhishek Sinha, Nupur Kumari, and Mayank Singh. On the benefits of models with pereepttaall--alinne gradients. arXiv preprint 2XXi::0005001499 2020.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition. 2009.
[4] Prafulla Dhariwal and Alexander Quinn Nichol. Diffiusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, 2021.
[5] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations. arXiv preprint arXiv: 00945, 2019.
[6] Roy Ganz and Michael Flad. Bigroc: Boosting image generation via a robust classifier. Transactions on Machine Learning Research, 2022.
[7] Roy Ganz and Michael Elad. Clipag: Towards generator-free text-to-image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages
January 2024.
[8] Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply robust- ness? In International Conference on Machine Learning, 2023.
(9] Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 2014.
[10] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In International Conference on Learning Representationss 2015.
[11] Will Grathwohl, Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2019.
Wang. Been-Kerrik Jacobsen, David Duvenaud, Mohammad
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in Neural Information Processing Systems, 2017.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
[14] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of iiffusion-based generative models. Advances in Neural Information Processing Systems, 2022.
[15] Simran Kaur, Jeremy Cohen, and Zachary C Lipton. Are perceptually-aligned gradients a general property of robust classifier?? arXiv preprint 19 1el:::91000444 2019.
[16] Peter E Kloeden, Eckhard Platen, Peter Kloeden, and Eckhard Platen. Stochastic differential equations. Springer, 1992.
[17] Fei Kong, Jinhao Duan, Lichao Sun, Hao Cheng, Renjing Xu, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Act: Adversarial consistency models. arXiv preprint arXiv:2311.14000 2023.
[18] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security. 2018.
[19] Yann LeCun, Sumit Chopra, and Raia Hadsell. A tutorial on energy-based learning. In Predicting Structured Data, 2006.

---

[20] Haoye Lu, Yiwei Lu, Dihong Jiang, Spencer Ryan Szabados, Sun Sun, and Yaoliang Yu. Cm- gan: Stabilizing gan training with consistency models. ICML 2023 Workshop on Structured Probabilistic Inference and Generative Modeling.
[21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[22] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, 2021.
[23] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 2016.
[24] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, 2015.
[25] Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
[26] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference Machine Learning, 2023.
[27] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 2019.
[28] Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. Advances in Neural Information Processing Systems, 2020.
[29] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[30] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
[31] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. In International Conference on Learning Representations, 2018.
[32] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International conference on machine learning, 2011.
12

---

A EBM via Classification
classifier fa Rd RC is a function that maps data points into logit vectors that can parameterize a posterior distribution of the form
cxp(fā(z))
exp(ff(x))
po(c|z) E. explfe (x))
(14)
No(x)
where we have defined the normalizing factor by No(x) = Ee= exp(f (I)) Axiomatically, let us use the logits of the classifier, as is, order define a joint probability (z)
po( exp(ff(x))
(15)
No
where No is yet another normalizing factor. Note that while the above two equations share the same nominator, their denominators are markedly different, as Ng normalizes over all c and æ, and thus No = [2 (z)
By marginalizing the joint probability (o e) over c, and assuming uniform probability for the classes, p(c) = 1/C for all c [1.2..... CO one can obtain the expression for pæ(z):
(x Ee exp(fā(x))
(16)
No
However, an EBM probability is defined generally by
po(x) exp(( Fo(()))))))))
(17)
Ze
and therefore, by matching terms in the above two equations we get
c
Es(x) = -log Σ
expls((x))
(18)
as suggested in Section 2.3 Note that these derivations enable us construct pe (c(z)) po(x.c)
pe(#)
When doing so, the normalizing factor cancels out, and we obtain the regular softmax.
B Challenges of EBMs
EBMs often require careful tuning of their learning parameters, the energy landscape they model can be complex and difficult to navigate. This complexity can lead to issues with training stability, where the model may fail to converge converge to undesirable local minima, resulting in poor generalization of new data. Additionally, EBMs are sensitive to the choice of the energy function. An inadequately defined energy function lead to model that either assigns high probability to non-data-like samples or fails to capture the diversity of the dataset, thereby affecting the quality of the generated images. The training objective usually used in EBMs aims adjust (a given in equation (4) such that the model's energy surface aligns with the data distribution, the loss function often employed is based on minimizing the difference between the energy of the real data and the expected energy of the model distribution. The objective is typically formulated as:
(())= Exz-pacca [E)( Ex-ps [E)(
(19)
Where Eg is the energy function and 0 is the model parameters. This expression is often approximated using sampling techniques like Markov Chain Monte Carlo (MCMC) because directly computing expectations under Py is computationally infeasible due to the unknown partition function (()).
EBMs require extensive computational resources, particularly due the need for sampling during training. Sampling methods such as Markov Chain Monte Carlo (MCMC) are computationally expensive and slow, especially when dealing with high-dimensional data like images and the ImageNet dataset in particular. The complexity and variety within Image demand a model that can capture vast range of features and variations, pushing the limits of an EBM's capacity to model such diverse data effectively.
13

---

C EBMs and Adversarial Training
The proposed loss in equation is closely related to GAN's training objective. but can also be leveraged for an energy-based training. Both the EBM loss function (19) and the proposed training objective aim to minimize the energy (or increase the likelihood) of observed data points and maximize the energy (or decrease the likelihood) of generated data. In EBMs, this is done by reducing the energy of real data samples and increasing it for generated samples.
In the logistic regression-basee approach that we take this work, the log-likelihooe of the real data under the model (represented by pe PT (c) -log (a (3reat)) })) minimized, while the log- likelihood of the generated data (represented by pe (Z c) -log(1 a (fō (Zgen))) is maximized. In other words, the energy we use for real/fake is the magnitude of the logit itself, while its normalized value (after soft-max) is the classification probability.
D Supplementary Results
In the following figures we present qualitative results demonstrating that the "boosted" results indeed look better perceptually. The enhancement visible on the edges and textures and leads to sharper and more high-contrast images, creating more perceptually pleasing outcome. Note that we recommend zooming in the images to see the changes clearly.
For reproducibility, we elaborate on the amount of steps and the step size used for boosting each generative model in Table 5
Figure 1: Top: Images generated by Consistency Models trained ImageNet 64x64. Bottom: Refined images by applying our algorithm.
Figure 2: Top: Images generated by Guided Diffusion trained on ImageNet 256x256. Bottom: Refined images by applying our algorithm.
14

---

Figure 3: Top: Images generated by Guided Diffusion trained on ImageNet 256x256. Bottom: Refined images by applying our algorithm.
Figure 4: Top: Images generated by Guided Diffusion trained on ImageNet 128x128. Bottom: Refined images by applying our algorithm.
Table 5: Hyper-parameters used in algorithm
Generative Method
Inference Steps
PGD Steps
Step Size
Consistency Training
35
0.1
Consistency Training
25
0.1
Consistency Distillation
15
0.15
Consistency Distillation
7
0.14
Admnet
250
0.14
BigGAN 2
15
0.1
Iddpm [22]
1000
0.14
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We clearly state novelty and findings and support these by appropriate experiments.
Justification:
Guidelines:
15

---

The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We address the limitations of this work in section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch provide intuition.
16

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss the experimental settings in section 4.2, elaborating on both training and sampling
We also add details in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Guidelines: We intend to share all our code through GitHub after the review process.
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/nips.ce)
for more details.
17

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. http://cpppppccccceeeeeeee/eeee/
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the details can be found in section 4.2 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The method presented is deterministic (PGD steps over chosen images and while leveraging deterministic loss function as guidance), and therefore we do not include error bars our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
18

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sections 4.1 and 4.2 describe these details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Our work operates on Consistency generated images that follow the ImageNet distribution.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work refines set of images generated by Consistency models, enhancing colors and textures. does not alter the visual information in its entirety and thus does no harm.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not a possibility of misuse of our findings.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use public domain data and cite it's origins in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer:
[NA]
Justification: No new assests are introduced in our work.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did crowdsourcing in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No humans were involved in experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "r0vZZmrTLk", "reviewer": "Reviewer_Jjs5", "summary": "The manuscript proposes a method for refining consistency-based image generation using a joint-trained classifier-discriminator. The method helps to improve the quality of the fast-sampling consistency models.", "questions": "see weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written\n\n2. the performance is significant", "weaknesses": "1. The motivation of the work is not clear. The main motivation of the work is to postprocess generated images to improve perceptual quality (line 36-38). In lines 39 - 41, the authors mention the adversarial attacks as the main reason why the perceptual quality is low. There is a gap between these two problems. I agree that an adversarial problem would cause poor alignment, but this phenomenon happens to other models as well, not only the consistency-based model. \n\n2. Furthermore, the proposed method does not only focus on the robustness problem (only in section 3.1.2). The motivation behind joint classifier-discriminator is not clear. Although this scheme is very popular in many generative models, it is not easy to point out its motivation in the context of the manuscript. From line 170 to 175, there is a mention about the drawback of EBMs, yet it is not clear how classifier-discriminator training would solve it.\n\n3. The analysis is lacked. From line 170 to 175, the authors mention about the sensitivity of EBM, instability and poor generalization. However, no evidence is provided. \n\n4. The qualitative improvement is limited and hard to observe. This raises the question whether the method is effective or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and insightful questions. These constructive comments provide valuable perspectives to enhance our work. Additionally, we are grateful for your recognition of the significant performance of our method. \nBelow, we address the reviewer’s specific concerns:\n\n> The motivation of the work is not clear. The main motivation of the work is to postprocess generated images to improve perceptual quality (line 36-38). In lines 39 - 41, the authors mention the adversarial attacks as the main reason why the perceptual quality is low. There is a gap between these two problems. I agree that an adversarial problem would cause poor alignment, but this phenomenon happens to other models as well, not only the consistency-based model.\n\nWe apologize for the lack of clarity in reference to this point. As correctly noted, the primary motivation of our work is to post-process generated images (mainly those created by Consistency models) to enhance their perceptual quality. This is achieved by training yet another network -- a joint classifier-discriminator model. As explained in the paper, this additional model must be trained adversarially in order to enable the desired boost in image quality. The joint classifier-discriminator model is leveraged via the gradients of this network, computed in order to apply an iterative PGD procedure. Please note that in all this process, we do not alter the generative model itself. \n\n>Furthermore, the proposed method does not only focus on the robustness problem (only in section 3.1.2). The motivation behind joint classifier-discriminator is not clear. Although this scheme is very popular in many generative models, it is not easy to point out its motivation in the context of the manuscript. \n\nThe motivation behind the use of a  joint classifier-discriminator model is the following: \n1. Given a classifier, its gradients can be used for pushing the input image towards better matching its referenced class. However, prior work [1] has shown that this is effective only if the classifier is robust, i.e. trained adversarially. \n2. Given an Energy-Based-Model (which we refer to as a `discriminator’), its gradients can be used for pushing the input image towards the image manifold, i.e. being aligned with the distribution of real images. \n3. Merging the above two into a single network, and robustifying both the classifier and the discriminator objectives, we end up with a single joint model which can guide the image improvement much better, which is the essence of this work.\n\nTo validate the effectiveness of our joint classifier-discriminator model, we have included an ablation study [Table 2 in the paper] which demonstrates that enhancing images using only a robust classifier is not as powerful as using our joint model. \n\n>From line 170 to 175, there is a mention about the drawback of EBMs, yet it is not clear how classifier-discriminator training would solve it.\n\n> The analysis is lacked. From line 170 to 175, the authors mention about the sensitivity of EBM, instability and poor generalization. However, no evidence is provided.\n\nEBMs and JEMs are known to be unstable during training. The work reported in [2] highlights these challenges in their limitations section. Similarly, [3] addresses the difficulties of training EBMs. Inspired by these papers, we propose utilizing robustness and deterministic sampling to overcome those issues. \\\nMore specifically, [2] suggests optimizing the energy portion of the classifier using Stochastic Gradient Langevin Dynamics (SGLD) during training. However, they reported encountering instabilities and convergence issues. ⁠In contrast, we found that using Projected Gradient Descent (PGD) along with our alternative loss objective (+ robustness) significantly enhances stability.\n\nTo further investigate the properties of SGLD, we explored the option of replacing PGD during inference and training time. In the training process we found that optimizing with SGLD lacks the robustness achieved with PGD and suffers from difficulties to converge; at inference though it surpasses PGD greatly. These new results will be incorporated into the revised version of the paper.\n\n>The qualitative improvement is limited and hard to observe. This raises the question whether the method is effective or not.\n\nWe appreciate the comment regarding qualitative results. In response, we will include more figures in the paper to illustrate the gain in visual quality. Some of those illustrations can be found in the attached file - figures 1,2. \\\nPlease note that while this improvement is mild in small (64x64) images, we have included new results that refer to larger (128x128 and 256x256) images, and there the benefit is clearly seen. \n\n\nWe hope these additional details and clarifications address the reviewers' concerns.\n\n\n[1] Ganz, Roy, et al. \"BIGRoC: Boosting Image Generation via a Robust Classifier.\" Transactions on Machine Learning Research. 2021. \\\n[2] Grathwohl, Will, et al. \"Your classifier is secretly an energy based model and you should treat it like one.\" International Conference on Learning Representations. 2019.‏\\\n[3] Song, Yang, et al. \"How to Train Your Energy-Based Models.\" arXiv e-prints 2021.‏", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vZvwhMeZn9", "reviewer": "Reviewer_4wrH", "summary": "The paper proposes a method to mitigate the quality degradation of  images generated by consistency-based generative models compared to diffusion models.\n \nThe main idea is to adapt the generated image based on the logits of an adversarial real/synthetic image discriminator to maximize the image resemblance to real images. A single classifier is trained jointly for both discrimnation between real/synthesized images and for assignment to designated categories.\nOptimization in the generative model's output space makes the method more efficient than methods based on updating the model parameters.\n \nThe method is evaluated on the ImageNet 64x64 dataset, showing performance gains in FID and IS over consistency-based models.", "questions": "### Weakness 1:\n1. Other datasets such as ImageNet with higher resolution, COCO or flicker faces would help to judge the method’s performance and generalizability.\n\n### Weakness 2:\n2. While the method outperforms baselines in quantitative evaluation, more visualizations should be provided, especially since the method is about generative models. I would recommend including  a dataset like Flicker Faces for visualizations since human perception is sensitive to artefacts in the human face.\n\n### Typos.:\n3. Line 93 - 'in two main ways []' - missing citations\n4. Line 124 – something went wrong at the beginning of the line, '1' should be 'Alg. 1' for better readability, also maybe the sentence should end after Alg. 1?\n\n### Presentation:\n5. While the paper is clearly written, it would be helpful to have a figure overview of the pipeline, rather than just equations.", "limitations": "Limitations are addressed in a dedicated section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is novel, simple (easy to understand and implement) and outperforms baselines on the evaluation dataset. It has the potential to inspire further research in this direction.\n2. The paper is easy to follow, all the methodology is well justified and well explained, design choices such as the loss function are supported by  ablations\n3. Both CT and CD, with different number of inference steps are evaluated", "weaknesses": "1. Evaluation on a single dataset with small resolution is not sufficient to judge the performance of genrative methods.\n2. Very limited qualitative results\n\nSee questions for more details on both.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we sincerely appreciate the time and effort you dedicated reviewing our paper. Your thoughtful and constructive feedback has been invaluable in highlighting the strengths of our work. Your insightful reviews and acknowledgment of our contributions inspire us to continue refining our work in this field. We address your concerns below:\n\n> Other datasets such as ImageNet with higher resolution, COCO or flicker faces would help to judge the method’s performance and generalizability.\n\nWe appreciate the important comment. In response, we have conducted additional experiments on two high-resolution versions of ImageNet (128 and 256), using both RN-50 and Wide ResNet 50-2.\nThe CT/CD models we have access to are limited to images from ImageNet of size 64x64. However, while our method primarily focuses on Consistency based image generation, it can be applied on images synthesized by any generative model. Therefore, we can evaluate our model on other generative methods at different resolutions. Furthermore, this test can be done, even without any further training or adjustments. We have applied our method to images synthesized by BigGAN and ADM-G at resolutions of 128x128 and 256x256. Our findings are reported in Table 3 in the attached file. \n\nWe also ran experiments on the LSUN dataset (256x256). We applied our method as is (no re-training) and the preliminary results are reported in the table below:\n\n| Method | Dataset | FID |\n| ---- | --- | --- | \n| ADM-G | LSUN-cats | 5.57 | \n| Robust Joint-Model | LSUN-cats | **4.97** | \n| DDPM | LSUN-bedroom | 4.89 |\n| Robust Joint-Model |  LSUN-bedroom | **4.75** |\n| ADM-G | LSUN-horses | 2.95 |\n| Robust Joint-Model |  LSUN-horses | **2.77** |\n\n> While the method outperforms baselines in quantitative evaluation, more visualizations should be provided, especially since the method is about generative models. I would recommend including a dataset like Flicker Faces for visualizations since human perception is sensitive to artefacts in the human face.\n\nWe appreciate the comment regarding qualitative results. We will include more figures in the paper to illustrate the gain in visual quality. Some of those illustrations can be found in the attached file - figures 1,2.\nPlease note that our work refers to data that carries labels for classification (e.g. ImageNet), and thus face images are not currently covered by our approach.  \n\n>Line 93 - 'in two main ways []' - missing citations\n>Line 124 – something went wrong at the beginning of the line, '1' should be 'Alg. 1' for better readability, also maybe the sentence should end after Alg. 1?\n\nThank you for the corrections; we will include them in our revised version.\n\n>While the paper is clearly written, it would be helpful to have a figure overview of the pipeline, rather than just equations.\n\nWe will attach an illustration of both the training and the inference parts of our proposed approach in the revised version of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bSywjHLKyn", "reviewer": "Reviewer_JUbG", "summary": "This paper proposes a post-processing method to improve the generated image quality of consistency models. The method mainly includes a joint classifier-discriminator, which contains an adversarial training objective and a gradual training approach. Experimental results show the effectiveness of the proposed method.", "questions": "Please refer to Weaknesses.", "limitations": "The limitations have been well discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(++) The method can be seamlessly integrated into generative models, facilitating applications.\n\n(+) The writing is easy to follow.", "weaknesses": "(--) The evaluations are limited. Only ImageNet at resolution 64x64 is utilized, which could not be representative since the development trend is high-resolution generation. Datasets that contain larger images should be considered, such as LSUN and FFHQ with resolution larger than 256x256. Meanwhile, more metrics, such as the precision and recall, are also worth considering for better evaluation about the data distribution.\n\nMinor issue:\n1. A typo at line 55: We show a ... improvements ...\n2. The case rules of the journal or conference names in the reference are inconsistent sometimes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we sincerely appreciate the time and effort you dedicated to reviewing our paper. Your thoughtful and constructive feedback has been invaluable in highlighting the strengths of our work.\nWe address your concerns below:\n\n>The evaluations are limited. Only ImageNet at resolution 64x64 is utilized, which could not be representative since the development trend is high-resolution generation. Datasets that contain larger images should be considered, such as LSUN and FFHQ with resolution larger than 256x256. Meanwhile, more metrics, such as the precision and recall, are also worth considering for better evaluation about the data distribution.\n\nWe appreciate the important comment. In response, we have conducted additional experiments on two high-resolution versions of ImageNet (128 and 256), using both RN-50 and Wide ResNet 50-2. \\\nThe CT/CD models we have access to are limited to images from ImageNet of size 64x64. However, while our method primarily focuses on Consistency based image generation, it can be applied on images synthesized by any generative model. Therefore, we can evaluate our model on other generative methods at different resolutions. Furthermore, this test can be done, even without any further training or adjustments, which is exactly the experiment we have conducted and reported herein. We have applied our method to images synthesized by BigGAN and Guided Diffusion (ADM-G) at resolutions of 128x128 and 256x256. Our findings are reported in Table 3 in the attached file. \\\nPlease note that our work refers to data that carries labels for classification (e.g. ImageNet), and thus face images as in FFHQ are not currently covered by our approach. \\\nReferring to the LSUN dataset (256x256), we applied our method as is (no re-training) and the preliminary results are reported in the table below:\n\n| Method | Dataset | FID |\n| ---- | --- | --- | \n| ADM-G | LSUN-cats | 5.57 | \n| Robust Joint-Model | LSUN-cats | **4.97** | \n| DDPM | LSUN-bedroom | 4.89 |\n| Robust Joint-Model |  LSUN-bedroom | **4.75** |\n| ADM-G | LSUN-horses | 2.95 |\n| Robust Joint-Model |  LSUN-horses | **2.77** |\n\nWe will also include precision and recall metrics in all the relevant tables in the paper. \n\n>A typo at line 55: We show a ... improvements ...\n>The case rules of the journal or conference names in the reference are inconsistent sometimes.\n\nThank you for this correction; we will include this in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MmPngkiTpP", "reviewer": "Reviewer_zvWf", "summary": "The paper introduces a post-processing method using a joint classifier-discriminator model to imporve the generation quality of consistency models. This model is trained adversarially, with the robust classifier providing gradient based on class assignment and the discriminator score. This post-processing method consistently improves the quality of the generated images in both single-step and double-step cases, resulting in better FID scores for both consistency distillation and consistency training.", "questions": "Please see the Weakness.", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The method has connections, and it is possible to unify the Robust Classifier and Energy-based Models with consistency models.\n\n- The proposed method is efficient and easy to plug-and-play in consistency distillation and consistency training. \n\n- The paper is overall easy to read and well organized.", "weaknesses": "- Although the method seems to be connected to the Robust Classifier and Energy-based Models, the authors did not give comprehensive theoretical derivation of the connections. The resulted loss is simply linear combination of binary CE and CE. \n\n- Is the proposed Robust Joint-Model specifically useful for consistency model? If so the authors may need to give analysis why this is the case. If not, it worth to study how the proposed method applicable with other generative models. \n\n- How does the radius $\\epsilon$ affects the final performance? Also, how do the other hyper-parameters affect the model performance? The authors should give ablation studies regarding these parts. \n\n- From the empirical results in Table 3, the improvement of the final loss is marginal compared to using CE and BCE only sometimes. \n\nSome minors:\n\n- line 124: algorithm (missed) 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and insightful questions. These constructive comments provide valuable perspectives to enhance our work. Additionally, we are grateful for your recognition of the efficiency of our method. \nBelow, we address the reviewer’s specific concerns:\n\n\n>Although the method seems to be connected to the Robust Classifier and Energy-based Models, the authors did not give comprehensive theoretical derivation of the connections. The resulted loss is simply linear combination of binary CE and CE.\n\nWe appreciate the reviewers' comment and the opportunity to clarify the connections between our method and Robust Classifiers as well as Energy-based Models. Below, we address the specific concerns regarding the theoretical derivation of these connections and the composition of the resultant loss.\nRobust classifiers are integral to our approach as they provide Perceptually Aligned Gradients (PAG), which are crucial for guiding the refinement of synthesized images. The iterative process of modifying images to maximize the conditional probability of a target class relies on the robust classifier's ability to produce meaningful and perceptually consistent gradients. These gradients ensure that changes to the image are visually aligned with the intended class, thereby improving image quality.\nThe discriminator component of our joint model serves an analogous function to an Energy-based Model. It uses the softmax values from the classifier to evaluate the proximity of the input image to the targeted data manifold. The energy function  can be derived from the classifier’s logits as shown in equation 5: \\\n $E_\\theta(x)=-log\\left[\\sum_{c=1}^C exp\\left(f_\\theta^c(x)\\right) \\right]$.\\\nThis formulation is aligned with the principles of Energy-based Models, where lower energy values correspond to higher probabilities of being real images, further justifications can be found in appendix A and C.\nWe acknowledge the reviewer's observation regarding the resultant loss being a linear combination of binary cross-entropy (BCE) and cross-entropy (CE). This design integrates the classification and discrimination tasks into a unified objective. The BCE component is inspired by the familiar GAN loss, focusing on distinguishing real from fake images, while the CE component ensures accurate class alignment. This dual loss structure is essential for harnessing the strengths of both robust classification and energy-based discrimination. We further analyze the relation between the robust classifier and energy-based discrimination empirically and add additional ablation study regarding the importance of the energy-based components. \n\n> Is the proposed Robust Joint-Model specifically useful for consistency model? If so the authors may need to give analysis why this is the case. If not, it worth to study how the proposed method applicable with other generative models.\n\nWhile our primary focus has been the enhancement of consistency models, we acknowledge that the principles underlying our Robust Joint-Model are general. Indeed, several additional experiments that are reported in the paper show that other generative models (ADM-G, BigGAN,...) may benefit from our post-processing algorithm. Furthermore, in all these experiments, we did not retrain the joint classifier-discriminator model, implying that while it has been trained on consistency images, it serves well other generative images. This zero-shot approach has also been practiced in new experiments that we now add, in order to include 128x128 and 256x256 images, generated by bigGAN and ADM-G. \n\n>How does the radius $\\epsilon$ affects the final performance? Also, how do the other hyper-parameters affect the model performance? The authors should give ablation studies regarding these parts.\n\nWe will add ablation studies regarding the effect of the radius on the performance in sampling and training, as this was the only parameter we tuned during training, the rest had negligible influence on the final results. The hyper-parameters we chose are specified in section 4.1. \nAs noted in the paper, for real images, the PGD attack was set with a radius of 1.0, chosen after experimenting with different values, as described in the table below. \n\n| Method | $\\varepsilon$ | FID | IS | \n| ---- | :---: | --- | --- |\n|CT (1 step) | 1.0 | 9.60 | 50.80 | \n|    | 2.0 | 9.98 | 40.96 |\n|    | 3.0 | 10.23 | 39.12 |\n| CD (1 step) | 1.0 | 4.95 | 51.98 |\n|      | 2.0 | 5.13 | 47.28 |\n|       | 3.0 | 5.24 | 46.25 |\n\nWe will provide extended results in our revised manuscript. \\\nRegarding the sampling process, we have provided the PGD parameters used in Table 4 (Appendix D) of our paper. We will include further analysis of the results obtained with different numbers of steps in our revised version.\n\n> From the empirical results in Table 3, the improvement of the final loss is marginal compared to using CE and BCE only sometimes.\n\nIn Table 3 we show the results for $\\ell_{CE}$ alone, $\\ell_{BCE}$ alone, the two together, and our loss that incorporates the above with a target value for the BCE loss - the last one is referred to in the table as “Our Loss”. \n\nAs for the improvements between these options: \n1. Our Binary Cross-Entropy (BCE) component is derived from the logits produced by the classifier. This means that it inherently utilizes the knowledge distilled from the classifier through the Cross-Entropy (CE) component. Consequently, the BCE component incorporates significantly more information than the CE component alone. This synergy between BCE and CE ensures that our model benefits from a comprehensive understanding of both the classification and discrimination tasks.\n2. The proposed final loss achieves an additional improvement of up to 0.15 FID over the plain $\\ell_{CE}-\\ell_{BCE}$ loss. Although this is not a major improvement, we should note that it does not require additional computational resources at all, compared to the simpler alternative.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' constructive feedback and their recognition of our method's novelty and simplicity (R2), as well as its potential for integration into various generative models (R2, R3, R4). We have carefully considered the main issues raised by the reviewers and have taken steps to address them.

We agree that providing more extensive quantitative and qualitative experiments is crucial for establishing the effectiveness of our approach. To this end, we have conducted additional experiments on two high-resolution versions of ImageNet (128 and 256), using both RN-50 and Wide ResNet 50-2. These results are reported in the attached file. \
Our findings indicate that the Wide ResNet 50-2 outperforms the RN-50 and the baselines, leading to substantial quantitative improvements, particularly in enhancing low-level details in high-resolution images. These improvements are reflected in the additional qualitative evaluations presented in the attached file.

Additionally, we explored the effects of Stochastic Gradient Langevin Dynamics (SGLD), which are typically used in the Energy-Based Models (EBMs) training and inference. Our findings indicate that replacing the PGD algorithm in the sampling process (described in section 3.2) with SGLD is beneficial and achieves significantly better results. However, training the joint model with SGLD led to poor performance. These results support our assumption that adversarial training is crucial to the process. We will include a detailed analysis in our revised version of the paper.

Meta-review: The paper proposes enhancing the realism of images from consistency-based generative models by using an adversarial discriminator's logits to optimize the output space, improving efficiency over parameter adjustments. A single classifier is trained for both real/synthetic discrimination and category assignment. Overall, the reviewers are satisfied with the authors' response. A minor concern is that the qualitative results mainly show color enhancements, with limited examples of significant improvements, such as detailed animal faces. It is better for the authors to clarify this issue further.
Decision: Accept (poster)