Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discrimination

Authors: shelly  golan
roy  ganz
michael  elad
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:49:46

Abstract: The recently introduced Consistency models pose an efficient alternative to diffusion algorithms, enabling rapid and good quality image synthesis. These methods overcome the slowness of diffusion models by directly mapping noise to data, while maintaining a (relatively) simpler training. Consistency models enable a fast one- or few-step generation, but they typically fall somewhat short in sample quality when compared to their diffusion origins. 
In this work we propose a novel and highly effective technique for post-processing Consistency-based generated images, enhancing their perceptual quality. Our approach utilizes a joint classifier-discriminator model, in which both portions are trained adversarially. While the classifier aims to grade an image based on its assignment to a designated class, the discriminator portion of the very same network leverages the softmax values to assess the proximity of the input image to the targeted data manifold, thereby serving as an Energy-based Model. By employing example-specific projected gradient iterations under the guidance of this joint machine, we refine synthesized images and achieve an improved FID scores on the ImageNet 64x64 dataset for both Consistency-Training and Consistency-Distillation techniques.

Main paper: 
 introduction: 
 Diffusion models, also known as score-based generative models, [13] , [24] , [27] , [28] , [29] have set new benchmarks in multiple fields, the prime of which is image generation [4] , [22] . A central aspect of these models is the iterative sampling process that gradually eliminates noise from an initial random vector, this way carving fair samples from a prior distribution. By controlling the depth of such processes, these methods offer a flexible balance between computational effort and sample quality. However, compared to single-step generative models like GANs [9] , the iterative generation procedure of diffusion models typically requires 10-2000 times more compute for sample generation [29] , [28] , [13] , causing slow inference and limited real-time applications. Addressing this shortcoming, Consistency models [26] have been developed to expedite the sampling process, offering a faster alternative with one or two-step data generation. Unfortunately, their reliance on an intricate distillation process tends to limit the ultimate quality of the generated images.Adversarial Consistency Training (ACT) [17] further refines this approach by incorporating a discriminator within the Consistency training framework, improving the generative model's quality by fine-tuning the image generation process for more realistic outputs. These methods offer new ways to overcome limitations in image fidelity, but require long and computationally expensive training.In this work we propose a novel and highly effective technique for post-processing Consistency-based generated images, enhancing their perceptual quality. Our approach is based on a joint classifierdiscriminator neural network, in which both functionalities are trained adversarially.The adversarial approach is critical in our training process, as we will further explain in section 3. We rely on past research that has shown that deep neural networks can be easily deceived by adversarial attacks [10] , [18] , [30] . Therefore, numerous strategies have been developed to make these networks more robust, with adversarial training becoming one of the most favored approaches [21] , [10] . During the analysis of such classifiers, a notable phenomenon was discovered: perceptually aligned gradients [31] , [5] . This trait implies that adjustments to an image, guided by the robust classifier, result in changes that are visually consistent with the intended classification.Our method utilizes the above phenomenon and iteratively modifies the images created by Consistency-models using a joint robust classifier-discriminator. While the classifier aims to maximize the conditional probability of a target class, the discriminator portion of the very same network uses the softmax values to maximize the energy difference between synthesized and observed data, functioning as an Energy-based Model [19] .In order to refine synthesized images, we employ example-specific projected gradient iterations under the guidance of this joint machine. The process is controlled by an image-dependent early-stopping mechanism, that allows us to selectively refine images that diverge from the observed data distribution, ensuring that those already close to it remain unaffected. To summarize, our contributions are as follows:1. We show significant quantitative and qualitative improvements in the perceptual quality of synthesized images for both Consistency-Training and Consistency-distillation, with an FID boost of 27.48% and 20.96%, respectively.2. We demonstrate a remarkable capability of adversarial robust classifiers: Serving as a joint classifier-discriminator informed of both synthesized and observed data distributions. This fusion leads to enhanced effectiveness of projected-gradient steps, leading to an additional improvement of 11.2% in FID results, over a classifier boosting alone [6] .3. While this work focuses on boosting Consistency based image generation, the proposed approach is general and applicable to any other generative model. While this is beyond the scope of the current work, we do present supporting preliminary such results.2 Background 
diffusion models and consistency: 
 Diffusion models [29] , [14] generate data (e.g., images) by starting from a plain random vector and manipulating it progressively by sequential denoising steps and noise perturbations. More specifically, let p 0 (x) denote the Probability Density Function (PDF) of true images. Diffusion models introduce a forward path, in which p 0 (x) is gradually diffused over the time interval 0 ≤ t ≤ T to a canonical Gaussian, p T (x) = N (0, I). This is formulated via a Stochastic Differential Equation (SDE) that describes a continuous noise-mixing process,EQUATIONHere T > 0 is a fixed constant, µ(•, •) and σ(•) are the drift and diffusion coefficients respectively, and {w t } t∈[0,T ] denotes the standard Brownian motion. We denote the distribution of x t as p t (x), where p t (x) is a σ(t)-blurred version of p 0 (x) due to the additive noise in x t .Diffusion models are all about designing the reverse process from t = T to t = 0, in which noise is filtered gradually to produce high quality images that serve as fair samples from p 0 (x). While the original diffusion techniques [13] offer an SDE-based reverse path, the more recent and more effective alternatives (e.g. DDIM [25] ) remove the intermediate stochasticity and offer an ODE-based reverse process of the formEQUATIONNotice the appearance of the Score-Function ∇ xt logp t (x t ) in this flow, which is approximated by an image denoiser.In practice, Diffusion Models are an Euler-Maruyama discretization [16] of the above equation, resulting with a chain of iterations that starts with a random canonical Gaussian noise x T , and gradually moves x t towards x 0 by steps of denoising that follow the above equation. The resulting x 0 can be considered as a fair sample from p 0 (x), assuming that the denoiser is well-trained.Consistency models [26] are heavily inspired by the theory described above, aiming to preserve the end-to-end behavior that takes us from x T to x 0 , but while expediting the process to a single (or a few) step. Given a solution trajectory {x t } t∈[0,T ] of the above reverse equation, the Consistency function is the trained network f : (x t , t) → x 0 . This function should have the property of self-consistency: For two temporal points t, s ∈ [0, T ] and their corresponding stages in the same trajectory, x t and x s , the outputs should be the same f (x t , t) = f (x s , s) = x 0 , thus explaining the name "Consistency". Such models can be trained in two main ways [26] . The first method (CD) involves distilling knowledge from a pre-trained diffusion model, allowing it to learn the distribution directly from the more computationally intensive yet well-performing diffusion process. The second method (CT) trains the function f (•, •) directly via a tailored loss, without the need for a pre-existing diffusion model.Consider a goal of refining generated images, so as to improve their perceptual quality. BIGROC [6] offers such a post-processing mechanism, by leveraging a robust classifier.Let f θ : R d → R C be a classifier that maps images in R d to scores for C classes, also known as logits, and denote it's c th output by f c θ (x). These logits aim to have a probabilistic interpretation, being related to the estimated posterior probability p θ (c|x). Starting with an arbitrary image x and taking gradient steps ∇ x f c θ (x) in order to maximize f c θ (x), one would expect this process to produce an altered image that better resembles the targeted class c. However, when using a plain classifier, even if it is very well performing, its gradients behave as a perceptually meaningless noise that is unrelated to the image content. This implies that, while the resulting image would tend to be classified to c, its visual appearance would be hardly changed from the original image x. This phenomenon is well-known, giving rise to the wide topic of adversarial attacks on neural networks [21] , [9] .Past research has shown that, as opposed to the above, robust classifiers produce Perceptually Aligned Gradients (PAG) [31, 15, 1, 8, 7] . These gradients are a trait of adversarially trained models, their content being consistent with the human visual perception [31] , [5] . This implies that when used within the gradient steps mentioned above that aim to pull x towards class c, robust models are expected to yield meaningful features aligned with the target class, thus modifying x in a perceptually convincing way.Harnessing the above knowledge, the BIGROC algorithm [6] proposes an iterative process for boosting the perceptual quality of synthesized images, assuming that they are classifiable via f θ (x). This is obtained by modifying any generated image x so as to maximize the posterior probability of a given target class ĉ, i.e, p θ (ĉ|x), where p θ is modeled by an adversarially trained classifier and ĉ stands for the estimated class the image belongs to. The outcome of this process is a modified version of x that is more aligned with the class ĉ from a human point of view, thus having a better perceptual quality.While there are various techniques for generating adversarial examples, the one used in BIGROC is Targeted Projected Gradient Descent [21] -a deterministic iterative process that operates as described in Algorithm 1, Π ε in this algorithm stands for the projection operator that ensures that the resulting image is ε-close to the starting image, and ℓ is the loss function (e.g., cross-entropy) that defines how the distance between the classifier output and the target label is computed. For simplicity, we can omit the loss and use the gradient of the softmax output of the classifier directly,EQUATIONWe would like to draw the readers attention to the fact that BIGROC [6] is totally unaware of the generated images' distribution, which it aims to improve. This brings us naturally towards the proposed strategy in this paper.Algorithm 1 Targeted Projected Gradient Descent (PGD)Input: • Robust Classifier f θ (x),• Input image x, and • Algorithm's parameters: radius ε, step-size α, no. of iterations T , and loss function ℓ. Set δ 0 ← 0. Set ĉ = f θ (x) or get an external assignment for the class. for t from 0 to T doδ t+1 = Π ε (δ t -α∇ δ ℓ(f θ (x + δ t ), ĉ)). end for Output: x adv ← x + δ TThe core idea behind Energy-Based Models (EBM) [19] is turning an arbitrary function into a probability distribution. Consider a learned energy function E θ (x) : R d → R, where x ∈ R d is an input image, and it's output is a scalar value in the range (-∞, +∞). This neural network induces a PDF p ϕ (x) that has a Gibbs form:EQUATIONClearly, this function assigns a probability greater than zero to any input. Z θ is known as the Partition Function -a normalization factor that ensures that the density integrates to 1. The negative sign in the exponent implies that low energy values are assigned to highly probable images, while high energy refers to poor looking images.EBMs are a family of algorithms that aim to learn E θ (x) such that the resulting p θ (x) aligns with the probability density function of the given training set. Given a learned EBM, image synthesis can be obtained by iteratively climbing the function p θ (x) using Stochastic Gradient Langevin Dynamics (SGLD) [32] . PGD, as described above, offers an appealing deterministic alternative to SGLD, by leveraging a loss-function and an additional ε-ball constraint in it's computation.Energy-based models and image classifiers can be shown to have common grounds [11] , as detailed in Appendix A. Assuming we have a classifier f θ (x) : R d → R C that maps images to scores for C classes, the probability of the c th label is represented using the Softmax function,EQUATIONThe normalizing denominator in the above equation can be turned into a candidate energy function for an EBM,EQUATIONIn words, the exponent of this energy is the sum of the logits' exponentials. And so, while the classifier's assignment decision, p θ (c|x), is independent of the induced energy, E θ (x), this value can be interpreted as the entity that defines the PDF p θ (x) that aims to match a given image distribution.In this work we leverage this possibility, and modify it, as discussed hereafter.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We address the limitations of this work in section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed approach: 
 In this work we propose a method to enhance the perceptual quality of images generated by Consistency-based techniques [26] . Our approach utilizes a robust joint classifier-discriminator to guide synthesized images to better resemble real images. In section 3.1, we explore this method's training objectives and technical details. The post-processing procedure, described in section 3.2, uses the guidance of the joint model in an iterative algorithm to refine the realism and visual appeal of synthesized images. In this process, the synthesized images, originating from the generated distribution p θ , are modified to align with the real images distribution p data .Our method aims to leverage the Perceptually Aligned Gradients (PAG) property [31] , and further extend and improve refinement capabilities of models that possess this trait by introducing the model with the distribution of both p θ and p data . To this end, we empower the classifier model by incorporating a robust discriminator within it, so as to improve guidance performance significantly with this joint model. This dual-training approach allows the proposed model to better differentiate and understand the unique features of both real and synthetic images.While Energy-Based Models (EBMs) [19] may align perfectly with our task of improving image quality by modifying out-of-distribution samples to align with the real data manifold, several challenges impact their practical application. These include the sensitivity of EBM training to the choice of the energy function used, leading often to training instability and poor generalization performance. EBMs necessarily rely on computationally complex sampling techniques (e.g. MCMC), which add to their overall complexity and instability. More on these matters is brought in Appendix B.In this work we propose a joint classifier-discriminator model that overcomes many of the EBM problems mentioned above. Our model aims to guide generated samples to be more realistic and emphasize visual features aligned with the target class. To do so, we propose a revised interpretation of the adversarial training objective and a modified view of the energy that serves the EBM part of our model. More specifically, the loss function proposed contains adversarial variants of both Cross-Entropy and Binary-Cross-Entropy components, each targeting specific aspects of the training as explained further below.The Binary-Cross-Entropy (BCE) loss we suggest is formulated as follows:EQUATIONwhere we define ℓ BCE (x, c) for an image x from class c viaEQUATIONThis loss admits values in the range [0, ∞). For real images, the value f c θ (x) should tend to be high, while for generated ones it should strive to low values. As such, this ingredient behaves as an energy function that serves our discriminator.In these equations, x represents the adversarially perturbed version of x, σ is the sigmoid function, f θ is a classifier and c is the class to which the sample x belongs. In this setting, x derived from x ∼ p data serves as an adversarial example designed to deceive the model by pushing it to be "less real". Conversely, x derived from x ∼ p θ represents a synthetic image that has been created by the Consistency model, and was adjusted to appear more realistic. As we show in our ablation study, these adversarial examples are essential to the training process.Note that in both attacks, the energy function of the classifier, as given in Equation 6, is not the one used for the PGD computation. Rather, we use only the c logit magnitude as an energy proxy. Similarly to EBMs, we suggest assigning high probability values to real images and low probability values to fake ones. Further discussion about the relation between the suggested adversarial training and EBMs can be found in Appendix C.The commonly used Cross-Entropy loss is formulated as follows:EQUATIONwhere ℓ CE (x, c) for an image x from class c is defined by a soft-max operation,EQUATIONwhich leads to values in the range [0, ∞), 0 for high classification probability and large values for indecisive classification. Here, x are the adversarially perturbed samples as described in the paragraph above, in the context of the BCE loss. 1 This loss ensures that the classifier can accurately classify real images to their correct classes, despite any potential adversarial modifications, thereby reinforcing the model's resilience to adversarial attacks and maintaining high classification accuracy on real data. Similarly to the BCE loss, this term calculates the CE loss for both the true and the generated images, maintaining the model's ability to classify generated images correctly.The overall loss is simply an addition of the above two expressions,L(θ) = L CE (θ) + L BCE (θ).This formulation ensures that the model learns to distinguish effectively between real and generated images, as well as classifying them correctly, enhancing its ability to deal with adversarially altered images and to generalize well across different types of image manipulations. More importantly, due to the adversarial training, the learned model has meaningful gradients (i.e., PAG [31] , [5] ), which are about to play a vital role in our inference process of pushing Consistency generated images to their improved versions. As our training strategy considers both classification and discrimination losses, these gradients carry both improvements geared by class adjustments and a migration from p θ to p data .In our training process, as described above, we initialize the joint model with a robust RN50 network [21] , while replacing its fully connected head with a randomly initialized one. As a consequence, we expect this network to inherit it's original classification robustness, which proves to be valuable for the overall image improvement obtained hereafter. Our training stands on the shoulders of this classifier, as we update the model parameters with the loss described above using only attacked (real and synthesized) images.When computing the adversarial attacks for the above losses, we operate differently on real and synthesized images. A real image x is modified by the PGD algorithm by maximizing the ℓ BCE (x, c) loss, aiming to push it away from p data . This is achieved by ascent directions that require a negative value of α in Algorithm 1. In contrast, attacks on the synthesized images aim to minimize ℓ BCE (x, c) with a positive α in the PGD algorithm.Referring to the later attacks on the synthesized images, x ∼ p θ ,PGD applies T steps of gradient descent that explore a local area with a small ε around the starting point of each input sample. We suggest to modify the activation of the PGD attack to a gradual form, using a large value of ε, as explained below. In forming the attacks mentioned above, our method aims to traverse the entire spectrum between fake and real samples, systematically eliminating any miss-classifications of fake images as reals ones by the model.We propose a strategy of gradually increasing the number of attack steps T throughout the training process. By extending the number of steps and initializing ε to be relatively large, the PGD attack can move further from the initial starting points, eliminating closer misleading examples first, and then proceeding to more challenging adversarial examples. Moreover, as the model becomes more adept at countering simpler attacks, incrementally increasing the complexity of the attacks ensure that the training continues to challenge the model effectively.To further improve our training process and ensure the model does not mistakenly classify highquality but still fake samples as genuine ones, we implement an early stopping mechanism based on the model's probability outputs. This mechanism halts the PGD process when the model assigns a higher probability to a perturbed fake image being real than it does to a perturbed real image in the same batch. This mechanism prevents the model from believing that fake images are more "real" than the real ones, preserving its' reliability and discriminative power.As we move to inference, we aim to harness the PAG obtained by the adversarial training of the joint model, in order to boost the perceptual quality of Consistency generated images. We propose an iterative algorithm designed to adjust images x ∼ p θ , by maximizing the probability that x belongs to class c and simultaneously minimizing the gap between p θ and p data . The choice of the loss function, ℓ, is critical in this part, as will be shown in the ablation study. To utilize the gradients of the joint model we start by proposing the following expression that we minimize via the PGD:ℓ(x, c) = ℓ CE (x, c) -ℓ BCE (x, c).(11) Notice the marked difference between the above and the loss used for the training. This function operates only on incoming synthesized images x and while knowing their designated class c. The rational in this expression has two pieces: A desire to minimize ℓ CE (x, c) so as to align the image with its target class, just as practiced by BIGROC. In addition, ℓ BCE (x, c) should strive for a high value, so as to make the image as realistic as possible, moving x closer to p data . We remind the reader that ℓ BCE (x, c) here refers to synthesized images (see Equation 7), and thus a high value corresponds to high perceptual quality.In practice, our tests indicate that the above loss tends to stray unstably, as ℓ BCE (x, c) admits too high values, diverting from good quality outcomes. A remedy to this effect is the following alternative loss:EQUATION)Here we aim to push ℓ BCE (x, c) towards a reference value ℓ 0 , set in the training process to beEQUATION) In words, ℓ 0 is the mean value of the BCE loss for real images. Minimizing the distance between ℓ BCE (x, c) and ℓ 0 pushes the modified image to be more "real", while also making sure that it does not admit a value above ℓ 0 , which represents an exaggerated quality.The overall algorithm for boosting images x ∼ p θ is described in Algorithm 2.Input:• Robust Joint Classifier-Discriminator f θ (x),• Input image x of a target class c, and • Algorithm's parameters: Radius ε, Step-size α, No. of iterations T , and loss function ℓ. Output: x boosted ← T argetedP GD(f θ , ℓ, x, c, ε, α, T ).An alternative approach is to sample using Stochastic Gradient Langevin Dynamics, similarly to [11] , as described in Algorithm 3. To further investigate the properties of SGLD, we explored replacing PGD with SGLD during both inference and training. During training, we found that optimizing with SGLD lacks the robustness achieved with PGD and suffers from convergence difficulties. However, during inference, SGLD surpasses PGD, as shown in Table 3 . The benefit of using SGLD during inference can be attributed to its ability to better explore multiple modes compared to methods like PGD due to it's stochastic nature.Input:• Robust Joint Classifier-Discriminator f θ (x),• Input image x of a target class c, and • Algorithm's parameters:Step-size α, No. of iterations T , noise level σ. for t from 1 to T dox t = x t-1 -α • ∇ xt-1 f θ (x t-1 ) [c] + σ • N (0, I). end for Output: x T .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we present an evaluation of the proposed boosting method. We begin by discussing the results obtained by our method compared to BIGROC [6] . Subsequently, we conduct an ablation study to further analyze various components of the model, as described in Section 3.During training, we use the pre-trained robust ResNet-50 (RN50) model [21] as a backbone, with a randomly initialized classification head. We configure the parameters of the Projected Gradient Descent (PGD) attack for fake images with an ε of 10.0 and a step size of 0.1, increasing the number of PGD steps every 1,000 training iterations to enhance adversarial robustness over time. For real images, the PGD attack is set with an ε of 1.0, a step size of 0.25 and 4 steps. The model is trained on four NVIDIA GeForce RTX 3090 GPUs. The same configuration applies to the training of the Wide-ResNet-50-2 (Wide-RN) model. In the sampling process, we apply the loss function outlined in Equation 12 and the step size is set as 0.1. The amount of steps is adjusted according to the generative model we enhance; further technical details can be found in Appendix D.We analyze the perceptual-quality boosting performance using Fréchet Inception Distance [12] , Inception Score [23] , Precision and Recall. Table 1 summarizes our results on the ImageNet 64×64 dataset [3] . Initially, we present the original FID and IS results as reported in the Consistency Models paper [26] . We then apply our boosting algorithm using both a robust classifier and our joint model. The results demonstrate that our proposed approach offers greater benefits compared to refining with a robust classifier alone (i.e., BIGROC [6] ). Notably, the proposed training of our joint model leads to significantly improved results. This boost in image quality is observed across both Consistency Training (CT) and Consistency Distillation (CD) methods, and when using either one or two generative steps. In terms of efficiency, one PGD step requires only 0.02 seconds, while a single Consistency (CT or CD) inference step takes 0.55 seconds, making 25 PGD steps comparable in cost to a Consistency inference. This suggests that our approach could be interpreted as providing a continuum trade-off between complexity and perceptual quality. For comparison, [17] used a GAN-based approach to improve CT synthesis with adversarial training, achieving an FID of 10.6, which slightly outperforms BIGROC but falls short of our joint model's performance boost. Note that their training is much more extensive than ours, and much heavier in computational complexity, due to the size of the CT network and the adversarial loss being used. As explained in Section 3.2, the choice of the loss function in Algorithm 2 is crucial. The results in Table 2 illustrate that utilizing only Cross-Entropy loss produces results that are similar to those achieved using BIGROC [6] , as expected. The joint model demonstrates substantial discriminative power, which further enhances FID scores when Binary Cross-Entropy loss is applied. In exploring the integration of these two components, we experimented with the loss function proposed in equation 11. Our findings indicate that this approach improves upon Cross-Entropy loss, though it does not consistently surpass Binary Cross-Entropy loss. Consequently, we recommend adopting the loss outlined in Equation 12, which shows an additional improvement in FID by approximately 0.1. We also compare sampling methods -Projected Gradient Descent (PGD) and Stochastic Gradient Langevin Dynamics (SGLD) -as discussed in Section 3.2 and presented in Table 3 . Our experiments reveal that SGLD offers advantages over PGD in certain contexts, particularly in terms of sampling diversity and fidelity. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We discuss the experimental settings in section 4.2, elaborating on both training and sampling hyper-parameters. We also add details in the appendices. 
conclusion: 
 This work presents a novel technique for leveraging the perceptually aligned gradients phenomenon for refining synthesized images. Our approach enhances the perceptual quality of images generated by Consistency-based techniques using a robust joint classifier-discriminator model. The dual functionality of our model, which serves as both classification and energy-based discrimination, leads to significant improvements in image fidelity.Limitations: In this work we have used the RN50 and Wide-RN 50-2 architectures for the joint model, as commonly employed in robust classification tasks. We note that this architectures are relatively simple and small, compared to the more complex ones proposed in recent years. As a consequence, our study is unavoidably constrained by the capabilities of the chosen models. Another limitation in our work is the reliance of our training on Consistency generated images only. Expanding the training-set to include images from additional generative models could further improve the overall boosting effect that we document in this paper.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification:Guidelines: We intend to share all our code through GitHub after the review process.• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Justification: All the details can be found in section 4.2 and in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The method presented is deterministic (PGD steps over chosen images and while leveraging a deterministic loss function as a guidance), and therefore we do not include error bars in our experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: Our work refines a set of images generated by Consistency models, enhancing colors and textures. It does not alter the visual information in its entirety and thus does no harm.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not see a possibility of misuse of our findings. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We use public domain data and cite it's origins in the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: A classifier f θ : R d → R C is a function that maps data points into logit vectors that can parameterize a posterior distribution of the formEQUATION)where we have defined the normalizing factor byN θ (x) = C c ′ =1 exp(f c ′ θ (x)).Axiomatically, let us use the logits of the classifier, as is, in order to define a joint probability p θ (x, c):EQUATION)where N θ is yet another normalizing factor. Note that while the above two equations share the same nominator, their denominators are markedly different, as N θ normalizes over all c and x, and thusN θ = x N θ (x)p(x)dx.By marginalizing the joint probability p θ (x, c) over c, and assuming a uniform probability for the classes, p(c) = 1/C for all c = [1, 2, . . . , C], one can obtain the expression for p θ (x):EQUATION)However, an EBM probability is defined generally byEQUATIONand therefore, by matching terms in the above two equations we getEQUATIONas suggested in Section 2.3. Note that these derivations enable us to construct p θ (c|x) = p θ (x,c) p θ (x) . When doing so, the normalizing factor cancels out, and we obtain the regular softmax.EBMs often require careful tuning of their learning parameters, as the energy landscape they model can be complex and difficult to navigate. This complexity can lead to issues with training stability, where the model may fail to converge or converge to undesirable local minima, resulting in poor generalization of new data. Additionally, EBMs are sensitive to the choice of the energy function. An inadequately defined energy function can lead to a model that either assigns high probability to non-data-like samples or fails to capture the diversity of the dataset, thereby affecting the quality of the generated images. The training objective usually used in EBMs aims to adjust p θ (x) given in equation ( 4) such that the model's energy surface aligns with the data distribution, the loss function often employed is based on minimizing the difference between the energy of the real data and the expected energy of the model distribution. The objective is typically formulated as:EQUATIONWhere E θ is the energy function and θ is the model parameters. This expression is often approximated using sampling techniques like Markov Chain Monte Carlo (MCMC) because directly computing expectations under p θ is computationally infeasible due to the unknown partition function Z(θ).EBMs require extensive computational resources, particularly due to the need for sampling during training. Sampling methods such as Markov Chain Monte Carlo (MCMC) are computationally expensive and slow, especially when dealing with high-dimensional data like images and the ImageNet dataset in particular. The complexity and variety within ImageNet demand a model that can capture a vast range of features and variations, pushing the limits of an EBM's capacity to model such diverse data effectively.The proposed loss in equation ( 7) is closely related to GAN's training objective, but can also be leveraged for an energy-based training. Both the EBM loss function (19) and the proposed training objective aim to minimize the energy (or increase the likelihood) of observed data points and maximize the energy (or decrease the likelihood) of generated data. In EBMs, this is done by reducing the energy of real data samples and increasing it for generated samples.In the logistic regression-based approach that we take in this work, the log-likelihood of the real data under the model (represented by p θ (x real |c) = -log (σ(f c θ (x real ))) is minimized, while the loglikelihood of the generated data (represented by p θ (x gen |c) = -log(1 -σ(f c θ (x gen ))) is maximized. In other words, the energy we use for real/fake is the magnitude of the c logit itself, while its normalized value (after soft-max) is the classification probability.In the following figures we present qualitative results demonstrating that the "boosted" results indeed look better perceptually. The enhancement is visible on the edges and textures and leads to sharper and more high-contrast images, creating a more perceptually pleasing outcome. Note that we recommend zooming in the images to see the changes clearly.For reproducibility, we elaborate on the amount of steps and the step size used for boosting each generative model in Table 5 . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.One might claim that a different attack should be practiced here, so as to accommodate the classification task. Our experiments show that, while this option can be practiced, it is expected to mildly improve the results, while consuming substantial computational efforts. 


Decision: The paper proposes enhancing the realism of images from consistency-based generative models by using an adversarial discriminator's logits to optimize the output space, improving efficiency over parameter adjustments. A single classifier is trained for both real/synthetic discrimination and category assignment. Overall, the reviewers are satisfied with the authors' response. A minor concern is that the qualitative results mainly show color enhancements, with limited examples of significant improvements, such as detailed animal faces. It is better for the authors to clarify this issue further.

Reviews: review: paper_summary: paper_summary: The manuscript proposes a method for refining consistency-based image generation using a joint-trained classifier-discriminator. The method helps to improve the quality of the fast-sampling consistency models.
strength_weakness: strength_weakness: 1. The motivation of the work is not clear. The main motivation of the work is to postprocess generated images to improve perceptual quality (line 36-38). In lines 39 - 41, the authors mention the adversarial attacks as the main reason why the perceptual quality is low. There is a gap between these two problems. I agree that an adversarial problem would cause poor alignment, but this phenomenon happens to other models as well, not only the consistency-based model. 

2. Furthermore, the proposed method does not only focus on the robustness problem (only in section 3.1.2). The motivation behind joint classifier-discriminator is not clear. Although this scheme is very popular in many generative models, it is not easy to point out its motivation in the context of the manuscript. From line 170 to 175, there is a mention about the drawback of EBMs, yet it is not clear how classifier-discriminator training would solve it.

3. The analysis is lacked. From line 170 to 175, the authors mention about the sensitivity of EBM, instability and poor generalization. However, no evidence is provided. 

4. The qualitative improvement is limited and hard to observe. This raises the question whether the method is effective or not.
questions: questions: see weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes a method to mitigate the quality degradation of  images generated by consistency-based generative models compared to diffusion models.
 
The main idea is to adapt the generated image based on the logits of an adversarial real/synthetic image discriminator to maximize the image resemblance to real images. A single classifier is trained jointly for both discrimnation between real/synthesized images and for assignment to designated categories.
Optimization in the generative model's output space makes the method more efficient than methods based on updating the model parameters.
 
The method is evaluated on the ImageNet 64x64 dataset, showing performance gains in FID and IS over consistency-based models.
strength_weakness: strength_weakness: 1. Evaluation on a single dataset with small resolution is not sufficient to judge the performance of genrative methods.
2. Very limited qualitative results

See questions for more details on both.
questions: questions: ### Weakness 1:
1. Other datasets such as ImageNet with higher resolution, COCO or flicker faces would help to judge the method’s performance and generalizability.

### Weakness 2:
2. While the method outperforms baselines in quantitative evaluation, more visualizations should be provided, especially since the method is about generative models. I would recommend including  a dataset like Flicker Faces for visualizations since human perception is sensitive to artefacts in the human face.

### Typos.:
3. Line 93 - 'in two main ways []' - missing citations
4. Line 124 – something went wrong at the beginning of the line, '1' should be 'Alg. 1' for better readability, also maybe the sentence should end after Alg. 1?

### Presentation:
5. While the paper is clearly written, it would be helpful to have a figure overview of the pipeline, rather than just equations.
limitations: limitations: Limitations are addressed in a dedicated section.
paper_summary: paper_summary: This paper proposes a post-processing method to improve the generated image quality of consistency models. The method mainly includes a joint classifier-discriminator, which contains an adversarial training objective and a gradual training approach. Experimental results show the effectiveness of the proposed method.
strength_weakness: strength_weakness: (--) The evaluations are limited. Only ImageNet at resolution 64x64 is utilized, which could not be representative since the development trend is high-resolution generation. Datasets that contain larger images should be considered, such as LSUN and FFHQ with resolution larger than 256x256. Meanwhile, more metrics, such as the precision and recall, are also worth considering for better evaluation about the data distribution.

Minor issue:
1. A typo at line 55: We show a ... improvements ...
2. The case rules of the journal or conference names in the reference are inconsistent sometimes.
questions: questions: Please refer to Weaknesses.
limitations: limitations: The limitations have been well discussed.
paper_summary: paper_summary: The paper introduces a post-processing method using a joint classifier-discriminator model to imporve the generation quality of consistency models. This model is trained adversarially, with the robust classifier providing gradient based on class assignment and the discriminator score. This post-processing method consistently improves the quality of the generated images in both single-step and double-step cases, resulting in better FID scores for both consistency distillation and consistency training.
strength_weakness: strength_weakness: - Although the method seems to be connected to the Robust Classifier and Energy-based Models, the authors did not give comprehensive theoretical derivation of the connections. The resulted loss is simply linear combination of binary CE and CE. 

- Is the proposed Robust Joint-Model specifically useful for consistency model? If so the authors may need to give analysis why this is the case. If not, it worth to study how the proposed method applicable with other generative models. 

- How does the radius $\epsilon$ affects the final performance? Also, how do the other hyper-parameters affect the model performance? The authors should give ablation studies regarding these parts. 

- From the empirical results in Table 3, the improvement of the final loss is marginal compared to using CE and BCE only sometimes. 

Some minors:

- line 124: algorithm (missed) 1.
questions: questions: Please see the Weakness.
limitations: limitations: The authors have discussed the limitations and potential negative societal impact of their work.

Comments: title: Thanks for the response
comment: The rebuttal has addressed my concerns.

I will increase the score.

Best regards,
Thank you for the response and effort! I am increasing my score based on the additional experiments.

My biggest remaining concern is the qualitative results; they mostly seem limited to color enhancements with maybe the exception of the robe image in Figure 1. 
I would still consider qualitative results with at least animal faces.
I appreciate the time and effort the authors have taken to address the concerns. The clarifications have addressed my questions and I have increased my score. Please make sure to incorporate these discussions in the final revision.
Thank you for your response. I recommend including these results in the paper, or the Appendix if they exceed the page limit. There are no more questions.