Representative Guidance: Diffusion Model Sampling with Coherence
Authors: Anh-Dung Dinh
Daochang Liu
Chang Xu
Venue: ICLR 2025 Spotlight (2025)
Keywords: generative models
diffusion model
Primary area: generative models
TL;DR: The paper investigates the inconsistency property of the diffusion model and propose a guidance method based on representative information to fix the inconsistency problem

Abstract: The diffusion sampling process faces a persistent challenge stemming from its incoherence, attributable to varying noise directions across different timesteps.
Our Representative Guidance (RepG) offers a new perspective to address this issue by reformulating the sampling process with a coherent direction toward a representative target.
From this perspective, classic classifier guidance reveals its drawback in lacking meaningful representative information, as the features it relies on are optimized for discrimination and tend to highlight only a narrow set of class-specific cues. This focus often sacrifices diversity and increases the risk of adversarial generation.
In contrast, we leverage self-supervised representations as the coherent target and treat sampling as a downstream task—one that focuses on refining image details and correcting generation errors, rather than settling for oversimplified outputs.
Our Representative Guidance achieves superior performance and demonstrates the potential of pre-trained self-supervised models in guiding diffusion sampling. Our findings show that RepG not only significantly improves vanilla diffusion sampling, but also surpasses state-of-the-art benchmarks when combined with classifier-free guidance.

Published as a conference paper at ICLR 2025
REPRESENTATIVE GUIDANCE: DIFFUSION SAMPLING WITH COHERENCE
Anh-Dung Dinh
Daochang Liu
School of Computer Science The University of Sydney anh-dung. di nh sydney edu. au
School of Physies, Mathematics and Computing The University of Western Australia daochang. 1 luê uwa. edu. au
Chang Xu
School of Computer Science The University of Sydney c. suussydney.ee edu. au
ABSTRACT
The diffusion sampling process faces a persistent challenge stemming from its incoherence, attributable to varying noise directions across different timesteps. Our Representative Guidance (RepG) offers a new perspective to address this issue by reformulating the sampling process with a coherent direction toward representative target. From this perspective, classic classifier guidance reveals its drawback in lacking meaningful representative information, as the features it relies on are optimized for discrimination and tend to highlight only a narrow set of class-specific cues. This focus often sacrifices diversity and increases the risk of adversarial generation. In contrast, we leverage self-supervised represen- tations as the coherent target and treat sampling as downstream focuses refining image details and correcting generation errors, rather than settling for oversimplified outputs. Our Representative Guidance achieves supe- rior performance and demonstrates the potential of pre-trained self-supervised models in guiding diffusion sampling. Our findings show that RepG not only significantly improves vanilla diffusion sampling, but also surpasses state-of- the-art benchmarks when combined with classifier-free guidance. source code: https::/gittie com hhungdinhaniee rep guidance
that
INTRODUCTION
In diffusion sampling processes Ho et al. (2020). persistent challenge arises from incoherence due to uncontrollable noise introduced at each timestep. As illustrated in Figure [ at each timestep, x, is used to predict the original image, which then aids in generating X,-1 in the next step. During training, the original images are sampled from the dataset, ensuring a consistent image distribution across all timesteps. However, during inference, the real dataset distribution is unavailable. Instead, the diffusion model draws from varying distributions at each timestep. incorporating different types of information, shown in the bottom row of Figure T This distributional shift between timesteps introduces incoherent features into the generated images. This paper addresses incoherence by framing discrepancy between the predicted image distributions across successive timesteps. Such discrepancies allow noise information to persist, leading to undesired artifacts in the generated images. For instance, an image of Leonberger may exhibit bizarre or inconsistent features in consecutive timesteps, hindering its transformation into a realistic depiction as the sampling process progresses, as illustrated in Figure Moreover, the generated images often lack crucial details, such as background elements and fine object features. While efforts such as DDIM [Sng et al. (2020a) have attempted to alleviate incoherence by removing random noise during sampling, they often do so at the cost of sample quality. As result, many recent diffusion models continue to rely on the mechanisms conventional DDPMs Ho et al. (2020).
"Corresponding Author

---

Published as a conference paper at ICLR 2025
Training
The same X
Xt
Xt
Sampling
Incoherence &o
Inferior Generation
Figure The top row is the real sampling process during training, where at every timestep, real images picked from coherent distribution. Nevertheless, during the inference phase, as in the row below, the predicted images at every time step have different distributions. The images with earlier timesteps more blurred than the images in the later stage of the sampling. This results in the incoherence between intermediate distributions.
Under our formulation of the incoherence, we propose a solution that involves tuning image features at each timestep to rectify incoherent features. We introduce guidance scheme termed Representative Guidance (RepG), which leverages information from representative vectors to steer the sampling process towards coherent direction. Moreover, unlike the traditional classifier guidance, where one-hot vectors represent classes, RepG represents each class through set of representative vectors containing features specific to that class. To harness the optimal representative information, we employ supervieer models prevalent in representative learning as our guidance model. The gradients derived using the pre-trained self-supervised model are directly integrated into the sampling process to facilitate feature tuning in generated images. In this sense, the sampling process can be viewed as a downstream task of the self-supervised models.
In comparison to the classifier guidance, which is popular method for enhancing the performance of the diffusion model, RepG offers multiple advantages. Firstly, our method provides a better representative target than the classifier guidance. The utilization of representative vectors for each class inherently contains valuable information for generative tasks. In contrast, the classifier guidance relies one-hot vectors representing each class, which offer limited information. This overly compact target leads to reliance on discriminative features within the classifier, which often proves insufficient for generative tasks and raises concerns about potential adversarial effects that could degrade the quality of generated images Dinh et al. (2023b).
Secondly, self-supervised models are trained to generalize well across datasets rather than being tailored to single task like classifiers. This characteristic helps mitigate the need for noise-aware training the guidance model, which can be prohibitively expensive, particularly for high-resolution images. Additionally, unlike noise-aware classifiers, self-supervised networks do not require memo- rizing noise patterns, making the guidance model more lightweight. For instance, our RepG, which leverages ResNer50, achieves efficiency in reducing computational time during sampling.
Thirdly, RepG does not compromise diversity, unlike the classifier guidance approach. While the classifier guidance alters images the class level to enforee diversity, RepG fine-tunes images the feature level. Consequently, while the former method encourages the generation images only with the popular features for each class, RepG preserves most of the image content while modifying faulty features and details.
In summary, proposed RepG operates distinctively compared to the classifier guidance. As for the classifier-fe free guidance, while the classifier- free guidance offers a trade-off between quality and diversity, method focuses upgrading details fine-tuning features, depicted in Figure
2

---

Published as a conference paper at ICLR 2025
RepG
X
RepG
: ag S (o()))
Figure 2: Leoneer The top row is the vanilla diffusion sampling process, and the bottom row is the sampling process with our Representative Guidance. From timestep 250 to limestep 185, both of the processes are similar. However, inconsistent features appeared the vanilla sampling process the black bubble exists at the head and the tail of the Leonberg at timestep 160. Without RepG, the process struggles to fix inconsistent features for the rest of the process. In contrast, RepG handles the case by removing the inconsistent features and making the image very clear from the time step 130. The RepG sampling process later focuses on improving other details such as hair, background, and surrounding objects. (Dataset: lmageNet156x226/ Diffusion Model: ADM)
3 Consequently, our RepG can complement the classifier-free guidance to enhance the generation quality further. Combining our method with the classifier-free guidance demonstrates superior performance compared to several SOTA baselines. The contributions of this paper are three-fold:
Model the incoherence the diffusion sampling process and introduce a suitable guidance scheme.
Propose the representative guidance target based on self-supervise pre-trained models.
Validate the results against a number state-of-the-ae baselines.
2 RELATED WORKS
Denoising Diffusion Probabilistic Models (DDPMs) Ho et al. (2020) and their score-based counter- parts Song & Ermon (2019); Song et al. (2020b) have become one of the most popular generative
models recently and replacing Generative Adversarial Networks (GANs) Odena et al. (2017) Kang
al. 202T); Sauer et al. (002)). The following works [Song et al. (2020a): Nichol & Dhariwal (2021) Dhariwal & Nichol (202T): Bao et al. (2022); Lam et al. (2022) improve the models in
different perspectives such as time reduction or sampling quality improvement. Recent trends in developing the Diffusion model leveraging the latent space for diffusion and denoising processes
such as DiT Peebles & Xie (2023), and Stable Diffusion Rombach et al. (2022) also offer diffusion models with less sampling time with good quality images.
Exposure bias Ning et al. (2023): Yu et al. (2023); Li al. (2023) is when the noise is accumulated through timesteps due to the lack of ground truth. However, the incoherence problem in this paper has different meanings. Incoherence means a mismatch between two distributions of predicted images two timesteps that should share the same information. This mismatch results in a gap, allowing incoherent features to be added to the images.
Guidance methods also emerge essential techniques to boost the performance of generated samples Dhariwal & Nichol 3) Nichol et al. (202T); Zheng et al. (2022): Dinh et al. (2023a b); Liu et al. (2023): ; Bansal al. (2023) In general, the noise-aware or off thel- classifier/CLD gradient is utilized to guide the diffusion sampling process to improve its performance in terms of FID. Classifier- free guidance Ho Salimans (2022) offers different way to trade off quality with diversity by combining conditional and unconditional diffusion models in the same framework. In Dinh et al. (2023b), the author points out that classifier guidance utilizes the most discriminative features only to do sampling, reducing the generated images' robustness and diversity. However, to achieve superior performance, these methods all give up diversity by significantly modifying details of the image to be close the the most common features of the conditional class. In this manuscript, propose a guidance method that fixes the details of the image instead of generating another one based on feature-level guidance.
3

---

Published as a conference paper at ICLR 2025
Although ProG Dinh al. (2023b) solves the problem of diversity suppression by including other classes' features, it still cannot avoid the fact that ProG is still based on discriminative features from classifier that not diverse enough for generative task. Thus, work utilizes supervipeee models that contain more general information. Self-supervised models Chen et al. (2020b); He et al. (2020); Chen & He (02T); Grill et al. (2020); CCen al. (2020a) aim to learn representative vectors that contain helpful information about data. While the applications of these models on generative tasks are still limited, this work shows that the pre-trained backbone from sell-supervised model is helpful without any training or fine-tuning. Other self self-supervised learning in diffusion models works all aim to one-tune the diffusion model in self superviuee manner or utilize the diffusion model as self-supervised model Hu al. (0033 Zhang et al. (2024).
3 BACKGROUND
DDPM: pe (x(x)) := po (Xa:1 Jdx with X1, Xg.... Xy are latent variables sharing the same dimen- sionality with the data X, ((x)) the main formulation of DDPMs with p(xr) M(xx; 0, I). The main aim of DDPMs training is to obtain the (x(:))) is the reverse process following the Marko- vian property Pe p(xr) II
Po Xx)), where po (x;- (xx) == (Nxx-11 1: (2,, t). [o t)).
The reverse process moves from a total noise image to a clear image. Hence, is used as a generator in the inference process.
The forward process corrupts the original data Xe to XT with Gaussian noise to train the 0 for serving the reverse purpose. This process a fixed Markov chain g(x1:1 (xo) == II g(x//xx1)) where q(x, 11) := [xxx vI 1x,-1, BrI). Be is the fixed variance scheduled from the start of the process.
From the given schedule, distribution of x, given Xy can be derived as:
((x:(xo) N (xx: vā,xo, (1 I
(I)
Denote a₂ = 1 - 33 and ā = IT Reverse from x2 given X distribution is derived as: ((x, X1, Xo) N(xx xo), 3,I)
(2)
Where mean value (X,, xo) var-iih
1-ce Xo + (1r(1-ār-1) X2 and variance B, :- 1-
Br. with
1-ue
I-as
reparameterization trick, we can sample the x as:
(1 a at
Xo (1 in-1)aar
Xx + 0₂
Similar to a Variational AutoFncoder Kingma & Welling (013), the optimization of 0 will be conducted via negative log-likelihood variational bound:
E log Po (x))) Ei- (-llogg())) Eạ21 log po (x
(4)
g(x, x1-1)
We re-write the Eq.
4 as:
B[- log po (xo) < B D
Σ Dxi(g(x--11xx (x log n (xn x1) t>1
In detail implementation. the 0 is chosen to be parameters of the noise predictor e ex(x,, The well-trained using Eq. can be used for sampling equation:
Xt-1
(x+
co (x1x, ()) + 0,2
VI
aq
4 METHODOLOGY
In this section, we first reformulate the sampling process to analyze the coherence issue. From Eq. 5
we first re-write the formulation of the sampling process
X (0+:
1)/ar11
VI (x + (1 δ(-1))
(c--1) Va, x,
022
(6)
5t
Vāi
Vāu
1 α
4

---

Published as a conference paper at ICLR 2025
The complete derivation of Eg can be found in Eq 24 in Appendix. Denote x/ as the prediction of Xy at time step t. From Eq we have x =
x
(x.)( as the prediction of x at the var
ya,
sampling step I. This results in new form of sampling equation:
11-00) aa (1
(1- -i-1)var
Xt-1
x
X, + 022
The Eq 7 the sampling from the distribution of q(x; X2, No) with já (xt, xo) (1-arly0 ar x6
1-6r
(1 ai- Iva, x, and o₁ which is matched with Eq and Eq the training of the DDPMs.
I-a;
In the training reverse phase in Eq there is a small assumption that Xy ~ g(xo). This results in information being passed from timestep into timestep
or the data distribution ((x,) is
consistent throughout all timesteps. However, this assumption is no longer valid during the sampling step. By assuming eo (x+,() e, we have:
= -N -X X2 vI at 0
X
(8)
However, the q(x) (x)) at two different timesteps are not the same, although they are both used for sampling X which is later used for sampling in Eq The illustration of the difference between these distributions can be found in Figure in the Appendix. The assumption that X ~ g(xp) at all timestep is not correct anymore and sample X from Eq. 7 can not hold. We define the incoberence problem as below:
Definition 4.1. Incoherence is the mismatch between predicted X distributions at different timestep l and mismatch between predicted &o distributions with real data distribution ((xo).
g (xx)) X ) * (x,x xx,) * ((x)) vt, > 1, #
(9)
The incoherence in the sampling process leaves the gap for the inconsistent features resulting from random noise appearing inside the image at some stage of the process. For example, in the top row of Figure 2 we observe the black bubbles the head and tail of the dog at the 160 timestep. The consequence is that the generated samples contain many blur details, inconsistent features, or unnecessary features.
4.1 REPRESENTATIVE GUIDANCE
From definition 4.1 gaps of inconsistent features result in poor-quality images. Thus, to solve the incoherence, need to make the distribution of intermediate samples {Xx xx) as close as possible to ((x)). However, the (xx)) is intractable during the sampling process. The intractability would make calculating any distance between these two distributions impossible.
Instead of calculating the direct distance between q (xx) and (xxo), we inject features information of xo during the sampling process in Eq 7 to force the sampling of x mimic the features of x at every timestep. First, we denote the fø Xx) as features extractor, parameterized by o, for Xo. Our
design aims to force x to have similar features fo (x6) as Xp. We denote ((ff((x)) f (x%) as the distance between two features.
Once again, since (xx)) is intractable, (x;) also intractable. To address this, instead of repre- senting fa (x)) at an instance-wist level, we encode the features of the entire dataset at class- wise level using ( c)). Here, g(x))e () is defined as an operation on the set of (x()) | x ~ (xooc)), where C denotes the class. The feature distance is then transformed into d () (xf)).
Given class e and data Xo. g(x0 c) results in the set of vectors representing the features of class c. Specifically, g(xo/c) {rf, ri..... ... ......,f.,, r where denotes the number of vectors required to represent class c. With C' classes, the entire dataset is encoded as = [Xx) 1), g(x,) (x,)2 ....................... g(xo [C}. This representation encoding enables the model to store compact set of representation vectors g( gxxple (c) for each class c, rather than storing the representative vectors f (xx) for the entire dataset. The selection of g and f will be discussed in Section 4.2
5

---

Published as a conference paper at ICLR 2025
As result, at each time step, given class c, refine the predicted Xy through the equation below:
x = x y 7V fxx
(10)
where y is the guidance scale.
From Eq
10 and given class the new sampling process with coherence is demonstrated in Eq.
X a+)
( / X0 -)- X + 0,2
(1
(1 at) / VVz (xx))
(11)
II (x1,)))) The guidance features from Xo provide consistent and reliable vāu target for Xy avoid the incoherence problem. We will discuss the similarity between Eq and Stochastic Gradient Descent process in Appendix [
with x
The choice of distance d can be varied. The rest of this section mainly discusses the options of d.
Negative Cosine similarity: At each timestep, we sample a vector rf g(xo The two vectors (x)) and rg can be matched via a negative cosine similarity loss as below:
(xg) x r
(or(fo(E)))
(12)
II (x)) I11
Contrastive loss: Apart from negative cosine similarity, the contrastive loss has also been used in many works on representative learning have presented [e et al.) (2020); Chen et al. (2020b a). The contrastive loss in our work is more toward the supervised contrastive rather than instance contrastive. We can define a positive pair as two vectors with the same classes and negative pair as two vectors with different classes. When sampling the image in class the loss for contrastive matching is:
(x H
exp
La(Jo(8)).V)
(13)
ii exp xF(
where H is the softmax temperature.
Replacing the matching equation in Eq 12 and Eq guidance equation:
13 into Eq
as C, we have the final sampling
(1 Oite
(1
(1 ar) //-1
Xt-1
X, 0:2
V (()o())) V)
Ct
Oz
1 Cz
(14)
4.2 REPRESENTATIVE TARGETS
In section 4.1 we have discussed a coherent guidance method given representative information from class This section will discuss the choice of the mapping function fo and the representative information for each class.
The most straightforward way is to naive classification for the guidance, where network such as ResNet [e et al. (2016) or a noise-aware classifier Dhariwal & Nichol (2021) is selected be classifier. This the case of the classic classifier guidance. The representative vector g(xolc) E {0, le reduces to a one-hot vector, with as the number of classes. However, the use of classification as representative information has many shortages. Firstly. the guidance reveals very little detail about the generated images. Since the classifier only processes the discriminative features, many details that are less discriminative for class are missed when using the classifier gradient to construct the image Dinh et al, (2023b). Secondly, the motivation for using classifier to construct images diffusion models is becoming weaker than the use of the classifier-free guidance. Since a conditional diffusion model already had class-conditioned information, the reason for using additional classification information to improve the performance of a conditioned diffusion model seems to be not strong enough to convince the community. As result, the research community often opts for classifier- free guidance Rombach et al. (2022); |Pebles Xie (202)). Thirdly, the use of classifier guidance is often associated with the very expensive training cost of noisy classifiers.

---

Published as a conference paper at ICLR 2025
Self- Self-supervised models known be very good generalizing data agnostic to augmentatioon/oise and separating image samples on representative spaces according to features He et al. (2020); Chen & He p: Jing et al. 22 Thus, we choose the self-supervised model to be our guidance model. The self-supervised models are pre-trained and we consider the sampling process as a downstream task of the model. Given real dataset Xo and pre-trained self-supervised model fa. instance o Xo is the tt instance in class c of the dataset. We have rf fo (zf)) The centre of each class on the representative space has the form yt Er;- We assume that the representative instances that closer to the mean values represent the most important features of the classes. We represent the whole class e via the representative information gxxo|c) as below:
g(xolc) = trx
Sol
Σ
min
(15)
RES.
Where Se is the list of indexes of K representative vectors that are selected to be the closest to the class mean representative vector f², We will discuss in section the value of K and different schemes to select representative vectors (x)lc) in addition to the "closest" scheme.
of representative class vectors representing the whole dataset to enable diffusion sampling with coherence using Eq[[4] Before the sampling process starts, V will be calculated in advance and stored as the network parameters for sampling.
Given Eq 15
have V {x(xx) 1 (x)o))),.... . o (xxoe))
5 EXPERIMENTAL RESULTS
Experiments are conducted to evaluate on ImageNet Deng et al. 2009) dataset with two resolutions 64x64 and 256x256 with 50000 generated samples. We first verify claims that our proposed
RepG helped to improve the details and fix the faulty information in the images qualitatively in
section5.] and quantitatively in section
After that, we will compare quantitatively with other
state-of-the-art methods such as BigGAN
Brock et al. (2018), ADM Dhariwal & Nichol (20T))
PxP Dinh et (2023a), ProG Dinh et (2023b). EDS [Zheng al. (2022), IDDPM Nichol
Dhariwal 2221 VAQ-VAE-2 Razavi et al. (2019) and Classiferr-fre guidance (CLSFree) Ho &
Salimans 2022p. Three baseline diffusion models are leveraged to evaluate the improvement of the
proposed Representative Guidance method are ADM Dhariwal & Nichol (202T), IDDPM Nichol &
Dhariwal 2021) and DiT Peebles & Xie 2023).
We denote that ADM or IDDPM as the ADM IDDPM diffusion model without guidance. ADM- is denoted for ADM with classifier guidance. PxP. ProG, EDS are advanced techniques to improve classifier guidance going after "+" sign. ADM-CLSFree and DiT-CLSFree are denoted for the application of classifier-fret free guidance on ADM and DiT respectively. RDM-CLSFree-C or DiT-CLSFree-G are denoted for applying the combination of classifier-free guidance and classifier guidance on ADM and DIT correspondingly.
5.1 INCOHERENT FEATURES ALLEVIATION
As discussed in section we observe incoherent features during the sampling process due to the incoherence of X at each timestep. This section shows that RepG successfully alleviates the inconsistent features in the generated images following three categories as in Figure In detail, RepG helps to improve the diffusion sampling process by fixing faulty features, removing unnecessary features, and upgrading details.
5.2 QUANTITATIVE IMPROVEMENT
This section compares the performance of our proposed RepG guidance with other state-of-the-art basclines as in [
Firstly, the use of RepG helps to improve the performance of vanilla baselines such ADM or IDDPM. Apart from the observation section 5.1 with qualitative improvement, we see a significant improvement in FID/sFID and precision when applying RepG ADM IDDPM. Secondly, given the same ADM diffusion model, ADM + RepG has a significantly better Recall value than other guidance methods such as ADM-G, ADM-CLSFrre, ADM-G PxPx + ProG, + +EDS+ ProG) which
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with state-of sfate-oo-theer the-art generative baselines ImageNet64x64 and ImageNer256x26. i denotes the ob-
Table 2:
We compare the use of different self-uuse models for our representative guidance.
tained score evaluated from generated im- ages frosfrom the published repo represents the number taken directly from the paper due to the lack of the source code or gen- erated samples. Other values are repro- duced from the published source code. The proposed RepG is shown to achieve better results than other stale-of-the-ie
Self-sup Model
sFID
Prec
Roc
ImageNet 64x64
W/o Guidince
2.07
4.29
0.73
0.63
MoCovvI
[69
3.42
0.75
7066
SimSiam
1.88
3.80
0.74
0.62
Moco-v3
1.81
3.93
0.76
0.62
Table 3: Different K values for our representative guidance
with several possible values K = {1,5, 10, 15}.
Model
FID
sFID
Prec
Rec
ImageNet64xx64
ImageNet 64x64
ADM
2.07
4.29
0.73
0.63
BigGAN
4.06
3.96
0.79
0.48
ADM
RepG (K=1)
1777
3,34
0.75
0060
TDDPM
2.50
3.78
0.73
0.62
ADM
RepGi (K=5)
1.69
3.42
0.75
0.62
IDPPM RepG
2.53
3.44
0.75
0.60
ADM
RepG (K=10)
1,73
3.43
0.75
0.62
ADM
2,07
4.29
0.73
0.63
ADM
RepCi (K=15)
1.82
3.45
0.75
0.62
ADM RepG
ADM-G
2.47
4.88
0.80
0.37
Table 4:
We compare the of different representative vector selection schemes.
ADM-G PxP
1.84
3.97
0.76
0.60
ADM-G ProG
ADM-G + EDS ProG
1:37
4.25
0.77
0.61
FTD
sFID
Prec Rec
ADM-CI.SPres
1.89
4,45
0.77
0.60
ADM-CLSFiee ProG
1.91
4.51
0.76
ImageNet644x64
ADM-CTSPrre RepG
1.67
3.44
0.78
0.61
ImageNet 256x256
ADM Rep
L66
3.3
0.75
70762
ADM RepG.Raa
2.04
0.74
BigGAN
7.03
7.29
0.87
0.27
DCTrass
VQ-VAE-2
31.11
17.38
0,36
0.57
Table 5:
The use of two matching losses used in sampling as mentioned in section Daffects the performance of the diffusion sampling process. The result indicates the superiority both
IDDPM'
12,26
5,42
0.70
0.62
ADM
10.94
8.02
(.69
0.63
ADM RepG
7,83
5.79
0.72
0.61
of the losses' performances compared without guidance. The contrastive achieves slightly better than negative cosine
ADM-G
4.58
5,23
0.81
0.52
ADM-G + EDS
3.96
5,00
0.82
0.52
ADM-G
similarity loss.
ADM-G + ProG
4.53
5.08
085
0.49
ADM-G + ProG EDS
3.84
5.00
0,83
0.51
ADM CL.Sree
3,76
4.45
0.77
0.53
Prof
3.81
4,46
0,77
0.53
ImageNet 64x64
ADM STree RepG
DVT-CCSPPres
2.27
3,50
0.82
0.58
W/o Guidance
2.07
4.29
0.73
0.63
4.56
Conirastive
T.69
3344
0.75
'0.6
DIT-CLSFree RepG
2.17
4.59
0.80
0.60
Cosine Similarity
1.75
3.57
0.75
0.60
indicates that RepG helps to keep the diversity better than other guidance methods (As highlight in pink column) Finally, The combination of RepG and CLSFree guidance outperforms other state-of- the-art art guidance methods such as PxP Dinh et al. (2023a), ProG Dinh et al. (2023b). EDS Zheng et al. (2022) CLSFree Ho & Salimans (2022).
Note: On ImagcNe1256x256, the RepG improves baseline ADM significantly but lags behind other guidance methods. This is expected as RepG only improves details and keeps diversity while other methods sacrifice diversity to achieve better quality. On ImageNet64x66, RepG outperforms other guidance methods due the information in ImageNe164x64 is less than its 256 counterpart and focuses on foreground objects. Improving objects' features is enough to beat other methods.
Classifier guidance failed to improve the performance of classifier-free guidance significantly (ADM- CLSFree-G+y GaieeeG+PrroGG and DiT- CWSSCCLLS+GGPPPP essence of the two methods. These two methods do the same thing: trade-off quality with diversity, which offers less improvement when combined. However, RepG successfully improves classifier-free guidance since RepG does a different task: tune the details of the images.
in Table [.. This is due to the overlapping trade-off
5.3 ABLATION STUDY
Section and 5.2 have shown qualitative and quantitative improvement compared to previous state-of -fate-of-the-ar the-art baselines. In the Ablation study, we discuss different choices for our models, such the choice of self-supervised models, the performance of the proposed methods different
8

---

Published as a conference paper at ICLR 2025
Faulty correction
Details upgrade
unnecessary features removal
W/o RepG
RepG
W/o RepG
RepG
Figure 3: RepG enhances diffusion sampling in three key ways: (1) correcting faulty features to improve realism (row 1), (2) refining object and buckground details, such sharpening the dog' hair and enhancing grass, trees, and fences (row 2), and (3) removing unnecessary elements like a human figure
incorrect body structure (row 3). ImageNer256x256
guidance scales, the number of representative targets utilized, and the performance comparison between contrastive matching loss (3)
13) and cosine similarity matching loss (Eq T2p.
5.3. DIFFERENT SELF-SUPERVISED MODELS
In all of the RepG results in Table I we use MoCo-v2 Chen et al. (2020b) as the backbone for guidance. This section compares different choices of pretrained self superiseee models in Table 2 In detail, three popular pre-trained self- supe-sueervieed models are utilized, which are MoCo-v2 Chen ctt al. (2020b), SimSiam Chen & He (2021), and Moco-v3 Chen et al.] The performance shows that MoCo-v2 achieves the best among the three models. The outperformance of Moco-v2 could be due to the representative information obtained by MoCo-v2 having contrastive information compared to SimSiam. hence obtaining more information about data than just clustering it. Moco-v3 delivers better FID than SimSiam but still not as good as Moco-v2, yet Moco-v3 offers better Precision.
5.3.2 GUIDANCE SCALES EFFECTS
Similar the classifier guidance
Dhariwal &
Nichol 2021): Zheng et al. (2022 Dinh et al.
(2023a our RepGi can also be controlled by
the guidance scale as in Eq TO We compare
7.5
the effect of the guidance scale the range of
60
Craat der Aiitio Rortarradii gadlaro
[0, 10] with = 0.0 in the diffusion sampling process without any guidance. Figure 4 shows the trend of FID and Recall when increasing the guidance scale. The generation quality of RepG is improved steadily without trading off diversity compared to classifier guidance. Improvement without trading off with diversity is expected since our method mostly keeps the content of fixing faulty features. The effects of increasing
25
Figure 4: We compare the Recall trend between the classifier and representative guidance (RepG). RepG shows a much more stable trend in diversity than classifier guidance.
the generated images while upgrading the details the guidance scale can be observed Figure 5
5.3.3 REPRESENTATIVE TARGETS
This section shows the effect of selecting representative targets for each class.

---

Published as a conference paper at ICLR 2025
Guidknce scale
Q
50
80
Figure 5: Ablation study the RepG guidance scale. Unlike classifier guidance, where increasing the guidance scale shifts the image toward simpler region (as shown in Figure increasing the RepG guidance scale enhances image details.
The K values: The different choices of K value in Eq 15 affects the performance. The experiment conducted on ImageNet64x64 as shown Table As we can see, given K=1, there is only one representative vector for one class, reducing the generated samples' quality and diversity. However, more than five representative vectors per class will confuse the sampling process and downgrade the performance. Understandably, including more representative vectors brings more features to be excluded due to the contrastive loss. The excluded features might include shared features between classes, which have become common due to the inclusion of more information.
Selection strategy: In the previous experiments, representative vectors are selected closest to the mean vector of all vectors belonging to class. We compare selection scheme with the random selection scheme in Table 4 RepG Rand denotes the random selection scheme. From the results, we show that proposed selection of representative vectors essential and verify hypothesis that the vector is close to the mean values of one class bearing crucial features of that class.
5.3.4 CONTRASTIVE MATCHING COSINE SIMILARITY MATCHING
In section 4.2 we discussed the two losses: the contrastive loss and the cosine similarity loss. Table shows the comparison between the two losses, which show that both of them improve the performance significantly compared to the baseline ADM Dhariwal Nichol (20I))
CONCLUSION
In this work, we formulate the problem incoherence in the diffusion sampling process, defined the mismatch between predicted image distribution two different timesteps. After that, propose a guidance method named Representative Guidance (RegG). RepG is based on representative information of class and pre-trained self-supervised models guide the sampling process. The representative information offers number of advantages compared to one-hot representation as in classifier guidance, such as rich information and information avoid incoherence problems.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 843 852, 2023.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201. ..6500,,,33,,,..00 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 1006, 2018.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Chen, Xic, and K He. An empirical study training self-supervised vision transformers. in 2021 ieee. In CVF International Conference on Computer Vision (ICCV), pp. 9620-9629.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Anh-Dung Dinh, Daochang Liu, and Chang Xu. Pixelasparam: gradient view on diffusion sampling with guidance. In International Conference Machine Learning, pp. 8120-8137. PMLR, 2023a.
Anh- Anh-Dung Dinh, Daochang Liu, and Chang Xu. Rethinking conditional diffusion sampling with progressive guidance. In seirth- Conference Neural Information Processing Systems, 2023b.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in neural information processing systems, 13:21271-21288 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 1cXiv:2207.12598 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Vincent Tao Hu, Yunlu Chen, Mathilde Caron, Yuki M Asano, Cees GM Snoek, and Bjorn Ommer. Guided diffusion from self-supervised diffusion features. arXiv preprint arXiv:2312.08885, 2023.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:21 10.09338, 2021.

---

Published as a conference paper at ICLR 2025
Minguk Kang, Woohyeon Shim, Minsu Cho, and Jaesik Park. Rebooting acgan: Auxiliary classifier gans with stable training. Advances in neural information processing systems, 34:23505-23511 2021.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. arXiv preprint 1rXiv:2203.1350: 2022.
Mingxiao Li, Tingyu Qu, Wei Sun, and Marie-Franncnne Moens. Alleviating exposure bias in diffusion models through sampling with shifted time steps. arXiv preprint 2.Xiv:23055/558. 2023.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 289-299, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. arXiv preprint 2131v:2301.11766, 2023.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, pp. 2642-2651. PMLR, 2017.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-42055 2023.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020b.
Hu Yu, Li Shen, Jie Huang, Man Zhou, Hongsheng Li, and Feng Zhao. Debias the training of diffusion models. arXiv preprint arXiv:2310.08442. 2023.
Junyu Zhang, Daochang Liu, Shichao Zhang, and Chang Xu. Contrastive sampling chains in diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Guangcong Zheng. Shengming Li, Hui Wang, Taiping Yao, Yang Chen, Shouhong Ding, and Xi Li. Entropy-driven sampling and training scheme for conditional diffusion generation. In European Conference Computer Vision, pp. 754-769. Springer, 2022.
12

---

Published as a conference paper at ICLR 2025
Algorithm 1 DDPM denoising process with representative guidance
Input: class labels y, classification scale V {x [1), (xx .2,, ((xolc)) according to
Eq 15
XT I)
pick class e and ((xolc) E
for t T. .... do
~ (0,1)
(T-mroo(xe.to))
Xp
Voc
6-
(1-au) vur
1-0, V [(f (x)), V) according to 14
Xt-1 vaz
(x
VI-ai
3-ac
co (xt, (x,, + oîg + 0+2 9
end for
A SAMPLING ALGORITHMS
Like DDPMs, our sampling only updates X at every time step f. We have the set of representative vectors V obtained in advance and stored as the model parameters used for sampling.
The mechanism is the same for latent diffusion, but we will decode the latent vector to x first. After that, the process is similar to Algorithm i
B EXPERIMENTAL DETAILS
All the experiments in this paper are conducted on A100 GPUs 40GB.
We have three hyperparameters in the paper, which are the number representative vectors in temperature H/ in Eq 13 and scale guidance in Eq 14
Eq 15
Table 6: All hyperparameters for producing the results are shown in this table.
Model
Datasets
K
H
?
Table 1
IDDPM RepG
ImageNet64x64
10.0
ADM RepG
1mageNet64x64
5
1
10.0
ADM-CLSFree + RepG
ImagcNet64x64
5
8.0
ADM RepG
ImageNer256x255
20.0
ADM-CLSFree + RepG
InageNe1256x256
10
20.0
DiT-CLSFree RepG
ImageNer256x256
15.0
Table 2
W/o Guidance
ImageNet64x64 ImageNet64x64
0.0
Moco-22/imSSiam/Moco33
10.0
Table 3
ADM RepG
ImageNet64x64
1,5,10,15
10.0
Table 4
ADM RepG HADM+RepG.RAND
ImageNe164x64
5
10.0
Figure 1,2.3.5.6,7
ADM RepG
ImageNet256x256
$0.20,0,,000,8800
Figure 4
ADM RepG
ImageNet64x64
2.0.4.0, 6.0. 8.0, 10.0
13

---

Published as a conference paper at ICLR 2025
Table 7: GPU hours on GPU are needed generate 50,000 images with 256x256 resolutions.
Diffusion Model: ADM/ Datasets: ImageNer256x2566
Model
Computational Cost (GPU hours)
No guidance
171.22
Representative Guidance
182.36
Classifier Guidance
247.84
Classifier-free Guidance
352.89
Table 8: GPU hours on GPU are needed to generate 50,000 images with 64x64 resolutions.
Diffusion Model: ADM/ Datasets: ImageNet64x64
Model
Computational Cost (GPU hours)
No guidance
16.71
Representative Guidance
17.55
Classifier Guidance
31.52
Classifier-free Guidance
32.64
C RUNNING TIME OF REPG COMPARED TO CLASSIFIER GUIDANCE
RepG utilizes much lighter model compared to noise-aware used in classifier guidance Dhariwal & Nichol (202T). As result, the calculation of gradients using this model much lighter compared to the noise- ware classifiers. We have the running time comparison in Table [nddd 8
D FULL DERIVATION OF EQUATIONS
Similarity between Eq. 11 and Stochastic Gradient Descent: We start from Eq 11 as below:
x, (1 a1) a1-1 /a-1 x/-
(1 dt /
X, 022
at
(1 at) va)
V %
d(g(xxll)) (c), (x()))
(16)
with x
( X
A1 11-01 to (X..
Similarly have X
Vāc-1
Va.
vāa
VI - -T-at-
10-150 10 1). Thus, we have Eq
16 equivalent to Eq
17
[1 - ar)
VI -c-
(1 - ---11) 0,z (1- Var1
xt-1
-x 1)
Var-1
an)
v x; (lxc
1
at
úi x +
vI Ci-1
c0 (1- (0(xx-1,,,-11) (1
x1 - 012)
oz 0:-11
(1 a1)
2 d(g fo(())
(17)
with x is obtained from Eq 16
The Eq 17 has very close form with Stochastic Gradient Descent optimization with X as
parameters and two gradients V, = 4= xh +
Vi-
vaLi co cx 1)
(1-0, 1ly - 0,
(1-&))vat-
and V, (1-ar) y d(g(xolc), fo (x))). We will show that this Eq 17 has a consistent objective function. From Eq [ and two timesteps f < tzz we have:
X1, = V Xo + v1 oit 61
(18)
X1, 0r, xo + /I 62
(19)
14

---

Published as a conference paper at ICLR 2025
From Xx, and Xt2, have the prediction of Xn is x (f1) and l2 is x
(t2).
We have:
x Xt, vI Cie Eo (1)
(f1)
(20)
Vān
X12 vI - ae (Xt,,
X
(t2)
(21)
Vih22
Replace Eq 18 and 19 into
Eq
20 and 21
we have:
ar, (E1 cp((x, (1x,,,,11 t1))
x
x0 +
(22)
Viitz
x/²) + Org (e2 ca (x 12)
(23)
v VOtg
X0 VI-oe, (xx.0)).)))
From Eq 22 and 23 we have at any timestep é, the distance between X
Vā:
which means ||*6 Xo || Vl-a,, [xx)x(xx.)))
Assuming that co is trained converge, assume [Xe, xxxx,1 (1) - eill S (x0, (2) <2/1. because when image is clearer, we also expect the
Va:
error should be smaller. The extreme case is |lel 1en-ee(x,,, [1)]l ||e2 co 12)1l
ea(Xx,
A. As
result ||S]]] - -xoll = /i-ar, A and ||*000 - xoll = Vi- Since 4 < l2,
Vi-āa V011
VI-aaz
Van
V012
Voi
which means ||x] (t) -xoll < ||x]((2
Vt, < <t2. Which means that from T to O, the sampling
process will update x so that | - Xol 7 min. We have the first gradient the Eq 17
V
a.-a: 1-a; x
VI-ā--
Vai
co cx(x 1) (1-āilva
(1-a1-1]/ar
022.
(1-ar), We can easily see the second gradient is the V2 I-dee y Vxx d( fz to minimize the distance d(g(xolc),
Thus, we can conclude that the sampling process as Eq 11 is process of Stochastic Gradient Descent to optimize two objectives. The first objective is ming; ( xoll and the second objective is ming (() (x))).
Full derivation of 6
Eq can be fully derived as below:
1
1 at
X
X1
cx(x 002
Vot
vT
at
- a4
(1
01-1)
1 ai cx(xxtt) +02
X;
x:) V1- C2
(1 â)
Va
at
X1
-I-ai ar
co(x,,,)) +
(1 01-1)
X1 0²2
āi
Va
Va,
- Ōi
at
a1)/a1111
X;
VI ==VI-dne(xxx (xE((xx))
(1 - a(-1)
+ 022 (24)
Vār
Van
1 C₁
E No DISTRIBUTION
Figure 6 shows the difference in the distributions of x at different timesteps.
F CLASSIFER GUIDANCE DIVERSITY SUPPRESSION
Similar to Dinh et al. (203bb), we reproduce the diversity suppression of classifier guidance as in
Figure 8
and 10
G MORE QUALITATIVE RESULTS COMPARISON FOR REPG
Figure
and 14 shows more examples of how RepG can help to fix details in the generated
images
15

---

Published as a conference paper at ICLR 2025
Figure 6: Visualization of x at different timesteps. X has different distributions when varies. The earlier timesteps have less information, while the later stage has clearer views of the images.
tmaestaf
RepG
Guidance scale
CLSG
RepG
Guidance scale
CLSG
Figure 7: RepG edit details in the image while Classifier Guidance (CLSG) generates another image with good discriminative feature. However, sometimes, the over-exploiting of discriminative features results in the lack of robustness features in the output.
H
MORE SAMPLES WITH REPG
Figure 13 shows several samples by DiT combined with RepG.
16

---

Published as a conference paper at ICLR 2025
Figure 8: Classifier guidance utilizes the same style to repeat features to all generated images. This due to the overexploitation discriminative features (front-face features) reducing the diversity of the diffusion model
Figure 9: Classifier guidance utilizes the same style to repeat features to all generated images. This is due to the overexploitation of discriminative features (front-face features) reducing the diversity of the diffusion model
17

---

Published as a conference paper at ICLR 2025
Figure 10: Classifier guidance utilizes the same style to repeat features to all generated images. This is due to the overexploitation discriminative features (lie-in-bed features) reducing the diversity of the diffusion model
Figure 11: ImugeNei156x2501laax:: tiger shark. The images on the left, shown before the arrow, are the erroneous outputs generated by ADM. while the images the right, after the arrow, depict the corrections made using RepG. The examples show that RepG can improve the details and fix the erroneous features.
18

---

Published as a conference paper at ICLR 2025
Figure 12: ImageNer256x266llaass green lizard. The images on the left, shown before the arrow, are the erroneous outputs generated by ADM, while the images on the right, after the arrow, depict the corrections made using RepG. The examples show that RepG can improve details/background. remove unnecessary features and fix erroneous features.
Figure 13: ImageNet25o:25iclas::: crane. The images the left, shown before the arrow, the erroneous outputs generated by ADM, while the images on the right, after the arrow, depict the corrections made using RepG.The examples show that RepG upgrade details, modify faulty features.
19

---

Published as a conference paper at ICLR 2025
Figure 14: 1mayeNe12562266llas::: English springer. The images on the left, shown before the arrow, are the erroneous outputs generated by ADM, while the images on the right, after the arrow, depict the corrections made using RepG. The examples show that RepG can improve details/background remove unnecessary features and fix erroneous features.
20

---

Published as a conference paper at ICLR 2025
Goldfish
Crampfish
Jellyfish
Snail
Red fox
Polar
bear
Figure 15: Sampling by DIT with RepG for several classes.
21


Reviews:
[{"review_id": "TaPrDAY2ek", "reviewer": "Reviewer_b4Kq", "summary": "This paper addresses the **incoherence problem in diffusion models**, where inconsistencies across timesteps lead to artifacts and reduced image quality. The authors introduce a novel approach by framing this issue as **feature discrepancy alignment** between predicted image distributions and representative features across timesteps. They propose **Representative Guidance (RepG)**, which uses representative vectors from self-supervised models to guide sampling coherently. This method successfully enhances the quality of generated images while preserving diversity, offering promising alternative guidance in diffusion models.", "questions": "1. The paper suggests that RepG is more computationally efficient than classifier-based guidance, which often requires noise-aware training. However, it’s unclear what the computational trade-offs are for RepG, given the potential overhead from integrating self-supervised models and tuning features at each timestep. Could the authors provide a more thorough comparison of computational efficiency?\n2. In the ablation study on the number of representative vectors 𝐾, lines 480-485, could you provide further explanation? The results suggest sensitivity to 𝐾 values. Given that the level of shared features between different classes may vary significantly across datasets and depends on the chosen self-supervised model, could the authors provide more context on how to select appropriate 𝐾 values? Additional explanations on the trade-offs involved would clarify the model's adaptability to different datasets and feature distributions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Originality:**  This paper tackles the important challenge of enhancing image generation quality in diffusion models. The authors approach this by tackling incoherence in diffusion models and redefining it as a mismatch in predicted image distributions across timesteps. This fresh perspective enables a unique and novel solution to this problem.\n2. **Novelty:** Unlike conventional classifier-based guidance, RepG utilizes pre-trained self-supervised models to provide representative vectors, making it effective for fine-grained image sampling. By treating sampling as a downstream task of self-supervised learning, RepG leverages the self-supervised features, which inherently contain richer semantic information than one-hot class labels. This choice addresses the common problem of classifier-guided diffusion models that often overemphasize discriminative features at the cost of generative quality and diversity.\n3. **Results:** The experimental results demonstrate that RepG improves image quality, producing generations with finer details and reduced artifacts while preserving diversity.", "weaknesses": "1. **Dependency on Self-Supervised Model Quality**:  It seems that RepG relies heavily on the quality of self-supervised models for guidance. For instance, MoCo-v2 yields the best results, while other models like SimSiam show only marginal improvement over classifier-free guidance (e.g., FID of 1.88 vs. 1.89 on ImageNet64x64). This limitation is not thoroughly addressed in the paper, raising questions about RepG's generalizability and performance across different datasets where MoCo-style representations may not be optimal.\n\n2. **Limited Applicability to High-Resolution Image Generation**: The paper shows promising results in lower-resolution settings, but lacks validation on high-resolution datasets. Since RepG’s main advantage lies in enriching details while preserving diversity, testing on high-resolution images could reveal strengths or limitations in its ability to maintain fine-grained coherence and detail at scale.\n\n3. **Need for More Convincing Quantitative Results**: While Figure 3 provides examples of improvements, some results appear limited or inconsistent. For example, in the last row on the right, although a human figure is partially removed, body parts remain, which makes the result more unrealistic. Additionally, the left example in that row seems to reduce image realism rather than improve it. Including more compelling examples would strengthen the argument for RepG’s claimed improvements in detail and coherence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mhJuBpNNKP", "reviewer": "Reviewer_eBdC", "summary": "The authors proposed an approach, namely representative guidance (RepG) to make the sampling process of diffusion model along a consistent direction. The proposed RepG is a plugin module, which can be easily adapted to existing framework. Particularly, RepG adopts the self-supervised representation to achieve the aim of consistent sampling direction of diffusion. The proposed approach is evaluated on publicly available ImageNet dataset, which presents improvements to existing diffusion models.", "questions": "Please refer to weakness section for details.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is easy to follow.\n2. The approach has been evaluated on publicly available dataset, which makes the experimental results convincing.", "weaknesses": "1. There are lots of typos in the paper, such as the caption of Table 1: ImageNet 256x26 -> ImageNet 256x256. The authors should carefully proof-read the manuscript.\n2. Some important baselines are missing from Table 1, such as the comparison between RepG and the existing approaches with IDDPM. The comparison with ADM backbone seems to be more comprehensive. The authors should illustrate the reason for such an experimental setting.\n3. With ADM backbone, the experimental results are also difficult to directly compare. Too much different setting of the ADM variant. The authors may consider to simply compare the methods with ADM + X (where X is RepG / PxP and the others).\n4. Why the performance of MoCo-v3 is lower than MoCo-v2 in Table 2?\n5. Please include the visualization results generated by existing approaches as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t6oNPVOOn0", "reviewer": "Reviewer_9ZFf", "summary": "The paper proposes a solution for improving diffusion sampling process that involves tuning image features at each time step to rectify incoherent features. For this purpose, the authors introduce a guidance scheme called Representative Guidance (RepG), which leverages information from representative vectors to steer the sampling process towards a coherent direction. Moreover, RepG represents each class through a set of representative vectors containing features specific to that class. To control the optimal representative information, RepG\nemploys self-supervision as guidance model. The gradients derived using the pre-trained self-supervised model are directly integrated into the sampling process to facilitate feature tuning in the generated images. Extensive experimental results demonstrate that RepG achieves superior performance in image generation.", "questions": "Here are my concerns:\n- Please provide more details on how the representative vectors in RepG capture more valuable information for generative tasks than the one-hot vectors used in classifier guidance.\n- You stated in the paper that RepG modifies \"faulty\" features in the generated images. How does your appoach identify which features are faulty? It could be possible that some key features could be mistakenly modified, thus affecting the image quality?\n- You mentioned in the paper that RepG is computationally efficient. Could you provide comparative metrics on the computational savings over traditional classifier-guided methods (including for instance image generation at different resolutions)?\n- Definition 4.1: t1 and t2 should be consecutive time steps?\n- Equation 13: In the denominator, the superscript of 'r' should be 'i', not 'c'.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is clearly written and easy to follow\n- The related work covers most of the relevant papers in the field\n- Experimental validation is extensive and demonstrates the superiority of the proposed approach", "weaknesses": "- Some details require more clarification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)