Token Perturbation Guidance for Diffusion Models
Authors: Javad Rajabi
Soroush Mehraban
Seyedmorteza Sadat
Babak Taati
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Diffusion Guidance
Guidance
Primary area: deep_learning
TL;DR: Token Perturbation Guidance (TPG) is a novel framework that applies perturbations directly in token space to guide the diffusion sampling process.

Abstract: Classifier-free guidance (CFG) has become an essential component of modern diffusion models to enhance both generation quality and alignment with input conditions. However, CFG requires specific training procedures and is limited to conditional generation. To address these limitations, we propose Token Perturbation Guidance (TPG), a novel method that applies perturbation matrices directly to intermediate token representations within the diffusion network. TPG employs a norm-preserving shuffling operation to provide effective and stable guidance signals that improve generation quality without architectural changes. As a result, TPG is training-free and agnostic to input conditions, making it readily applicable to both conditional and unconditional generation. We also analyze the guidance term provided by TPG and show that its effect on sampling more closely resembles CFG compared to existing training-free guidance techniques. We extensively evaluate TPG on SDXL and Stable Diffusion 2.1, demonstrating nearly a 2x improvement in FID for unconditional generation over the SDXL baseline and showing that TPG closely matches CFG in prompt alignment. Thus, TPG represents a general, condition-agnostic guidance method that extends CFG-like benefits to a broader class of diffusion models.

Token Perturbation Guidance for Diffusion Models
Javad R
Soroush Mehraban 1,2,3
Seyedmorteza Sadat*
Babak Taati!.2.
!University of Toronto
Vector Institute for Artificial Intelligence
KITE Research Institute
*ETH Zürich
frajabi, taati]@cs... toronto..oooom. soroush. soroush. wwt.:mmmmmaaaammai.u....oo utwronto....... ca
seyedmorteza, sadat@inf .etzz.c... ch
Abstract
Classifier-free guidance (CFG) has become an essential component of modern diffusion models to enhance both generation quality and alignment with input conditions. However, CFG requires specific training procedures and is limited to conditional generation. address these limitations, we propose Token Per- turbation Guidance (TPG), a novel method that applies perturbation matrices directly to intermediate token representations within the diffusion network. TPG employs norm--oessee preeerriiiiiingg shuffling operation provide effective and stable guidance signals that improve generation quality without architectural changes. As a result, TPG training-free and agnostic to input conditions, making it readily applicable to both conditional and unconditional generation. We further ana- lyze the guidance term provided by TPG and show that its effect on sampling closely resembles CFG compared to existing training-free guidance tech- niques. Extensive experiments on SDXL and Stable Diffusion 2.1 show that TPG achieves nearly a 2x improvement in FID for unconditional generation over the SDXL baseline, while closely matching CFG in prompt alignment. These re- sults establish TPG
general, condition-agnoste guidance method that brings CFG-like benefits a broader class of diffusion models. The code is available https: /ttp:///gtttt..tt/ com/Tamaaaaaaoo Taat Teeiiieeemmooooee Seann-ertuus perturbation Guidance
Introduction
Diffusion models [1, 3] have emerged as the main methodology behind many successful generative models for images [4, 5], videos [6, 7, 8, 9, 10], audio [11, 12], and 3D objects [13, 14, 15). Despite their thcoretical capacity to produce high-fidelity data, unguided diffusion models often suffer from poor sample quality, manifesting visual artifacts, lack of semantic consistency, and insufficient sharp details [16]. To mitigate these issues, classifier-free guidance (CFG) [17] has become the de facto approach steer the generation process toward higher quality and more semantically aligned outputs. However, CFG is inherently limited to conditional generation and requires a specific training strategy that randomly replaces the input condition with a null condition.
In response to these constraints, several alternative guidance techniques have emerged, aiming to extend CFG-like benefits to broader settings [18, 16, 19, 20, 21]. These approaches often manipulate components of the denoiser network, such as attention layers, to construct effective guidance signals. However, they either require additional specialized training offer limited improvements in prompt alignment and generation quality (particularly w.r.t. unconditional generation). Accordingly, there remains need for training-free guidance mechanism that works across both conditional and unconditional settings while improving generation quality and semantic alignment similar to CFG.
In this paper, we revisit existing attention-based guidance techniques and aim to bridge the gap between the effectiveness of CFG and that of training-free. condition-agnosse methods. We observe
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CFG
PAG
SEG
TPG (OURS)
Motted
Tuaestos
981
941
901
861
821
781
741
701
601
501
201
Figure 1: Visualization of the denoising process over time for different guidance strategies: CFG [17],
PAG [19], SEG [18], and our proposed TPG. Each row shows generated images at various denoising 981 (left) to f (right). The red box highlights the early-to-middle denoising stage (t = 821 to 1 = 741), where CFG and TPG demonstrate clearer structure (e.g. horse face) and consistency. The text prompt used is "a female in black jacket is riding brown and white horse".
time steps, from
that CFG effectively recovers global structure and coarse details during the early denoising steps. whereas existing waining-freee methods tend to produce over-smoothed results at the same stage (see highlighted regions in Figure 1). This is problematic because early denoising steps are critical for both image quality and prompt alignment, they establish global structure, major shapes, and coarse semantics before the network begins refining fine details [22, 23]. If the model fails to capture correct semantics, object placement, overall composition at this stage, it may never fully recover from that high-level mismatch in later refinements. This lack sufficient carly-step guidance likely explains why existing methods often yield only marginal improvements in prompt alignment and generation quality compared to CFG.
Motivated by these insights, we introduce Token Perturbation Guidance (TPG), novel method that directly perturbs intermediate token representations within the diffusion network, without requiring additional training or architectural changes. TPG employs token shuffling as the core operation to provide effective guidance signals. Specifically, token shuffling (i) linear, (ii) preserves token norms, and (iii) disrupts local structure while maintaining global statistics. As shown in Figure 1, TPG exhibits behavior similar to CFG and, compared to other training-free free methods, more faithfully recovers both global structure and fine details at early denoising stages.
We evaluate the effectiveness of TPG on both conditional and unconditional image generation using SDXL [5] and Stable Diffusion 2.1 [24]. Our results show that TPG achieves nearly 2x improvement in FID for unconditional generation compared the SDXL baseline and also significantly outperforms existing perturbation-bast that TPG closely mirrors CFG in terms of alignment and its effect on the sampling process, i.e., positively aligning with ground-truth noise in low-frequency bands, remaining largely orthogonal at other frequencies, and following a similar norm profile throughout denoising. These findings establish TPG as general, condition-agnostic guidance method that extends CFG-like benefits to broader class diffusion models, including those for unconditional generation.
guidance techniques. Furthermore, we observe
2 Related work
Score-based diffusion models [25, 1, 26, 3] reverse a forward noising process by learning the score function, the gradient of the log data density, at multiple noise levels to progressively transform pure Gaussian noise into realistic samples [26]. This principled estimator of the data distribution has outperformed previous generative modeling methods in both fidelity and mode coverage [23]. Since the score represents an explicit gradient field, the sampling trajectory can guided by incorporating auxiliary gradients, leading to powerful guidance techniques such as classifier guidance (CG) [23] and classifier-free guidance (CFG) [17]. These guidance methods significantly enhance image quality and prompt alignment, albeit at the cost of oversaturation [27] and reduced diversity [17, 28).

---

Although CFG improves image fidelity and alignment with the input condition, it is inherently restricted conditional generation. Moreover, since its guidance signal is defined as the difference between conditional and unconditional denoising outputs, CFG requires specific training procedures and its sampling trajectory can overshoot the desired conditional distribution, leading to skewed or oversimplified images [29]. Autoguidance [16] builds on CFG by introducing deliberately weaker, also known as the "bad version" of the noise predictor, i.c., a less-trained denoiser network, to produce the guidance signal. While this avoids CFG's s reliance on the unconditional score, identifying an effective "bad version" is non-trivial and still requires training and careful tuning of the model. By contrast, method requires additional training or architectural changes.
Recently, attention- based perturbation methods have shown promising results in improving the quality of generated images by leveraging or modifying the attention maps within the diffusion model's s attention blocks. Self-Attention Guidance [20] uses the model's own attention map to blur the denoiser input. Perturbed Attention Guidance (PAG)[19) replaces the attention map with an identity matrix to form the guidance signal, while Smoothed Energy Guidance (SEG) [18] applies Gaussian blurring to the attention maps. Although these techniques enhance image quality without additional training or auxiliary models, their impact on image quality and prompt alignment remains limited compared to CFG. In contrast, we show that TPG provides a stronger guidance signal, bridging the gap between the effectiveness of CFG and that of training- trining-frre guidance methods both in terms of image quality and alignment with the input condition.
3 Background
Diffusion Models Denoising diffusion models generate samples from a data distribution Pasta (x) by reversing a gradual noising process [1, 25). In the forward process, a clean sample Xo Pdata is progressively corrupted into noise Xy through a stochastic differential equation (SDE):
dx f(x, t)dt + g({)dw,
where f(x. t) and gt) are predefined functions representing the drift and diffusion coefficients, and dw is increment of the standard Wiener process. A widely adopted formulation is the variance- preserving (VP) diffusion process [3], where x, gradually approaches an isotropic Gaussian as + T. typical parameterization f(x, f) = - 1 (t)x and g(t) =
/B(t) where ((t) defines the noise schedule. With this formulation, the denoising process given by the following reverse-time SDE:
dx -
3(t)x -1-
B(t log dt /B(t)dw
(2)
where Vx log pr (x) is the score function, representing the gradient of the log- bog-density of the noisy data at time f. This gradient indicates the direction in which a sample should be updated to increase its likelihood w.r.t. the noisy data distribution. The unknown score function is typically approximated by a neural network sx(x t). trained using denoising score matching [30]. Given a noisy sample x, at time i, the network predicts the score of the marginal distribution pr. Training is performed by minimizing a weighted denoising score matching loss:
1 (()) = 2 2.(" [(t) E ExxePeap(x (xo) 11000 (xe, Vx, log Prjo Prjo xxx [[arr,
(3)
where (x Xy) is the known Gaussian transition kernel of the forward SDE. Once trained, sam- ples are generated by integrating the reverse SDE using so(x, () R Vx log (x) as an approximation to the score function. Conditional generation is enabled by training a score network that receives additional conditioning signals, c.g., class labels or text prompts, input. In this case, the score network Sy (x, f, c) approximates the conditional score V p: c).
Guidance Although diffusion models trained via denoising score matching have strong theoretical foundations, the score approximations are often inaccurate due to limited model capacity. As result, unguided sampling using the learned score function tends to produce blurry and low- fow-fidelity images, especially in complex tasks such as text-conditional generation. To address these limitations and improve generation quality, classifier-free guidance (CFG) [17] was introduced to steer the reverse diffusion trajectory toward higher quality outputs by linearly interpolating between conditional and unconditional score estimates. In general, guidance in diffusion model can be defined as:
50(X,, c) st (xx, c) + 71s (xe, c, sg (xe, c, t)],
(4)
3

---

Algorithm Token Perturbation Guidance (TPG) for Diffusion Models
Require: Noisy input XT I), shuffling matrices Sk,t, of perturbed layers L, score function (denoiser) so, guidance scale y, total time steps T
1: for t = =T...., do
2: I/ Forward pass without perturbation
3: sg (x+) 4 59 (x,,1))
4: // Forward pass with perturbation
5: Run the network se (x{xx) a second time with the following modification to have sy (x:):
6:
for each layer do if k E C then Apply token perturbation: H H Il Apply token perturbation guidance 59(x,,t) t), + (x,) + 7 (xt) s0 (x())
7:
8:
9:
10:
11:
// Update sample
12:
X + SolverStep(x,, 5x(xe,,))
13:
return 250
where $i (Xe, c, estimates the desired direction (typically the conditional score sp (X,, (x,, c,)). and Ag (Xe, C, 1) acts as a negative score. In such guidance methods, samples are effectively pushed more toward a positive signal and away from negative signal. CFG, by assigning H6 (X,, G, () = s0(x,, 2, t), the samples are pushed away from the score of the unconditional data distribution and more strongly toward the given condition. Other guidance techniques implement sg (x,, t) with less-trained diffusion model [16], or by perturbing attention maps or input pixels [19, 18, 20].
Token perturbation guidance
We next introduce Token Perturbation Guidance (TPG), novel guidance method that directly perturbs token representations within the diffusion model. Unlike previous upproaches that modify model weights or attention mechanisms, TPG operates the intermediate token representations in the denoiser during inference. To improve sample quality, we use token shuffling as simple choice of the perturbation, which preserves global structure while disrupting local patterns. TPG is training- free and does not require any changes to the model architecture, effectively extending the benefits of CFG to broader class of diffusion models.
Let H BxXXC denote the intermediate hidden representations, where the batch size, N the number of tokens, and C' is the feature dimension per token. We apply the shuffling operator N along the token dimension to get H' SH. These shuffled tokens are used to define the negative score 5g (x,, c, f) in TPG. The shuffling operation satisfics the following properties:
Linearity: This ensures that the perturbations can be expressed as a matrix multiplication, enabling efficient implementation within the denoising process. Thus, TPG does not add any noticeable overhead to the sampling process, and it has practically the same sampling cost as CFG.
Norm preservation: The shuffling matrix is orthonormal, meaning it satisfies S = and acts rigid rotation or permutation in the embedding space. This property guarantees that the preturbation preserves token norms, i.e., we have |SS | As result, the magnitude of the feature representations remains unchanged, which helps maintain their statistical properties and prevents internal covariate shift [31].
Algorithm summarizes the inference procedure of TPG. At each timestep. two forward passes are performed: one standard, and one with token perturbations applied at selected layers via shuffling matrices Sk.r, unique at each time step / and each layer k. The outputs are combined to guide the denoising trajectory toward higher-quality samples. Accordingly, TPG can be applied both conditional and unconditional models and does not require additional training of the base model.

---

(
(b)
(c)
180
Figure 2: Analyzing the behavior of different guidance methods across denoising steps. (a) Cosine similarity between the added guidance term Ae in Eg = 6g + -ae and the true noise e. SEG and PAG exhibit negative alignment intermediate steps, while TPG and CFG maintain near-zero cosine values, indicating orthogonality to the noise. (b) Cosine similarity between the full guided score éœ and e. Compared SEG and PAG, TPG behaves more similarly to CFG across sampling. (c) lz norm of the guidance term Ae. TPG and CFG follow nearly identical trends, both starting around 40 and increasing steeply in the later denoising steps. contrast, SEG and PAG maintain consistently low norms throughout.
5 Comparing the behavior of TPG with other guidance methods
To better understand how different guidance strategies influence denoising, we analyzed their interac- tion with the true noise signal across various steps of the denoising process. In this experiment, we selected 1,000 images from the MS-COCO 2014 validation set [32]. Instead of starting from pure random noise, we corrupted each image with noise corresponding the noisy input. This noisy image was then passed through the denoiser to produce the guided output for various methods (e.g., SEG, CFG, and TPG). We analyzed the angle between the predicted and ground-truth noise and examined the frequency components by partitioning the spectrum (up to radius 0.7) into 29 bins. Importantly, we did not use the denoiser output to progress to the next step; rather, for each time step, we reapplied noise directly to the clean image.
specific time step t to generate
As shown in Figure 2, TPG and CFG produce guidance vectors that are nearly orthogonal to the ground-truth noise throughout the trajectory, indicated by cosine values close to zero. This aligns with observation shown in [33], which demonstrates that the CFG update term can be decomposed into parallel and orthogonal components. Importantly, the parallel component primarily contributes to oversaturation. is believed that orthogonality enables more effective steering of the predicted direction, and generally, the orthogonal component contributes to improved image quality. In contrast, PAG and SEG exhibit strong negative alignment during the middle steps, suggesting that they temporarily oppose the denoising direction. Figure (b) further shows that, compared to SEG and PAG, TPG more closely mirrors the behavior CFG in terms of alignment between the predicted and ground-truti noise. Additionally, Figure (c) shows that TPG and CFG exhibit similar guidance magnitudes across all denoising steps, while SEG and PAG show substantially lower norms. This indicates that the update terms in CFG and TPG more influential throughout the sampling process compared to that in PAG and SEG.
Figure complements the frequency domain. TPG and CFG remain almost perfectly orthogonal to the ground-truth noise across all frequencies and time steps, except for slight positive tilt in the lowest frequency bands. SEG, however, exhibits a clear negative stripe at medium frequencies during intermediate steps, confirming that its correction momentarily opposes the desired direction. The norm heatmaps further reveal that CFG and TPG inject a strong oow-frquenccy. signal in the early steps, while nigh-frreuency modifications primarily occur in the later denoising steps. In contrast, SEG operates with less energy and displays a markedly different norm pattern when modifying high-frequency content, indicating relatively weaker approach to detail refinement. This behavior aligns with Figure 1, where the reduced energy across frequency bands leads to overly smooth generations in the initial steps.
step-wise analysis by illustrating how each guidance method behaves in
In summary, these results show that the TPG update closely mirrors the behavior of CFG both in di- rection and frequency content across different denoising steps, suggesting more effective performance compared to previous training-free free guidance methods like PAG and SEG.
5

---

coside d) -
costée é) TPG
de. c) SEG
133
0.2
01.0
1.22
0.35
-03
loo(1 || CFG
log(1 + +|4e||-) (-ell---T TPG
log(1 + ( SEG
Frequency analysis of guidance residuals throughout sampling. Each heatmap shows either the cosine similarity between the guidance term Ae and the ground-truth noise e (top row), or the & norm of the guidance term (bottom row), function of frequency bin (horizontal axis) and denoising step (vertical axis; 1000 + 1). Top: For both CFG and TPG, the guidance term remains almost orthogonal to the noise across all frequencies, with mild positive bump in the lowest bands. In contrast, SEG transitions from weak positive alignment in the early steps to a pronounced negative stripe centered medium frequencies. Bottom: CFG and TPG concentrate most of their energy in the lowest frequency bin and inject significantly larger magnitudes than SEG, whose energy remains up two orders of magnitude smaller throughout the denoising process.
Figure 3:
6 Experiments
Setup All experiments are conducted via official implementations and pre-trained checkpoints provided by publicly available repositories. We build upon the current open-source state-of-the-at Stable Diffusion XL (SDXL) [5] as our primary baseline, and include the Stable Diffusion 2.1 (SD 2.1) [34] to show the generality of our method and analysis. Moreover, TPG is easy to implement, and it can be added into existing diffusion models as glug-and-plaa module with just a few lines experiments, the compared methods are evaluated using their original configurations and default guidance scales. The guidance scale for TPG is fixed at 3.0.
of additional code.
Metries We adopt Fréchet Inception Distance (FID) [35] as our principal metric because it correlates well with human preferences and jointly captures image quality and diversity. To quantify prompt alignment, we report the average CLIP Score [36] between sampled images and their corresponding prompts. Overall perceptual quality is further evaluated with the Inception Score and an Aesthetic Score [37]. Moreover, all experiments are conducted by generating 30k samples for each method (unless stated otherwise) and evaluated on the MS-COCO 2014 validation set [32].
6.1 Quantitative results
Table presents a quantitative comparison of TPG against vanilla SDXL, PAG, SEG, and CFG for both unconditional and conditional image generation tasks. In unconditional generation, TPG outperforms all compared methods by achieving notably lower FID and sFID scores (69.31 and 18.18. respectively), indicating better overall image quality and diversity. Additionally, TPG achieves the highest Inception Score (17.99) while maintaining a competitive Aesthetic Score, further reflecting the perceptual quality of TPG outputs. In conditional generation, CFG achieves the best results overall, with TPG closely following and consistently outperforming vanilla SDXI., PAG, and SEG in FID, sFID, and CLIP Score metrics.

---

Table 1: Quantitative comparison of TPG against vanilla SDXL [5], PAG [19], SEG [18], and CFG [17] for unconditional and conditional image generation. Lower FID and sFID scores indicate superior image quality and diversity, while higher Inception, Aesthetic, and CLIP scores indicate enhanced perceptual quality and semantic alignment. TPG achieves the best metrics in unconditional generation and closely matches the performance of CFG in conditional generation.
Setting
Metric
Vanilla SDXL [5]
PAG [19]
SEG [18]
CFG [17]
TPG (Ours)
FID4
124.04
98.83
82.64
69.31
sFID
78,91
94.71
74.98
44.18
Unconditional
Inception Scoref
9.19
13.74
13.22
17.99
Aesthetic Scoret
5.02
5,94
6.15
6.14
FID4
48.97
20.49
23.94
12.79
17.77
sFIDL
43.71
28.78
31.50
23.31
24.32
Conditional
Inception Scoret
22.10
34.66
30.29
42.75
34.89
Aesthetic Scoret
5.37
6.11
6.18
6,20
6.12
CLIP Scoret
27.47
29.67
29.49
32.03
30.15
Table 2: Quantitative comparison of TPG with vanilla Stable Diffusion 2.1 [24]. PAG [19], and
SEG [18] for unconditional generation. TPG outperforms other baselines all evaluated metrics.
Metric
Vanilla SD [24]
PAG [19]
SEG [18]
TPG (Ours)
FID4
25.24
21.30
20.98
16.69
Inception Score
24.59
28.80
25.15
36.28
Aesthetic Score t
5.07
5.93
5.83
5.97
Clip Score
27.74
29.03
28.53
29.30
Table 2 further compares TPG against vanilla Stable Diffusion 2.1, PAG, and SEG in unconditional image generation. TPG consistently achieves the best performance, significantly outperforming the other methods with the lowest FID score (16.69) and highest Inception Score (36.28). It also maintains competitive results in terms of Aesthetic Score (5.97) and CLIP Score (29.30), highlighting TPG's strength in producing high-quality and semantically aligned images.
6.2 Qualitative results
Figure 4 highlights the differences in unconditional generations produced by various guidance methods. Vanilla SDXL, PAG, and SEG often generate abstract or repetitive textures lacking clear semantic structure, while TPG consistently produces well-structured and realistic scenes. Across various initial seeds, TPG less prone to generating abstract patterns and more likely to form coherent spatial layouts with identifiable objects. Moreover, visual artifacts such unnatural textures or distortions are more noticeable outputs from baseline methods, whereas TPG reduces such artifacts, resulting in cleaner and realistic generations.
We also show a qualitative comparison of conditional image generation across different guidance methods in Figure 5. As can be seen, CFG and TPG consistently produce the highest-quality images, with sharp details and strong alignment to the text prompts. In contrast, other basclines such as SEG and PAG often generate outputs that deviate from the given condition, leading to less faithful semantic content. Furthermore, visual artifacts, such as distorted shapes or inconsistent textures, more frequently observed in SEG and PAG outputs, while TPG exhibits improved robustness, generating cleaner and more semantically accurate images with fewer artifacts.
6.3 Ablating other norm preserresnn perturbation methods
To assess the importance of shuffling during guidance, Table reports the results of various norm- preserving perturbation methods compared to the vanilla baseline. In addition to the shuffling strategy used in TPG, we evaluated Sign Flip, Hadamard, and Haar transforms, all described in Appendix A. While these three alternatives provide slight improvements in generation quality over the baseline, their gains are modest. In contrast, shuffling yields substantially larger improvement in both FID and Inception Score, indicating significantly better image quality and diversity. Although all perturbations are orthogonal and preserve token norms, they influence the features differently. Shufiling randomly reorders tokens, disrupting local patterns while preserving recoverable global
7

---

Vanilla SDXL
PAG
SEG
TPG (Ours)
Figure Qualitative comparison of unconditional generations produced by Vanilla SDXL [5]. PAG [19], SEG [18], and our method (TPG). TPG achieves more realistic generations compared to other training-fres free guidance methods.
Table 3: Comparison of different token perturbation methods evaluated using 5K samples.
Metric
Vanilla
Sign Flip
Hadamard
Haar
Shuffling
FID4
131.57
119.23 10.98
120.54 10.34
118.47 10.75
78.43
Inception Scoret
9.21
18.26
structure, which facilitates stronger guidance during inference. By comparison, Hadamard and Haar transformations mix all tokens together, potentially distorting useful information and weakening the guidance signal. Sign Flip merely alters the sign of each token, which may not offer strong enough signal to steer samples effectively toward desirable regions of the data distribution.
7 Conclusion and discussion
In this paper, we introduced Token Perturbation Guidance (TPG), novel, training-free method for enhancing the quality of diffusion models by directly perturbing intermediate token representations. TPG employs token shuffling to define effective guidance signal, extending the benefits of classifier-

---

Vanilla SDXL
CFG
PAG
SEG
(Our))
Prompt: "A man riding skateboard down brick sidewalk." "
Prompt: fire hydrant niddle large paved
Prompt: "A woman with glasses that is holding piece of bread."
Prompt: "A picture of sone shoes on table."
Prompt: "There is a nan playing tennis on the court."
Figure 5: Qualitative comparison of conditional generations produced by Vanilla SDXL [5], CFG [17],
PAG [19], SEG [18], and method (TPG). TPG is able achieve good quality and prompt alignment compared to other baselines such as PAG and SEG.
free guidance to a broader range of models. Through extensive experiments, we demonstrated that TPG improves the quality of both conditional and unconditional generation, while also enhancing prompt alignment in conditional setups. Our analysis further showed that, unlike existing attention- based perturbation methods such as SEG and PAG, the behavior of TPG closely resembles that of CFG in terms of the direction and frequency content of the guidance term. We thus consider TPG simple, aou-andd-lay method that effectively bridges the gap between existing training- guidance methods and CFG both in quality and prompt alignment.

---

References
[1] Jonathan Ho, Ajay Jain, and Picter Ahbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6833 2020. 1, 2,
[2] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint axXv:20010.0552,, 2020.
[3] Yang Song. Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint I. 2020. 3,
[4] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformen for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[5] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Rohin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint xr11:2330.00192, 2023. 2, 6, 7, 8, 9, 19, 20,
[6] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable vidco diffusion: Scaling latent vidco diffusion models to large datasets. arXiv preprint 15X15223 2023.
[7] Omer Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat. Junhwa Hur, Guanghui I Liu, Amit Raj, et al. Lumiere: space-time diffusion model for video generation. In SIGGRAPH Asia 2024 Conference Papers, pages 11, 2024.
[8] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Vigeodo: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408 06072, 2024.
[9] Sihyun Yu, Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, and Anima Anandkumar. Efficient video diffusion models via content-framt motion-latent decomposition. arXiv preprint arXiv:2403.14145 2024.
[10] Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Xintao Wang, Zhaochen Yu, Xin Tao, Pongfei Wan, Di ZHANG, et al. Videotetris: Towards compositional text-to-video generation. Advances in Neural Information Processing Systems, 37:29489-29513, 2024. 1
[11] Zeyue Tian, Yizhu Jin, Zhaoyang Liu, Ruibin Yuan, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. AudioX: Diffusion transformer for soything-to-auude generation. arXiv preprint arXiv:2503. 10522, 2025.
[12] Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[13] Ruikai Cui, Weizhe Liu, Weixuan Sun. Senbo Wang, Taizhang Shang, Yang Li, Xibin Song. Han Yan. Zhennan Wu, Shenzhou Chen, et al. NeuSDFusion: A spatial-aware generative model for 3d shape completion, reconstruction, and generation. In European Conference on Computer Vision, pages 1-18. Springer, 2024.1
[14] Ying- Tian Liu, Yuan-Chen Guo, Guan Luo, Heyi Sun, Wei Yin, and Song-Hai Zhang. PI3D: Efficient text-lo-3d generation with pseudo-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19915-19924, 2024.
[15] Zizheng Yan, Jiapeng Zhou, Fanpeng Meng, Yushuang Wu, Lingteng Qiu, Zisheng Ye, Shuguang Cui, Guanying Chen, and Xiaoguang Han. DreamDissector: I Learning disentangled text-to-3d generation from 2d diffusion priors. In European Conference Computer Vision, pages 124-141 141. Springer, 2024.
[16] Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with a bad version of itself. Advances in Neural Information Processing Systems, 37:52996-53021 2024. 1, 3,
[17] Jomathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXiv: 2022. 1, 2. 7, 9, 16, 19
10

---

[18] Susung Hong. Smoothed energy guidance: Guiding diffuusin models with reduced energy curvature of attention. arXiv preprini arXiv:2408.00760 2024. 2, 4, 7, 8, 16, 17, 18, 29,,0,22 21
[19] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang. Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. In European Conference on Computer Vision, pages 1-17. Springer, 2024. 2, 4. 7, 8, 9, 16,
17. 18, 19, 20,
[20] Susung Hong, Gyuseong Lee. Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1462-7471, 2023. 1.3,4
[21] Seyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, and Romann M Weber. No training, no problem: Rethinking classifier-free guidance for ditfusion models. arXiv preprint arXiv:2407.02687, 2024.
[22] Binxu Wang and John J Vastola. Diffusion models generate images like painters: an analytical theory of outline first, details later. arXiv preprint vrXiv:2303.024999 2023. 2
[23] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022. 2. 7, 17, 18
[25] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-226 2265. pmlr, 2015. 2,3
[26] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[27] Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. 2
[28] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffusion models through condition-annealed sampling. In The Twelfth International Conference on Learning Representations, 2024. 2
[29] Tuomas Kynkätänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.00000 07724, 2024. 3
[30] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674 2011.
[31] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015. 4
[32] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. 1500755.600 Springer, 2014. 5,6
[33] Seyedmorteza Sadat, Otmar Hilliges, and Romann M Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations, 2024. 5
[34] CompVis. stable-diffosion---12 http:://hhanggggaacee.cco////t/s
2022. 6
[35] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 6
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR,
2021. 6

---

[37] Christoph Schuhmann. Improved Aesthetic Predictor: CLIP+MLP Aesthetic Score Predictor, 2022.
Accessed: 2025-06-06.
12

---

A Orthogonal token perturbation matrix designs
TPG directly applies structured perturbations to the intermediate token embeddings within the denoiser during inference. Specifically, consider the intermediate hidden- activations of the denoiser at a given layer, represented as tensor H Bx where B denotes the batch size, N the number of tokens, and C' the dimension of each token's 's feature vector. At each denoiser layer and diffusion time step we apply the perturbation by multiplying the token-embedding with an orthogonal (or approximately orthogonal) matrix PA. E R.NN along the token dimension:
=HP
The primary goal of these perturbations is to preserve global information flow while disrupting local correlations that may lead to overfitting or artifacts. To achieve this, we investigated four distinct perturbation methods for PR.::
Token Shuffling. Represented by a permutation matrix Sk. NNN, where it denotes the denoiser's s block index and i the time step. The permutation matrix rearranges the tokens by selecting exactly one token from each position and assigning it to new position; mathematically. this means that each row and column contains exactly one entry of "I", while all other entries are zeros. It simply changes the order of tokens without altering their magnitude or norm, satisfying:
S S Sk.. = I.
Random Sign Flipping. This perturbation method is defined using diagonal matrix ka.a RN*N, whose diagonal entries d, are drawn independently and identically from [+1.-1] Fach token's embedding is thus flipped in sign independently. By construction, the matrix DR.; ensures that the f2-norm of every token embedding is preserved, by satisfying the orthogonality condition:
D DR,DK,. D I.
- Transform (WHT). The WHT uses normalized Hadamard matrix W E R N whose entries are +1 /VN and which satisfies W T w = IN. When N is power of two (i.e. N = 2m),, we compute the transform of an N C' token matrix in m (Ng)(N) iterative
stages. We begin with (0)) X, and for each stage 8 = 1, I,..., ,, update
W
W (s-1)
W
(a-1)
j+2 1
j 1 N 2--
2-- +1
( 1
(=-1)
(6-1)
j+2e-1.3
After m stages, the result (m) equals W X. This structured, deterministic mixing redistributes each token's information uniformly across all others while preserving every token's fo-norm.
Haar- Orthogonal Perturbation. At each denoiser block k and diffusion time step i, we generate a dense orthogonal matrix Q E N by first sampling A ~ ~N 1) NxN and then computing its QR decomposition
= Q R. We set Q Q. Since the entries of i..d. Gaussian, the orthogonal factor Q is distributed uniformly (with respect to the Haar measure) over the orthogonal group ((N), ), and by construction
Qk. =1
This yields an isotropic rotation in the N-dimensional token-index space, mixing all token positions globally without changing their norm.
Fach method preserves the overall norm and energy of the embeddings but changes their local structure uniquely effective ways. Empirically, these operations break up small-scale noise patterns that the denoiser might overfit, while still carrying global structure for high-qualic sample generation. Among these methods, token shuffling typically provides the best overall performance: it is straightforward to implement, has minimal computational overhead, and consistently achieves significant improvements in both diversity and fidelity of generated samples.
13

---

Table 4: Effect of guidance scale a on generative performance. Increasing o improves both FID and Inception Score up to r=3, after which performance degrades, suggesting an optimal balance between diversity and fidelity at moderate noise levels.
FID
SFID
Inception Scoret
Precisiont
Recall1
0
136.01
86.42
7.48
0.21
0.31
84.05
68.35
16.05
0.38
0.37
2
77.25
70.81
17.39
0,45
0.35
3
76.00
75.92
18.47
0.45
0.37
4
77.62
80.29
17.87
0.44
0.30
5
79.44
83.23
17.11
0.43
0.32
6
81.52
86.63
16.39
0.41
0.30
7 84.17
90.15
15.39
0.40
0.28
Table 5: Quantitative comparison of Vanilla SD3, TPG, and PAG for unconditional generation, evaluated over 5k generated samples. TPG achieves the lowest FID and highest Inception Score, showing improved fidelity and diversity over other baselines.
Method
FID
sFID
Inception Scoref
Precision
Recall
Vanilla SD3
113.86
91.09
11.06
0.26
0.28
PAG
138.08
216.65
9.13
0.25
0.15
TPG (Ours)
83.01
71.59
13.34
0.46
0.42
Table Quantitative comparison Vanilla SD3, CFG, TPG, and PAG for conditional generation. evaluated over 5k generated samples. CFG achieves the highest precision, while TPG attains balanced trade-off between quality and diversity.
Method
FIDJ
sFID
Inception Scoret
Precision
Recall
Vanilla SD3
33.81
63.17
29.62
0.45
0.45
CFG
21.22
58.14
42.10
0.70
0.42
PAG
41.32
117.03
25.65
0.42
0.30
TPG (Ours)
21.37
57.01
34.49
0.56
0.49
B
More ablation
Effect of guidance scale TPG We investigate the impact of the guidance scale the quality and diversity of generated samples. As shown in Table 4, increasing a from to 3 consistently improves both FID and Inception Score, indicating better fidelity and diversity. The performance peaks at d=3 with the lowest FID (76.00) and the highest Inception Score (18.47), suggesting an optimal trade-off between structure preservation and generative diversity. Beyond this point, larger guidance scales (a > 3) lead to gradual degradation across all metrics, implying that excessive stochasticity introduces artifacts and reduces consistency in generation.
Compatibility with ViT-based models To evaluate the generalization ability of TPG beyond the U-Net architecture, we compare it with existing perturbation-basee guidance methods, including PAG, using the Stable Diffusion 3 model. The results for both unconditional and conditional generation shown in Tables and 6. Unlike prior perturbation based approaches that were specifically designed for U-Net and fail transfer effectively. TPG demonstrates strong cross-architecture compatibility. In the unconditional setting, TPG achieves the best overall performance, significantly reducing FID and improving Inception Score compared to both Vanilla SD3 and PAG. the conditional setting, TPG performs competitively with CFG, achieving comparable FID and recall while maintaining high visual fidelity. These results confirm that TPG generalizes effectively across different architectures and retains its effectiveness without modification.
14

---

Table 7: Quantitative comparison of shuffling applied to down, mid, and up layers for unconditional generation using SDXL, evaluated over 5k generated samples.
Layers
FID
SFID
Inception Scoret
Precisionf
Recallt
Down layers
76.00
75.92
18.47
0.45
0.37
Mid layers
100.80
142.40
12.27
0.29
0.32
Up layers
102.28
154.87
12.81
0.30
0.23
Table 8: Quantitative comparison of Token Shuffling (ours), Token Blurring. and Vanilla for uncondi- tional generation using SDXL., evaluated over 5k generated samples. Token Shuffling substantially improves all quality metrics, demonstrating its effectivennes in enhancing generative fidelity and diversity.
Method
FID4
sFID4
Inception Scoret
Precisiont
Recallf
Vanilla
136.01
86.42
7.48
0.21
0.31
Token Blurring
157.67
184.40
6.70
0.18
0.23
Token Shuffling (ours)
76.00
75.92
18.47
0.45
0.37
Ablation on layer selection Following prior works such as SEG and PAG for U-Net-based models, examined perturbing the downsampling, mid, and upsampling layers separately. Table shows that TPG is most effective when the perturbation is applied only to the downsampling layers (i.e., the encoder part of the U-Net). We also experimented with combinations of down, mid, and up layers, but these did not lead to improvements and, in some cases, resulted in degraded performance.
Exploring non-norm preserving perturbations We further investigate the effect of non-norm preserving perturbations by introducing Token Blurring, which applies Gaussian blur kernel to the input tokens. As shown in Table 8, Token Blurring leads to clear degradation across all evaluation metrics, with higher FID and lower Inception Score compared to our Token Shuffling strategy. In contrast, Token Shuffling maintains feature magnitude consistency while introducing structured diversity, resulting in improved fidelity and perceptual quality. These results indicate that preserving the token norm during perturbation crucial for stable and semantically coherent generation.
C Limitations and future work
Despite these strengths, with CFG itself, TPG still requires two forward passes through the diffusion network, leading to increased sampling time compared to the unguided case. Additionally, although TPG significantly improves quality in most situations, the guidance term may remain limited in extreme out-of-distribution scenarios that are not captured by the learned distribution of the base model. We consider addressing these limitations interesting direction for future research.
D Societal impact
Generative modeling, particularly in the domains of images and videos, holds immense potential for misuse, raising important ethical concerns. While advancements in sample quality, such as those achieved through our method, can make generated content more realistic and convincing, this heightened believability can unfortunately facilitate the spread of disinformation. Such misuse may have far-reaching negative effects on society, including the amplification of existing stereotypes and the inadvertent reinforcement harmful biases. Although our improvements do not introduce entirely new uses for the technology, they may nonetheless increase the risk of these unintended consequences. therefore crucial to remain vigilant and consider the broader societal impacts that enhancements in generative modeling capabilities might entail.
15

---

CFG
PAG
SEG
TPG (OURS)
Matted
Thaestops
981
941
901
861
821
781
741
701
601
501
301
1
Visualization of the denoising process time for different guidance strategies: CFG [17], PAG [19], SEG [18], and our proposed TPG, Each row shows generated images at various denoising time steps, from 981 (left) (right). The text prompt used is "A red stop sign underneath green street signs".
Figure 6:
(a)
(b)
(c)
284
Drensiin
Figure 7: Analysis of guidance behavior across denoising steps in unconditional setting, (a) Cosine similarity between the added guidance term Ae and the true noise e. (b) Cosine similarity between the full guided score éø and (c) &g norm of the guidance term Ae.
E Additional qualitative results
In this section, we provide additional qualitative results to showcase the effectiveness and adaptability of our Token Perturbation Guidance (TPG) method across different generation tasks and to compare its performance with other existing methods.
16

---

coside,
133
833
log(1 [Ac]|a] TPG
log(1 (5 PAG
log(1 |Al SEG
SEG
Figure 8: Frequency-ster analysis of guidance residuals unconditional setting. Each heat-map plots either the cosine similarity between the guidance term Ae and the ground-truth noise (top row) or the f-norm of the guidance term (bottom row) as a function of frequency bin (horizontal axis) and denoising step (vertical axis; 1000 1).
ENG
Tree calmin
TIG Ohan
Figure 9: Qualitative comparison of conditional generation based on Stable Diffusion 2.1 [24] produced by PAG [19], SEG [18], and our TPG.
17

---

3NG
S03
(0xs)
Figure 10: Qualitative comparison of unconditional generation based on Stable Diffusion 2.1 [24] produced by PAG [19], SEG [18), and our TPG.
Em
Figure 11: Qualitative comparison of face images based on Stable Diffusion 2.1 [24] generated by SEG [18] and by TPG under both conditional and unconditional settings. SEG [18] clearly produces unrealistic patterns in the generated faces.
18

---

Vanilla SDXL.
CFG
PAG
SEG
TPG (0ws)
Prompt: "Two young children playing Frisbee on beach."
Prompt: "A pair of parking meters that take credit cards as well as change."
Prompt: "A wonan flies kite very high in the sky."
Prompt: "Cows crossing country road behind large truck."
Prompt: blue yellow fire hydrant."
Figure 12: Qualitative comparison of conditional generations produced by Vanilla SDXL [5].
CFG [17], PAG [19], SEG [18], and our TPG.
19

---

Varilla SDXL
PAG
SEG
TPG (Ours)
oo
Figure 13: Qualitative comparison of unconditional generations produced by Vanilla SDXL [5].
PAG [19], SEG [18], and TPG.
20

---

Varilla SDXL
PAG
SEG
TPG (Ours)
Figure 14: More qualitative comparison of unconditional generations produced by Vanilla SDXL [5].
PAG [19]. SEG [18], and TPG.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are supported by detailed empirical analysis.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Section 7.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: Yes]
Justification: The paper provides all the necessary information to allow for full reproduction the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have included our code as ZIP file with the submission, and will make the implementation of our method publicly available.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the test details.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Calculating the error bars in our context requires excessive computational resources.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We provide hardware details in Section
22

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://wwrrr..cccbbiiii/caaccaaa
Answer: [Yes]
Justification: The paper conforms to the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader societal impacts is discussed in the appendix.
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper uses current pretrained models, and does not release any new model or data.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We well-established datasets and models with proper academic licenses.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper did not involve human participants, so no risks, disclosures, or IRB approvals required.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper uses LLMs only for writing, editing, formatting purposes.
23


Reviews:
[{"review_id": "5790ZJF2Re", "reviewer": "Reviewer_83Sp", "summary": "This paper introduces Token Perturbation Guidance (TPG), a new method designed to improve the generation quality and alignment of diffusion models. TPG addresses limitations of Classifier-Free Guidance (CFG), which requires specific training and is restricted to conditional generation. TPG works by applying a norm-preserving shuffling operation directly to intermediate token representations within the diffusion network. Authors examine their method in various experiments to show the effectiveness of the proposed methods.", "questions": "1. This paper lacks sufficient novelty and can be considered a simplified version of citation [18]. Moreover, the proposed method in [18] can lead to a semantically different model, whereas the method proposed in this paper is overly simplistic and may fail to bring any fundamental improvement. The proposed method in [18] should also considered as a baseline.\n\n2. The method does not adapt well to the ViT architecture. For instance, before and after the perturbation, the relationships between features are essentially only affected through the positional encoding, which may be functionally similar to simply shuffling the input. In fact, this should be included and discussed as a baseline.\n\n3. The paper should include a discussion on the impact of the perturbation scale. From the current perspective, the proposed method may lead to instability in the output, depending on how the perturbation is applied.\n\n4. The paper should further quantify the benefits of the proposed method during the \"early-to-middle\" training stages, rather than presenting only a single figure.\n\n5. The method shows only marginal improvements. The performance gap between the conditional setting and CFG should be analyzed and explained in detail.\n\n6. The paper does not provide a reasonable strategy for determining which feature layers (L) should be perturbed. This omission may lead to significant manual tuning effort.", "limitations": "yes", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Well-organized and Clear Writing.\n\n2. Interesting Idea.\n\nWeaknesses\n1. Limited Novelty: The method appears to be a \"vanilla\" version of the method proposed in [18].\n\n2. Unreasonable methodology: The proposed method is overly simplistic and straightforward, lacking a clear analysis of how it works and how it differs from other simple baselines.\n\n3. Limited Performance Improvement: The experimental results show only modest performance gains.", "was_revised": true, "final_justification": "Thanks for the explanation. I still believe that this paper is a simple version of existing method, therefore I keep my score unchanged.", "rebuttal": "We would first like to thank the reviewer for finding our proposed method to be an interesting idea and for acknowledging the well-organized and clear writing of our paper. We would like to address the concerns and questions raised as follows:\n\n## Comparison with AutoGuidance\nAutoGuidance [1] uses a smaller/weaker version of the same conditional model for guidance. This requires training a separate model with reduced capacity and shorter training time, as well as accessing two models during sampling. In contrast, our method involves no additional training or architectural modifications. Since a weaker version of Stable Diffusion is not publicly available, a direct comparison between our method and AutoGuidance is not feasible. Training such a model would exceed our computational resources, requiring several weeks on multiple high-end GPUs. On the other hand, our method is training-free and can be readily applied on top of any pretrained model, which we consider an important advantage of TPG over AutoGuidance.\n\n## Simplicity of TPG\nWe acknowledge the simplicity of our proposed method, as the reviewer noted. However, we view this straightforwardness as a key strength rather than a limitation. TPG offers a simple and easy-to-implement adaptation for existing diffusion models that is condition-agnostic and requires no additional training. It consistently improves generation quality across a range of setups, including both conditional and unconditional generation. Notably, we believe that one of the main reasons for the popularity and widespread adoption of classifier-free guidance is its ease of use. While CFG is limited to conditional generation and requires a specific training setup, TPG can be seen as a similarly practical approach that extends these benefits to a broader class of diffusion models.\n\n## Analysis on why TPG works\nAs discussed in the paper, our main motivation came from the observation that other guidance perturbation methods fail to mimic the behavior of CFG, especially during the early denoising steps. This limits their effectiveness and results in lower image quality and weaker prompt alignment across various setups. Unlike previous approaches that operate in weight space or attention modules, we chose to operate in token space by forming guidance signals through shuffling, which disrupts local patterns without altering the global statistics of the features. One of our most important and interesting findings is that both TPG and CFG maintain a near-zero cosine similarity with the ground truth noise during the initial and intermediate steps. This aligns with the concept of orthogonality, previously introduced in [2]. It is believed that orthogonality enables more effective steering of the predicted direction, and generally, the orthogonal component contributes to improved image quality. We will clarify this point further in the final version of the paper.\n\n## Concerns about marginal improvements\nWe have shown that TPG not only narrows the gap between CFG and other training-free methods in conditional generation but also achieves state-of-the-art performance in the unconditional setting. As illustrated in Figure 4, other methods often fail to produce semantically coherent images, whereas TPG consistently generates results with higher visual quality and semantic consistency. This is further supported by our quantitative evaluations, where TPG outperforms other training-free guidance baselines by a noticeable margin.  For example, Table 2 shows TPG achieves an FID of 69.31 compared to 124.04 for vanilla SDXL, which is nearly a 2$\\times$ improvement. Taken together, we consider the improvements achieved by using TPG to be both fundamental and significant.\n\n## Compatibility with ViT-based models\nWe would like to emphasize that our method is not limited by model architecture and is *compatible* with ViT-based models (e.g., Stable Diffusion 3) as well. While previous perturbation-based guidance methods (e.g., PAG and SEG) were specifically designed for the U-Net architecture and may not generalize well to other backbones, we found that this limitation does not apply to TPG. To demonstrate this, we conducted an experiment comparing TPG with other baselines using Stable Diffusion 3, and report results for both conditional and unconditional generation below.\n\n\n|               |    FID ↓   |   sFID ↓   | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:-------------:|:----------:|:----------:|:----------------:|:-----------:|:--------:|\n| Vanilla SD3   |  113.86    |   91.09    |      11.06       |    0.26     |   0.28   |\n| **TPG**       | **83.01**  | **71.59**  |    **13.34**     | **0.46**    | **0.42** |\n| PAG           |  138.08    |  216.65    |      9.13        |    0.25     |   0.15   |\n\n>**Table 1:** Quantitative comparison of Vanilla SD3, TPG, and PAG for unconditional generation, evaluated over 5k generated samples.\n\n|                    |    FID ↓   |   sFID ↓   | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:-------------:|:----------:|:----------:|:----------------:|:-----------:|:--------:|\n| Vanilla SD3     |  33.81  | 63.17  |      29.62       |    0.45     |   0.45   |\n| **CFG**         | **21.22** | **58.14** | **42.10**    | **0.70**    | **0.42** |\n| TPG             |  21.37  | 57.01  |      34.49       |    0.56     |   0.49   |\n| PAG             |  41.32  | 117.03 |      25.65       |    0.42     |   0.30   |\n\n>**Table 2:** Quantitative comparison of Vanilla SD3, CFG, TPG, and PAG for conditional generation, evaluated over 5k generated samples.\n  \nThis experiment shows that TPG not only achieves the highest quality in unconditional generation, but also performs nearly on par with CFG in the conditional setting. These results highlight the robustness of our method and confirm that TPG is not restricted to a single architecture. We will include this experiment in the final version of the paper.  \n  \n## Ablation on the guidance scale used in TPG\nWe assume the reviewer’s question refers to the guidance scale used in TPG, as our pipeline does not include any perturbation parameter. To address this, we have conducted a quantitative comparison across different guidance scales, and the results are given below:\n\n|         |  FID ↓  | sFID ↓ | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:-------:|:-------:|:------:|:----------------:|:-----------:|:--------:|\n| σ = 0     | 136.01 |\t86.42| 7.48| 0.21 | 0.31\n| σ = 1   |  84.05  | 68.35  |      16.05       |    0.38     |   0.37   |\n| σ = 2   |  77.25  | 70.81  |      17.39       |    0.45     |   0.35   |\n| σ = 3   | **76.00** | **75.92** |  **18.47**   |  **0.45**   | **0.37** |\n| σ = 4   |  77.62  | 80.29  |      17.87       |    0.44     |   0.30   |\n| σ = 5   |  79.44  | 83.23  |      17.11       |    0.43     |   0.32   |\n| σ = 6   |  81.52  | 86.63  |      16.39       |    0.41     |   0.30   |\n| σ = 7   |  84.17  | 90.15  |      15.39       |    0.40     |   0.28   |\n\n>**Table 3:** Quantitative comparison across different guidance scales (σ) for unconditional generation using SDXL, evaluated over 5k generated samples.\n\nWe will include this experiment as an ablation study in the final paper. Although a scale of 3 yields the best performance, TPG is robust to this choice, with other values also achieving reasonable results relative to the baseline ($\\sigma=0$).\n\n## Quantifying the benefits of early-to-middle denoising steps\nTo the best of our knowledge, there is no established metric that directly quantifies the impact of early-to-middle denoising steps in the sampling process. However, we believe these steps play a crucial role in shaping the semantics of the generated images. As such, improvements in FID and CLIP scores may be interpreted as indirect evidence of effective denoising in early sampling stages. As shown in Tables 1 and 2, our method achieves the best FID and CLIP scores in both unconditional and conditional generation settings compared with other perturbation-based guidance methods. This likely reflects the benefits of enhanced early-to-middle denoising and aligns with the behavior observed in CFG. Additional results are provided in the Appendix and more can be added upon request if needed.\n\n## Ablation study for layer selection\nFollowing prior works such as SEG and PAG for U-Net-based models, we tested perturbing the downsampling, mid, and upsampling layers separately. Our initial experiments showed that TPG is most effective when the perturbation is applied only to the downsampling layers (i.e., the encoder part of the U-Net). We also experimented with combinations of down, mid, and up layers, but these did not lead to improvements and, in some cases, resulted in degraded performance. In response to the reviewer’s question, we have conducted an ablation study on layer selection for SDXL in the unconditional generation setting. The results are provided below and will be included in the final version of the paper.\n\n|              |   FID ↓   |   sFID ↓   | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:------------:|:---------:|:----------:|:----------------:|:-----------:|:--------:|\n| Down layers  | **76.00** | **75.92**  |    **18.47**     |  **0.45**   | **0.37** |\n| Mid layers   |  100.80   |  142.40    |      12.27       |    0.29     |   0.32   |\n| Up layers    |  102.28   |  154.87    |      12.81       |    0.30     |   0.23   |\n\n>**Table 4:** Quantitative comparison of shuffling applied to down, mid, and up layers for unconditional generation using SDXL, evaluated over 5k generated samples.\n  \n  \n-------------\n>[1] Karras T, Aittala M, Kynkäänniemi T, Lehtinen J, Aila T, Laine S. Guiding a diffusion model with a bad version of itself. Advances in Neural Information Processing Systems. 2024 Dec 16;37:52996-3021.  \n[2] Sadat S, Hilliges O, Weber RM. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations 2024 Oct 3."}, {"review_id": "qDj4pfnnbE", "reviewer": "Reviewer_959p", "summary": "The authors propose a training-free, inference-time guidance procedure for improving the generation quality of diffusion models by applying unique random shuffling operations on inputs per timestep and model layer and merging the corresponidng outputs with the outputs of standard forward passes. Unlike classifier-free guidance (CFG), the proposed method does not require training with conditioning dropout and can be applied to unconditional generation. The proposed method outperforms existing unconditional inference-time guidance methods for image generation as measured by FID and Inception Score on SDXL, and in some cases is competitive with CFG for conditional generation.", "questions": "__Questions__\n\n* Did the authors experiment with only performing perturbations at a subset of layers or timesteps?\n\n* Did the authors conduct experiments with other (potentially non-norm-preserving) perturbations that destroy local information while preserving global information, e.g. repeating latents in blocks, Gaussian blur on latents, etc.?\n\n* Can the authors comment on the potential interplay between positional encoding and the shuffle operations? In particular, can the authors clarify whether shuffling is performed before any positional encoding is applied to tokens (meaning that local/positional information is actually destroyed and not just obfuscated)?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "__Strengths__\n\n* The proposed method appears to be very simple and can serve as a drop-in improvement for existing models\n\n* The authors identify characteristic behaviors of CFG when compared to attention perturbation guidance methods, including stronger emergence of both high-level (structure) and low-level (detail) features early in the generation process\n\n* The analysis of the othogonality of guidance vectors to ground truth noise is interesting. To some degree this echoes recent work (https://arxiv.org/abs/2410.02416, ICLR '25) highlighting the importance of the orthogonal component of guidance vectors for generation quality (albeit with respect to conditional prediction vectors)\n\n* The authors demonstrate that the proposed shuffling perturbation outperforms other norm-preserving perturbations\n\n\n__Weaknesses__\n\n* The experiments presented in Figure 2 are currently explained/motivated loosely in terms of finding a guidance method that \"behaves like CFG\"; it would be nice to see some additional explanation of why these behaviors are desirable for the generation process. In particular, I think the authors could briefly explain why orthogonality is desirable\n\n* The authors do not appear to mention at which blocks (k) or timesteps (i) they perform perturbations in the main body of the paper -- it seems like all layers and steps, but this could be clearer. Similarly, the description of the shuffle operation in the main body of the paper also lacks detail -- readers will likely wonder \"is it a simple random permutation of all tokens, or some localized shuffling variant (as implied in line 120)?\"\n\n* Given the simplicity of the proposed approach, it is disappointing that the authors evaluate it in only two models (both Stable Diffusion variants, both trained with the same V-objective parameterization as far as I can tell, both presumably trained on similar data). For example, while it might not permit the same analysis in terms of guidance vector norms, orthogonality, etc., guidance is commonly used for masked language modeling / discrete diffusion for both images and text, and it would be interesting to see whether token perturbation guidance produces similar improvements.\n\n* CFG is evaluated in Table 1 for conditional generation with SDXL but not in Table 2 for conditional generation with SD2.1", "was_revised": true, "final_justification": "The authors addressed the majority of my concerns in the rebuttal (e.g. generalization to different architectures, comparison with other perturbation methods) and committed to clarifying key aspects of the paper. As a result, I have raised my score.", "rebuttal": "We appreciate the reviewer’s thoughtful comments and their recognition of TPG’s strengths. Please find our detailed responses below, and we would be happy to engage in further discussion if needed.\n\n## Why is orthogonality desirable?\nWe thank the reviewer for raising this important point and would like to provide additional clarification, which we will also clearly highlight in the revised paper. The concept of orthogonality originates from the observations made by [1]. They demonstrated that the CFG update term can be decomposed into parallel and orthogonal components. Importantly, the parallel component primarily contributes to oversaturation, whereas the orthogonal component enhances image quality. Inspired by this insight, we analyzed the orthogonality of guidance vectors relative to the ground truth noise. Our observations indicate that both TPG and CFG maintain a near-zero cosine similarity to the ground truth noise during initial and intermediate sampling steps, demonstrating similar behavior. In contrast, SEG and PAG exhibit distinctly different alignment patterns. We discuss these results further in Section 4 of our paper, and additional analyses can be provided upon request.\n\n## How to perform the perturbation\nWe apply TPG to all time steps and down-sampling layers in the U-Net architecture, except for the initial layers, which we empirically found to degrade performance when perturbed. As such, these initial layers are excluded. To maintain consistency across architectures, we shuffle all input tokens before applying positional encoding in the transformer blocks. We will provide further details and explicitly clarify these design choices in the final version of the paper. Additionally, we have conducted an ablation study on layer selection for SDXL in the unconditional generation setting, with the results reported below.\n\n\n|                 | FID ↓   | sFID ↓  | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:---------------:|:-------:|:-------:|:-----------------:|:-----------:|:--------:|\n| **Down layers** | 76.00   | 75.92   | 18.47             | 0.45        | 0.37     |\n| **Mid layers**  | 100.80  | 142.40  | 12.27             | 0.29        | 0.32     |\n| **Up layers**   | 102.28  | 154.87  | 12.81             | 0.30        | 0.23     |\n\n>**Table 1**: Quantitative comparison of shuffling applied to down, mid, and up layers for unconditional generation using SDXL, evaluated over 5k generated samples.\n\n## Question about evaluation and model selection\nWe acknowledge the reviewer’s concern regarding our evaluation and would like to provide further clarification. Since most prior works, such as PAG and SEG, are specifically designed for U-Net–based diffusion models, we prioritized our experiments on SDXL and SD2, both well-established models, to ensure a fair experimental setup and comparability with prior work. In response to the reviewer’s suggestion, we have also evaluated the performance of TPG on Stable Diffusion 3, which employs a completely different training setup and a transformer-based architecture. The results of this comparison are reported below and will be included in the paper.  These results confirm the robustness and generality of TPG, showing that its effectiveness extends beyond any particular model or architecture.\n\n|               |    FID ↓   |   sFID ↓   | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:-------------:|:----------:|:----------:|:----------------:|:-----------:|:--------:|\n| Vanilla SD3   |  113.86    |   91.09    |      11.06       |    0.26     |   0.28   |\n| **TPG**       | **83.01**  | **71.59**  |    **13.34**     | **0.46**    | **0.42** |\n| PAG           |  138.08    |  216.65    |      9.13        |    0.25     |   0.15   |\n\n>**Table 2:** Quantitative comparison of Vanilla SD3, TPG, and PAG for unconditional generation, evaluated over 5k generated samples.\n\n|                    |    FID ↓   |   sFID ↓   | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:-------------:|:----------:|:----------:|:----------------:|:-----------:|:--------:|\n| Vanilla SD3     |  33.81  | 63.17  |      29.62       |    0.45     |   0.45   |\n| **CFG**         | **21.22** | **58.14** | **42.10**    | **0.70**    | **0.42** |\n| TPG             |  21.37  | 57.01  |      34.49       |    0.56     |   0.49   |\n| PAG             |  41.32  | 117.03 |      25.65       |    0.42     |   0.30   |\n\n>**Table 3:** Quantitative comparison of Vanilla SD3, CFG, TPG, and PAG for conditional generation, evaluated over 5k generated samples.\n\n## Unconditional generation results in Table 2\nWe thank the reviewer for raising this point. We have noticed that the mention of \"conditional generation\" in Table 2 is a typo; the results shown correspond to unconditional generation using SD2.1. Since unconditional generation is generally a more challenging task, our intention was to highlight the effectiveness of TPG in this setting. We will correct this in the revised version of the paper and also include results for conditional generation with SD2.1 for completeness.\n\n## Exploring other perturbations\nFollowing the reviewer’s suggestion, we conducted an additional experiment to explore perturbations that are not norm-preserving. One such perturbation is applying a Gaussian blur kernel to the input tokens, which we refer to as Token Blurring. As shown in the following table, Token Blurring demonstrates suboptimal performance compared to Token Shuffling in TPG. We present the results of this perturbation below, and we will include these findings in the paper.\n\n|           | FID ↓   | sFID ↓  | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:---------:|:-------:|:-------:|:----------------:|:-----------:|:--------:|\n| Vanilla   | 136.01  | 86.42   | 7.48             | 0.21        | 0.31     |\n| Token Blurring | 157.67 | 184.40 | 6.70          | 0.18        | 0.23     |\n| **Token Shuffling (ours)** | **76.00** | **75.92** | **18.47**  | **0.45** | **0.37** |\n\n>**Table 4**: Quantitative comparison of Token Shuffling (ours), Token Blurring, and Vanilla for unconditional generation using SDXL, evaluated over 5k generated samples.\n\n\n------\n>[1] Sadat S, Hilliges O, Weber RM. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations 2024 Oct 3."}, {"review_id": "dIO7fSQWJU", "reviewer": "Reviewer_gsL2", "summary": "This paper introduces a training-free way to steer diffusion models by applying a norm-preserving random shuffle to token embeddings during each denoising step. Because the shuffle matrix is orthonormal, token norms stay unchanged, so the method attaches to any frozen checkpoint without retraining or architectural edits and works for both unconditional and conditional sampling. A pair of forward passes—one standard, one shuffled—produces a guidance vector whose magnitude schedule and low-frequency focus closely mirror CFG, suggesting that TPG captures a similar gradient signal while remaining condition-agnostic.\n\nExperiments on SDXL and SD 2.1 show that TPG almost halves FID for unconditional generation and narrows the conditional gap to CFG to roughly five FID points, while clearly outperforming other training-free baselines such as PAG and SEG across FID, sFID, Inception Score, and CLIP alignment. Frequency-domain analysis confirms that the TPG guidance vector stays nearly orthogonal to ground-truth noise and tracks CFG across timesteps, and an ablation over alternative norm-preserving transforms finds random shuffling most effective. TPG still lags CFG slightly on prompt fidelity, but it extends high-quality guidance to checkpoints that lack dedicated training.", "questions": "N/A.", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. This paper introduces TPG, which perturbs the intermediate token embeddings by applying a norm-preserving shuffle matrix. Because the operation affects only the activations during inference, it can be inserted into any SD (or similar) checkpoint without retraining, fine-tuning, or editing the weights. The authors supply clear pseudo-code to show how the perturbation is applied.\n2. On SDXL, TPG almost halves the unconditional FID and improves Inception Score from 9.2 to 18.0. In text-conditioned sampling it closes most of the gap to classifier-free guidance (CFG): FID falls from 49 to 18, while the CLIP alignment score rises from 27.5 to 30.1, matching or exceeding other training-free baselines. Similar trends hold on SD 2.1, showing the method is not tied to one backbone. \n3. The paper measures cosine similarity and frequency-band alignment between the TPG guidance term and the ground-truth noise across denoising steps. The curves show that TPG stays almost orthogonal to the noise and matches the magnitude profile of CFG, whereas PAG and SEG point in the wrong direction. This supports the claim that token shuffling gives a CFG-like gradient signal.\n4. Results are evaluated on 30k samples, four complementary metrics (FID, sFID, Inception Score, CLIP/Aesthetic), two model backbones, and an ablation that compares four different norm-preserving transforms.\n5. Implementing TPG requires only a second forward pass and a few lines that insert the shuffle at chosen layers, so the computation and memory overhead is modest.\n6. Paper is well organised and easy to follow.\n\nWeakness:\n1. Although TPG narrows the gap, it does not yet surpass CFG in conditional generation: on SDXL, conditional FID is 17.8 for TPG versus 12.8 for CFG. The paper could discuss the gap and whether larger guidance scales or mixed strategies might close it.\n2. The only evidence for why a norm-preserving shuffle mimics the CFG gradient is empirical. A theoretical analysis—perhaps leveraging properties of orthonormal transforms in residual networks—would strengthen the contribution and clarify why the method work.", "was_revised": true, "final_justification": "The authors have addressed my concerns and questions about the paper. I think that TPG proposed in this paper is an interesting and novel method. Therefore, I believe that this paper deserved to be accepted.", "rebuttal": "We appreciate the reviewer’s recognition of the strengths of our method and are pleased that the paper was found to be well organized and easy to follow. Our detailed response is provided below, and we remain open to further discussion and questions.\n\n## Comparison between TPG and CFG\nIt is important to emphasize that CFG is specifically designed and trained to enhance alignment with prompts in conditional generation. As such, surpassing its performance in this setting is inherently challenging for general, training-free methods like TPG. Nevertheless, unlike CFG, which is restricted to conditional generation, TPG demonstrates superior performance over other methods in unconditional generation and significantly narrows the gap between perturbation-based guidance methods and CFG. Please also note that as mentioned in our response to Reviewer 83Sp, this gap is nearly eliminated in the case of Stable Diffusion 3, further highlighting the strengths of our pipeline.\n\n## Combining TPG with CFG\nThank you for the insightful suggestion regarding mixed strategies. Indeed, combining TPG with CFG is feasible, as they operate through complementary mechanisms and influence the model in distinct ways. Following the reviewer’s suggestion, we conducted additional experiments to evaluate the combined effect of TPG and CFG. Our results show that this combination leads to improved performance, and further gains may be achievable by optimizing the weighting scheme used for combining the predictions from CFG and TPG.\n\n|    Method    | FID ↓   | sFID ↓  | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:------------:|:-------:|:-------:|:----------------:|:-----------:|:--------:|\n|    TPG       | 23.82   | 61.32   | 34.94             | 0.53        | 0.47     |\n| **TPG + CFG**| **22.93** | **62.61** | **37.48**      | **0.55**    | **0.46** |\n\n>**Table 1**: Quantitative comparison of TPG and the combined TPG + CFG approach for conditional generation using SDXL, evaluated over 5k generated samples. \n\n## Impact of larger guidance scale on TPG\nEmpirically, we found that a guidance scale of 3.0 yields the best results and have therefore adopted it as the default value for TPG. Additionally, we conducted comprehensive evaluations across a wide range of guidance scales to investigate the effect of larger values, as suggested by the reviewer. As shown below, increasing the guidance scale beyond 3.0 did not lead to improved performance. Please also note that larger guidance scales lead to comparable performance to our optimal scale, demonstrating the robustness of TPG with respect to the guidance scale.\n\n|     σ     |  FID ↓  | sFID ↓ | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:-------:|:-------:|:------:|:----------------:|:-----------:|:--------:|\n|    3    | **76.00** | **75.92** |  **18.47**   |  **0.45**   | **0.37** |\n|    4    |  77.62  | 80.29  |      17.87       |    0.44     |   0.30   |\n|    5    |  79.44  | 83.23  |      17.11       |    0.43     |   0.32   |\n|    6    |  81.52  | 86.63  |      16.39       |    0.41     |   0.30   |\n|    7    |  84.17  | 90.15  |      15.39       |    0.40     |   0.28   |\n\n>**Table 2:** Quantitative comparison across different guidance scales (σ) for unconditional generation using SDXL, evaluated over 5k generated samples.\n\n## Analysis of the Behavior of TPG and CFG\n\nWe thank the reviewer for this question. As discussed in the paper, our primary motivation came from the observation that existing guidance perturbation methods struggle to mimic the behavior of CFG, particularly during the early denoising steps. This limits their effectiveness and results in lower image quality and weaker prompt alignment across various settings. Unlike prior approaches that perturb weights or attention modules, we chose to operate in token space by introducing guidance signals through shuffling, which disrupts local patterns without altering the global statistics of the features. That said, we acknowledge that our findings are mainly empirical, and a rigorous theoretical analysis of TPG would be a valuable direction for future work."}, {"review_id": "vdPwGOLouA", "reviewer": "Reviewer_fVGy", "summary": "this paper proposes a simple yet effective method to apply guidance to diffusion inference. in particular, the proposed method operates on the intermediate token representations using a shuffling operator. the experiments shows its effectiveness especially in unconditional generation scenarios.", "questions": "1. in the paper, S is defined as a shuffling operator, and is described as \"Shuffling randomly changes the order of tokens. This breaks local patterns but still keeps the global structure recoverable\" in line 218-220. but according to the statement between line 128-134, S seems not to be a transformation that is more than \"changing the orders\". what exactly is \"permutation\" and \"shuffling\" mean in the paper? extra analyses and visualization on the S matrix would be good.\n\n2. why the behavior of TPG closely resembles that of CFG (as shown in Fig 2, Fig 3)?  does it imply TPG is a special case of CFG? \n\n3. can TPG and CFG be used together? what would be the potential benefits (or problems)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strength:\n1. new inference guidance algorithm is an important yet under-investigated topic. \n2. very good writing, clear presentation which is easy to follow / to reproduce.\n3. contributions of the proposed method is well justified, either in theory or in experiments. \n\nweakness:\n1. in condition generation experiments, the method does not outperform the widely-adopted cfg, in terms of generation quality or computation efficiency.", "was_revised": true, "final_justification": "the rebuttal solves most of my concerns.", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and valuable feedback. We have addressed all comments in detail below, and we would welcome any further discussion.\n\n## Performance of TPG for conditional generation\nWe would like to emphasize that CFG is specifically designed and trained to improve prompt alignment in conditional generation. As such, surpassing its performance in this setting is inherently challenging for general, training-free methods, such as TPG. Nevertheless, unlike CFG, which is limited to conditional generation, TPG shows superior performance over other methods in unconditional generation and significantly narrows the gap between perturbation-based guidance methods and CFG. As noted in our response to Reviewer 83Sp, this gap is nearly eliminated in the case of Stable Diffusion 3, further highlighting the strengths and generality of our pipeline. Regarding computational efficiency, as the reviewer pointed out, there is no significant difference between TPG and CFG: both require two forward passes through the diffusion model to generate guidance signals. Moreover, since the shuffling operation used in TPG is extremely lightweight, the runtime for both methods is practically identical, resulting in negligible computational overhead from TPG.\n\n\n\n## Permutation vs. Shuffling\nWe apologize for any confusion caused. In our paper, the terms “shuffling” and “permutation” refer precisely to the same operation (i.e., randomly changing the order of tokens). We will revise the paper to consistently use a single term and clarify this explicitly in the definitions. Thank you for pointing this out.\n\n## Similarity between TPG and CFG\nWhile TPG and CFG exhibit some similar behaviors in the properties of their update terms (as illustrated in Fig. 2 and Fig. 3), TPG should not be viewed as a special case of CFG. The similarities primarily arise from their common ability to effectively steer predictions toward higher-quality samples. However, the underlying mechanisms of TPG and CFG differ fundamentally: CFG explicitly leverages conditional information (e.g., text prompts), whereas TPG introduces perturbations through random permutations (shuffling) in the token space. Consequently, TPG does not require any specific training and naturally extends to unconditional generation scenarios.\n\n\n\n## Combining TPG and CFG\n\nWe thank the reviewer for this insightful question. Indeed, TPG and CFG can be effectively combined, as they operate through complementary mechanisms. Following this suggestion, we conducted an additional experiment to investigate the combined effect of TPG and CFG. Our findings confirm that integrating CFG with TPG further enhances overall performance. However, it is important to note that combining these methods requires an additional forward pass through the model, resulting in improved performance at the expense of increased computational overhead. Further improvements may be possible by optimizing the weighting scheme used for combining the predictions from CFG and TPG.\n\n|    Method    | FID ↓   | sFID ↓  | Inception Score ↑ | Precision ↑ | Recall ↑ |\n|:------------:|:-------:|:-------:|:----------------:|:-----------:|:--------:|\n|    TPG       | 23.82   | 61.32   | 34.94             | 0.53        | 0.47     |\n| **TPG + CFG**| **22.93** | **62.61** | **37.48**      | **0.55**    | **0.46** |\n\n>**Table 1**: Quantitative comparison of TPG and the combined TPG + CFG approach for conditional generation using SDXL, evaluated over 5k generated samples."}]

Author rebuttal:

Meta-review: This paper proposes a token perturbation training-free guidance to apply a norm-preserving shuffling operation on tokens within the diffusion model to improve the generation quality. Experiments are conducted using SDXL and compare the proposed guidance with classifier-free guidance and also other related methods.

Reviewers commented positively on the writing quality, simple plug-in method, and the meaningful insight into the guidance behaviour of diffusion models. Reviewers queried about the novelty compared to similar works, the limited performance gain, the evaluation on limited baseline models, and the lower results for conditional generation. Most of the concerns have been well addressed in the rebuttal and using the supplemented results.

After the rebuttal, the remaining concern is about the distinction between the proposed method and an existing paper. Although they share some similarity in high-level ideas, the proposed method has satisfactory novelty given its different design and training-free property. This is the main reason for acceptance.
Decision: Accept (poster)