DeRaDiff: Denoising Time Realignment of Diffusion Models
Authors: Ratnavibusena Don Shahain Manujith
Teoh Tze Tzun
Kenji Kawaguchi
Yang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: alignment
diffusion models
Primary area: generative models

Abstract: Recent advances align diffusion models with human preferences to increase aesthetic appeal and mitigate artifacts and biases. Such methods aim to maximize a conditional output distribution aligned with higher rewards whilst not drifting far from a pretrained prior. This is commonly enforced by KL (Kullback–Leibler) regularization. As such, a central issue still remains: how does one choose the right regularization strength? Too high of a strength leads to limited alignment and too low of a strength leads to "reward hacking". This renders the task of choosing the correct regularization strength highly non-trivial. Existing approaches sweep over this hyperparameter by aligning a pretrained model at multiple regularization strengths and then choose the best strength. Unfortunately, this is prohibitively expensive. We introduce _DeRaDiff_, a _denoising-time realignment_ procedure that, after aligning a pretrained model once, modulates the regularization strength _during sampling_ to emulate models trained at other regularization strengths—_without any additional training or fine-tuning_. Extending decoding-time realignment from language to diffusion models, DeRaDiff operates over iterative predictions of continuous latents by replacing the reverse-step reference distribution by a geometric mixture of an aligned and reference posterior, thus giving rise to a closed-form update under common schedulers and a single tunable parameter, $\lambda$, for on-the-fly control. Our experiments show that across multiple text–image alignment and image-quality metrics, our method consistently provides a strong approximation for models aligned entirely from scratch at different regularization strengths. Thus, by enabling very precise inference-time control of the regularization strength, our method yields an efficient way to search for the optimal strength, eliminating the need for expensive alignment sweeps and thereby substantially reducing computational costs. The official implementation is available at https://github.com/itsShahain/DeRaDiff.

Published as a conference paper at ICLR 2026
DERADIFF: DENOISING TIME REALIGNMENT OF DIFFUSION MODELS
Ratnavibusena Don Shahain Manujith", Teoh Tze Tzun", Kenji Kawaguchi, Yang Zhang National University of Singapore (shahain, zeoh. .ze. tzz.,zun, tuun, yangzhang/ nus. edu ken j @nus. edu.sg sg
Realigned Model
DeRaDiff Approximation
Figure I: DeRaDiff' re-approximates model aligned from scratch. Top row consists of images generated by an SDXL model aligned from scratch at = 5000 KL regularization strength. Bottom row consists of images obtained via DeRaDiff sampling via anchoring SDXL model aligned at KI. regularization strength of 3 = 2000 with no further retraining.
ABSTRACT
Recent advances align diffusion models with human preferences to increase aes- thetic appeal and mitigate artifacts and biases. Such methods aim to maximize a conditional output distribution aligned with higher rewards whilst not drifting far from a pretrained prior. This is commonly enforced by KL (Kullback-Libleer regularization. As such, a central issue still remains: how does one choose the right regularization strength? Too high of a strength leads to limited alignment and too low of a strength leads to "reward hacking". This renders the task of choosing the correct regularization strength highly non-trivial. Existing approaches sweep over this hyperparameter by aligning pretrained model at multiple regularization strengths and then choose the best strength. Unfortunately, this is prohibitively expensive. We introduce DeRaDiff. denoising-time realignment procedure that, after aligning a pretrained model once, modulates the regularization strength dur- ing sampling to emulate models trained at other regularization strengths without any additional training or fine-tuning. Extending secoding-time realignment from language to diffusion models, DeRaDiff operates over iterative predictions of con- tinuous latents by replacing the reverse-step reference distribution by geomet- ric mixture of aligned and reference posterior, thus giving rise to closed- form update under common schedulers and a single tunable parameter, A, for on-the-fly control. Our experiments show that across multiple text-image align- ment and image-quality metrics, our method consistently provides a strong ap- proximation for models aligned entirely from scratch different regularization strengths. Thus, our method yields an efficient way to search for the optimal strength, climinating the need for expensive alignment sweeps and thereby sub- stantially reducing computational costs. The official implementation is available at i thub. com wtt www./itsthaaaain/eeeaaaaaaa
Indicates equal contribution. Subject matter correspondence Shahain Manujith Shbahaii@uunus.cduu>

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Text-to-image (T2I) diffusion models (Ho et al. (2020); Rombach ct al. (2022) now underpin state- of-the-art image generation. Sampling has been made efficient by techniques such classifier-free guidance and latent diffusion, unlocking applications like style transfer, image-to-iman tongeee translation, and inpainting (Dhariwal & Nichol (2021); Saharia al. (2022), Most modern systems are trained in two stages: (i) pretraining, which optimizes the diffusion objective on large-scale data; and (ii) aligmment, which adapts behavior to tasks or human preferences via supervised fine-tuning (SFT)
(Lee et al. ) or reinforcement learning (Black et al. (2023), Clark et al. (2023) )
persistent challenge in alignment balancing adaptation with fidelity the pretrained prior. This trade-off is typically controlled by proximity penalty- most commonly a Kullback-Leibler (KL) evverence--beteee the aligned and reference distributions. The associated regularization strength is pivotal: if too strong, the model under-adapts; if too weak, it drifts and risks reward hacking (Amodei et al. (2016)) 6); Stiennon et al. (2020); Bai et al. (2022); Lewis et al. (2020) ). Unfortunately, identifying the right hyperparameter generally requires expensive sweeps that are prohibitive for large diffusion models (Ho et al. 22020; Rombach al. 2022)).
To this end, propose DeRaDiff. a denoising-time realignment procedure. In the context of lan- guage modeling, realignment is defined as the post-hoc adjustment of the regularization strength 3 3-effectively modulating the proximity to the reference model by geometrically mixing the ref- erence and aligned distributions at inference time (Liu et al. (2024)). While this enables LLMs to vary alignment intensity via discrete logit manipulation, applying this principle to generative art presents a distinct challenge: diffusion models do not output single-step probabilities over a finite vocabulary, but rather operate via the iterative denoising of continuous latents. Our key insight is derivation of tractable, closed form formula for the geometric mixture of a reference and aligned diffusion models' distribution that is parameterized by A that adjusts the effective regularization strength relative to the aligned model's 's regularization strength, B. Crucially, À is tunable on-the-fly during inference.
As such, the closed-form update is presented in theorem | for the realigned reverse process, pro- viding both a theoretical basis and an efficient implementation (see Algorithm []] Quantitative (Section (5) and qualitative results (Figure 8 Figure 3) show that DeRaDiff preserves downstream performance while obviating retraining. Moreover, we achieve substantial compute savings, as de- scribed in scction Our contributions can be summarized as three-fold:
A theoretical extension of decoding-tine realignment to diffusion processes, yielding closed-form stepwise realignment posterior integrated into the reverse diffusion process.
DeRaDiff, denoising-time realignment method that approximates models aligned dif- ferent regularization strength without additional training by modulating alignment during sampling.
Experimental evidence that DeRaDiff enables precise control of alignment strength and accelerates Rtteeettlee hyperparameter exploration, substantially reducing compute while preserving downstream performance.
2 RELATED WORK
Alignment of diff'usion models. A growing body of work aligns diffusion models using prefer- ence signals or task rewards, including DDPO (Black et al. 2023). DRaFT (Clrk et 2023)
DPOK (Fan et al. 2023), AlignProp (Prabhudesai et al. 2023) and Diffusion DPO (Wallace et al.) training efficiency of alignment proce- dures. Central to their stability is the choice of regularization strength toward reference model: insufficient regularization permits distributional drift and reward hacking, whereby models score high rewards but fail on the intended task. (Amodei et al. (2016); Stiennon et al. (2020); Bai et al.
2023) These methods chiefly study the effectiveness
2022 Lewis et al. (202)) )
Decoding-time alignment of sampling distributions. To avoid retraining for each task or pref- erence setting, recent work has considered decoding-ting time control of the sampling distribution. One line of work leverages unconditionally pretrained diffusion models together with pretrained neural
2

---

Published as a conference paper at ICLR 2026
(a) 0=
(b) À = 0.75
(c) 1=1
(d) 2
(c) =
(a) 0=
(b) À = 0.5
(c) 1=1
(d) À 7.5
(c) A 10
Figure 3: On-the-fly modulation with DeRaDiff. Applied to SDXL (Podell et al. 2023). DeRaDiff adjusts alignment at inference via a scalar X. Increasing À decreases the effective regularization (more alignment to human preferences) and increases the aesthetic quality and increases prompt adherence expected, while maintaining À [0. (Top: "Typhoon in teacup... ", bottom: "A smiling beautiful sorceress... "). However, increasing beyond pushes the model beyond the aligned regime, resulting in degradation in aestheties and inducing reward-hackinge artifacts as expected for models trained on too low a regularization strength.
networks to enable diverse conditional generation tasks (He et al. (024)).. Another line of work em- ploys Sequential Monte Carlo sample from reward-aligned target distributions inference-time (Kim et al. (2025); Wu et al. (2023a)). While effective, these approaches generally do not exploit the presence conditional model that has undergone alignment.
Decoding-time realignment in language models. In language modeling, Liu al. (2024) in- troduced decoding- time realignment, offering theoretical framework and empirical validation for decoding-tine realignment of discrete next-token distributions. Our work differs by developing an analogous realignment mechanism for continuous diffusion trajectories, adapting realignment to the iterative denoising process and thereby enabling inference-tima control of regularization strength without additional alignment for diffusion models.
Decoding-time realignment in diffusion models. Diffusion Blend (Cheng et al. 2025 intro- duced a decoding-time realignment for diffusion models under the score-based SDE (Song et al.] 2020) paradigm. Our work differs by providing decoding-time realignment approach under the DDPM paradigm (Sohl-Dickstein et al. 2015). Although (Karras et al. 2022) established a theo- retical equivalence between DDPM and SDE paradigms, foundation for an exact closed-form Gaussian update on the stepwise realigned posterior under mild assumptions. To the best of our knowledge, work is the first to introduce realignment under the DDPM paradigm and provide a theoretical foundation for the stepwise posterior.
approach establishes the theoretical
3 BACKGROUND
Diffusion models. We follow the common latent diffusion formulation (Rombach al.
2022
given noise schedule parameters (orz, af ) a denoising diffusion model (Ho et al. (020)) Sohl- Dickstein al. (015)) defines Markovian reverse process. Here,
T
Po ( II
po (xe-1 xz),
Po(Ir-1 xx) = -N(x
T of
ال11) (1)
1),
3

---

Published as a conference paper at ICLR 2026
KL-regularized RL fine-tuning, Following Jaques et al. (2017) and Jaques et al. (2020). alignment commonly cast as reward maximization with a KL penalty to keep the fine-tuned model near a pretrained reference:
max serppeeeee ao)] Ke (
Pe
Here, DD is distribution of prompts and / > controls the trade-off between reward and proximity the reference model. Sweeping is the standard way to find the desired alignment strength but is computationally expensive; our method provides an inference-time tool to cheaply explore this space for Diffusion Models. The unique global optimum of Equation 2 for the discrete case was explored by Ziegler et al. (2019). KKrbbk et al. (2022), Rafailov et al. (202)) We provide a natural extension of the unique global optimum for the continuous case (see appendix A.2 for details):
Prer le)e (c)
p (zooc)
(3)
J Peer(zole)ee ( + dz
Realignment at decoding time Decoding- -imee realignment (Liu al. (2024) blends reference and an aligned model at sampling time. Extending this idea to diffusion models requires handling continuous per-step posteriors rather than discrete next-token distributions. In the next section, we derive a closed-form per-step Gaussian interpolation and give a complete sampling procedure (algorithm D
Full technical derivations are provided in appendix[A]
METHOD
4.1 REALIGNMENT & STEPWISE APPROXIMATION
We follow the formulation of decoding-tinge realignment (Liu (2024)) which expresses the realigned model as a geometric mixture of the reference and aligned densities. Concretely, the full-sample posterior is given by (see appendix A. for details):
P313/\(<zoc)) = Pref (e)ole)2-3 1 (m0
(4)
/ Prot Prot (xol((b)))2-- pāl p de
1- which is the normalized version of Piel pT PB Direct evaluation of Equation 4 is intractable for diffusion models as it requires marginalizing all intermediate latents. We therefore apply a stepwise denoising approximation and apply the same geometric mixture of the densities at each step:
Pree(Fe-1 we, c) 1 px] c)^
po [o13/N[sse-1 1
(5)
I Pror(x'-1 Ir, c) ( 1 c)* dx
Interpretation. Equation blends reference and aligned densities by raising each complementary powers. Equation [ applies an analogous idea at each denoising step, enabling sampling with the effect of alignment without retraining. The parameter À controls the KL regularization strength. When A = 0, the regularization strength 3/ //A is infinite, thus recovering the original Pref model (as seen in Equation 4.. When À 1, have 3/> B, which recovers the aligned model po]8). When 0 < À < 1, the new model pe (3/3 is an interpolation between the two models, which is the most stable and yields the best performance (see Figure 3) as it is convex combination between the log densities. When À > 1, then pa [3/ /3/A] uses a lower regularization strength than the strength with which the anchoring model po [3] has been trained with. However, this extrapolation process is no longer convex combination and may cause the covariance matrix (see thcorem to be non non-positive definite and ill-conditioned, which can lead to deterioration performance.
Assumptions. For the statements used in Theorem 1, we assume that the following are true: (i) per-step posteriors are well-approximated by Gaussians (scalar or diagonal variance) and (ii) the interpolation weight À is in the range of [0, 1] (because if À > 1, this corresponds to extrapolation and may cause performance degradation due to absence of positive definiteness of the new covariance matrix).
4

---

Published as a conference paper at ICLR 2026
4.2 DENOISING TIME REALIGNMENT
Theorem 1 (Closed-form per-step denoising realignment). Denoting pi po( u2
N (0 1 and of 0ip- البية oz for brevity, Let
Pre(FRe-1a4e = 11
pa13 (T = (111111122 02
Then, for any interpolation weight
[0, the stepwise realigned posterior
Pref Prof 22, (2,,c)1=x 1-X
(x (
It, c)a
(018/)(f6-11244) (2.
(6)
Prer(e'-11 1 c), 1ー p pb13) De, c) dx
is Gaussian with closed-form parameters:
Enew
1- A -1
)) 2 03
Pnew Exnon
{
(1
(1 in oz u2
A
0
Moreover deterministic scheduler posterior transform (including schedulers used by DDIM/DDPM samplers) preserves the Gaussian form of // (o13/\,, allowing the closed-form update above to be applied at each denoising step.
Proof sketch. Note that (2 c)¹ p0l8] c) exp (-2 |2 M1 m1ll +
A | - p2l 1)). We then define Enew = ( A )
I and Unew = Enew (لااا + P2). of
Following this, one sees that the product can be written as unnormalized Gaussian. Finally, using algebraic manipulation with respect to the integral, we arrive at a normalized Gaussian from which can sample. We note that Enew is guaranteed to be positive definite for À E [0, 1] and of, of > 0. Moreover, this same closed form update applies iteratively at each denoising step. We provide full and detailed derivation which is available appendix A.3
Corollary (Positivity and scalar simplification). If ofot,o3 > and X E [o, 1). then o > O and the interpolated posterior is valid Gaussian. In the isotropic (scalar) case, the o and Pnew are as follows
0f0² new (1
(
A
Mnew 2
2
P1
+
T H2
(8)
0i
02
which is implemented in algorithm
forces non convex combination, as such, since < 0. may cause the new covariance matrix to not be positive definite and ill-conditioned. But empirically, DeRaDiff continues to approximate a model with lesser effective regularization for moderate before instability occurs (see fig. (
Remark As seen in eq. 77.
Multi-reward extension We also prove that DeRaDiff can be extended to the very general case of multi-reward modelling (Ramé al. (2023). Jang al. (2023), MMicchell al. (2023)). A full proof given at appendix A.4
Pt, It-1s Mrg Pt-1 E RDD Note that need not be equal of our derivation handles this more general

---

Published as a conference paper at ICLR 2026
4.3 ALGORITHM
Algorithm DeRaDiff Sampling
Require: Reference model εq..: Aligned model Eggnss interpolation weight X E [0, 1J. prompt p.
guidance scale number of inference steps N, scheduler with timesteps { / and corre- sponding noise levels {or {o
f (ncode(p)
2: Cmull + Encode
Get unconditional embedding Sample initial latent from a standard Gaussian distribution
3: Bty ~ (0.
4: for N...., do
5:
I - 14 1pecv f 4-1
6: Tt 3
7:
e'el + Egg ot, Coull) + (Eo (xr, oe, Eom (I,, o2, Cmult)
8:
cuned x- Egmnl (Ii, Tt. Caull) y (Eoms (I2, (82200r 0z, c) Exrmd (2it, Ct, Crult))
Compute
Classifier-Free Guidance predictions for both models.
9: 11, of + SchedulerPosterior ( E L, (prov) 12, 22 < Schedulerrosterrior (2, sumed L, /pee)) Calculate posterior mean μ and variance σ² for the distribution at 4prev-
11:
nne
12:
Mnew + ohew
1-1 of M1 + oz H2)
13:
z =~,(0.
14:
Dtoo Knew + 2 Voncn
end for
16: Tout &- (AE.decode((::t))
return I
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Our experiments constitute the following steps:
1. Obtain reference and realigned models. We obtain public releases of model checkpoints (SDXL 1.0) and initialize the reference model Pret- Then, we use an arbitrary alignment method (eg: DiffusionDPO,, Wallace et al. (2023)) to align the reference model while mini- mizing the KI. divergence where the regularization strength B, which yields the realigned model pol { We also perform experiments on Stable Diffusion 1.5, which can be found in appendix A.8
Obtain outputs from Denoising Time Realignment. For given prompts e, we apply algo- rithm with varying À values to obtain samples from fo [3/A], allowing us to approximate various different regularization strengths without alignment from scratch.
3. Compare denoising time realignment samples against retrained models. We compare the downstream reward achieved by samples generated from pe[3/A| to those of po|3/A| which is model that aligned completely from scratch.
To comprehensively assess DeRaDiff's ability to approximate the performance of models aligned from scratch, we sample a batch of 500 prompts from union of the Pick-a-Pic v1 and HPS datasets and test DeRaDiff's approximation capability on three metrics which cover various aspects of image generation, namely PickScore. HPS v2 and CLIP. The SDXL 1.0 model is aligned at wide range of regularization strengths 8 {500, 1000, 2000, 5000, 8000, 10000), and at time, aligned model at a particular 3 used as an anchor model to approximate other alignment strengths. We do the same for SD1.5, whose results are provided in detail in appendix A.8

---

Published as a conference paper at ICLR 2026
PickScore errors, À E [0,
crrars tanget)
(pairs)
15
MAE
3.55 x 10
ITAL gairs gars +|0
Median |A]
2.83 x 10
0.6
75th / 90th
3.31 x 10
6.95 x 10
Max
1.10 x 10 3
Frac S 5x1001
86.7%
Frac S 10"3
93.3%
DCD0 ot
d,505 thewchild
PickScore mean PickScore sid
0. 0.230509
duccnn Brwebold
0.001 434
Figure 4: ECDF absolute PickScore errors DeRaDiff is used on aligned SDXL models.
A PPickScoreapprox
PickScore when
Reggdarraatiinn Streegth
F- I-Regelatiation Streagth
Reg.dariatiiin Strength
Figure 5: Line graphs for the average PickScore rewards gained by SDXL models realigned from scratch along with line graphs for the average PickScore rewards gained from DeRaDiff using chor SDXL models with 3 500 (left plot), B = 1000 (middle plot) and B = 2000 (right plot) regularization strengths.
5.1.1 PICKSCORE
PickScore (Kirstain et al. 2023) is caption-aware image reward model trained under a Bradley- Terry objective on pairwise preferences. Given a tuple (c, LA, LB, y). where e is the prompt, L and Ia are candidate images, and E {0, 1] indicates whether LA is preferred, CLIP-based encoder with an MLP head produces a real-valued score so(c, The induced preference probability is Pr(TA > TB c) = of (c, TA) so (c, TB)) with (--) the logistic sigmoid. We use PickScore as learned reward targeting human-perceived quality under the provided caption; unless otherwise stated, higher indicates stronger preference.
As seen in fig. 4 the typical approximation error is extremely small (median 2.83 x 10 ss 20% of the PickScore std) when DeRaDiff approximates human appeal to images on aligned SDXL models. Roughly 87% of approximations have errors I 10-4 DeRaDiff produces near- identical PickScore ratings for the vast majority of cases, meaning the human appeal of images produced by DeRaDiff and models aligned entirely from scratch are near-identical.
As seen in fig. 5 DeRaDiff able to meaningfully control the regularization strength on the fly without retraining by closely matching the SDXL models that were aligned entirely from seratch. Thus DeRaDiff enables testing of various regularization strengths without training, allowing one to search for the optimal strength, eliminating the need for expensive alignment sweeps. Moreover, using DeRaDiff, one can identify a promising range of regularization strengths and only align at these strengths, substantially reducing computational costs.
5.1.2 HPS v2
Human Preference Score v2 (HPS v2) (Wu et al. 2023b) is caption-aware preference model trained on the Human Preference Dataset v2 (HPD v2), a large-scale corpus of pairwise judgments designed to approximate human ratings of text-to-image outputs. HPD v2 comprises the order of 7.9 x 1 binary choices over 4.3 x 10s prompt-imas pairs spanning real photographs and generations from diverse T2I models. To this end, how well DeRaDiff matches human-preferene be-
7

---

Published as a conference paper at ICLR 2026
Smaller & Anchauy- Larrger Tangeix
Mosiity
0000-
0.000
0.00
K220
-15
0000
30
6:201
22500
20
6300
1..00
10
0208
10 010A-
0.877
45
0.00
0.283
0.239
0.291
0.202
02200
03113
Orgginll HPP (Sarge
(a) Scatter plot of approximated HPS Mean vs. original HPS (target) shows DeRaDiff approximations closely match human pref- erence scores.
(b) Bland-Altman plot of DeRaDiff approximations for mean CLJIP scores showing no systematic semantic fi- delity approximation bias.
Figure 6: Graphical plots of statistical analysis of DeRaDiff's approximations.
haviours of models realigned from scratch by presenting and analyzing scatter plot. As seen in fig. 6a each point (Ir, y). corresponds to a a specific approximation of a Starget SDXL model using specific Sanchor anchoring SDXL model using DeRaDiff where x corresponds the mean HPS score obtained by the DeRaDiff approximation of the Ptanget model and y corresponds to the mean HPS score obtained by the Bearget model. Moreover, spective log, (Shorget/Banchor) (Bnnhor)) value with the goal of encoding the gap between the regularization strengths of Banchor and Starget. Here we see that points lie around the identity line and the linear fit is close to it. This indicates that DeRaDiff is able to match and recover the human-pree preference scores of images from models aligned entirely from scratch. Moreover, inferring from the color scale, this indicates that approximations is near-identical or even better when an anchor ,3 approximates tar- get 33 that close-by, but performance degrades smoothly with increasing anchor-to to-target distance. Overall, this figure provides a faithfulness check: DeRaDiff enables low-cost, inference-tie align- ment that is able to preserve human preference outcomes of models aligned entirely from scratch. Detailed statistical analysis is provided in appendix A.8
colour code each point (I, y) with its re-
5.1.3 CLIP
CLIP (Hessel et al. 202 provides general-purpose text-image relevance score without explicit training on human preference pairs. For caption-image pair x). we compute the cosine similarity of normalized embeddings, $CLIP(p,
Encap (ncmpl)))
Ensea(p)] Enewa In our evaluations, CLIP is treated semantic fidelity baseline to complement preference-traiind metries (HPS v2, PickScore), helping to disentangle prompt adherence from aesthetic appeal. To demonstrate how DeRaDiff maintains semantic fidelity and that DeRaDiff has systematic semantic approximation bias when considering the preservation of semantic-fidelity we present a Bland- Altman comparison for DeRaDiff approximations on SDXL models in fig. 6b In this Bland-Altman plot, for each point (x, y) with label Banchor > Bporget- & refers to the average of (a) the CLIP score gained by the DeRaDiff approximation of a Starget reference model using a Banchor SDXL model the anchor and (b) the original CLIP score gained by the target Starget SDXL model. And refers to the dif- ference between (a) and (b), i.e. the difference between the CL.P score gained by the DeRaDiff approximation and the CLIP score gained by the target model that was aligned from scratch. We use a similar colour scheme for each point as was described section 5.1.2 Here, fig. 6b demon- strates that DeRaDiff approximations have negligible average bias and show very small absolute differences (maximum A x 4.5 x 10 30-3, 1.2% of Horig. where Porig is the mean of all CLIP values generated by models aligned completely from scratch) and that the Bland-Altman mean dif- ference is -0.001018, which is -0.273% of Morig. Furthermore, the 95% limits of agreement is [- [-0.004496, 0.002461). Further analysis is provided in appendix A.8. These results indicate that
8

---

Published as a conference paper at ICLR 2026
Alazed Ines wrradt 33p
Bellabui lren Bie 330 2000
Abgeed ivee smat p- 2032
Algand bas scsadh 100
DeltaDet buz p 185 2000
Algaed bos sasahi 2006
A red vintage bicycle leaning against a mossy brick woll.
A close-up up macro shot of honeybee on purple lavader flower.
Alanet
150
Dotaairr Abgred srett 2028
Alggne bux scmith
2356 Abgzed from urrent
A night sky eopossee showing viar truils above a lone free.
A red vintage cur driving along a constal cliff mad during sunset.
Figure 7: DeRaDiff undoes reward hacking. For each panel, left image from SDXL model aligned at 3 = 500 = 2000 using an SDXL anchor aligned 3 500, right = reference image from SDXL model aligned at / = 2000. The image details and style are successfully recovered by DeRaDiff.
center = DeRaDiff approximating an SDXL model aligned at
DeRaDiff preserves prompt-to-image semantic fidelity with no systematic bias, particularly when E [0, 1]. Taken together, these results show that DeRaDiff preserves prompt-to-image semantic fidelity within measurement noise in CLIP, thus rendering DeRaDiff once more capable of tuning the regularization strength on the fly during inference accurately preserving semantics all the while obviating the need to perform multiple costly retrainings.
5.2 QUALITATIVE ANALYSIS
As seen in table I DeRaDiff is capable of producing highly accurate training-free approximations particularly in the case for X E [o, 1]. and able meaningfully control the regularization strength at inference time on the fly. We provide further detailed statistical analysis in appendix A.88 Across both SDXL and SD1.5, mean absolute errors are extremely small (all 0.02 in absolute terms) and remain well below 0.5% when taken with respect to the respective means. The results show that DeRaDiff reproduces the average behavior of models aligned entirely from scratch for the case arbitrary anchor aligned at 3 2000.
[0, 1]. In [ show the performance of DeRaDiff We observe that the performance of DeRaDiff is generally stronger when applied to approximating models aligned with regularization strengths that higher than that of the anchor model. This is explained by the fact that when 0 S X < 1, DeRaDiff performs a convex combination, as seen in Equation 5 The experiments show that this interpolation is stable and is thus reliable surrogate to approximate the performance of models aligned at such regularization strengths. When À > 1, the combination not convex discussed in theorem [ This leads slightly less accurate approximations. Furthermore, as seen in fig. 5land fig. 7 our experiments demonstrate that DeRaDiff can provide a reliable approximation of models aligned from scratch even when using reward hacked model as the anchor. As reward hacked models have small / values, we can undo the effect of reward hacking by using the reward hacked model as an anchor and utilise a small À value to reverse the effect of reward hacking (as seen in fig. 77.. However, note that due to the stepwise denoising approximation and numerical approximation errors, one must also expect to see certain cases where the re-approximation may not be visually similar, even though the RLHF scores of the re-a re-appoxiimation closely match those from a model aligned completely from scratch. We further provide detailed evaluations of DeRaDiff's capability undo reward hacking in appendix(A.b6 A.6.2
6 COMPUTE SAVINGS
aligning a SDXL model at single 3 takes se 336 GPU hours, which is ze 52, 416 TFLOP-hour hours (FP16 Tensor-core equivalent) at a sustained load of 50%, 1.887 1020 floating point operations (+s 188.7 EFLOPs).. If naive pipeline
In experimental setup detailed in appendix A.7

---

Published as a conference paper at ICLR 2026
Table 1: Training free approximation of DeRaDiff when À [0,
CLIP
HPS
PickScore
Model
MAE
MAE (% of pi)
MAE
MAE (% of μ))
MAE
MAE (% of p1)
SDXL
0.001 604
0,430
0.000 770
0.265 000
0.000 355
0.154 000
SD1.5
0.001 557
0.448 000
0.001 175
0.425
0.000 718
0.332 000
Notes: MAE mean absolute error between DeRaDiff outputs and images generated by models aligned from scratch across all regularization strength anchors. For cach metric and model, is the evaluated mean metric value when aligned from scrateh across all evaluated regularization strengths; reported percentages are MAE divided by j.. very detailed statistical analysis is provided in
Tasks (Anchor 3 = 2000)
Target Model 8-values
500
1000
2000
5000
8000
10000
PickScore
Actual
0.2273
0.2314
0.2313
0.2311
0.2312
0.2308
Approximated
0.2243
0,2296
0.2313
0.2312
0.2309
0.2305
Absolute Difference (%)
1.3451
0.7831
0.0000
0.0611
0.1399
0.0987
HPS
Actual
0.2869
0.2919
0.2921
0.2916
0.2910
0.2903
Approximated
0,2852
0.2918
0.2921
0.2911
0.2902
0.2897
Absolute Difference (%)
0.5890
0.0299
0.0000
0.1701
0.2688
0.2061
CLIP
Actual
0.3628
0.3757
0.3752
0.3763
0.3751
0.3752
Approximated
0.3643
0.3738
0.3752
0.3751
0.3752
0,3751
Absolute Difference (%)
0.4022
0.5077
0.0000
0.3041
0.0310
0.0282
Table 2: Comparison of mean rewards achieved various metrics by using aligned [ 2000 SDXL model as an anchor. DeRaDiff closely matches the models that were aligned completely from scratch. In particular, when À < 1, the largest absolute percentage difference for PickScore, HPS and CLIP are 0.1399%. 0.2688%, 0.3041% respectively, thus demonstrating the accuracy of DeRaDiff's approximations.
aligns a single SDXL model N regularization strengths, the costs will scale N 336 GPU hours (or N x 188.7 EFLOPs). However, on the other hand, DeRaDiff requires aligning only once, thus the cumulative wall-time and FLOPs are reduced by factor of N. For instance, using DeRaDiff instead of naively aligning of N = 3, 5, 10 yields approximate GPU-hour savings of 66.7%, 80%, and 90% respectively, and EFLOP savings of 377.4 EFLOPs (N=3), 754.8 EFLOPs (N=5), and 1,698.3 EFLOPs (N=10) respectively. Thus, by using DeRaDiff for finding the optimal range of regularization strengths in place of a naive alignment sweep at B's, one capable of reducing the run-time and FLOPs by percentage of & V. which represents substantial saving in computational costs and run-time. However, DeRaDiff requires two forward passes at inference, but taken in totality, this overhead is still always smaller compared to full alignment sweeps. Moreover, this inference overhead can still be reduced by using prompt encoding caching parallelized inference.
CONCLUSION
In this work, we introduced DeRaDiff, theoretical expansion of decoding time realignment dif- fusion models, a framework enabling one to modulate the regularization strength of any aligned dif- fusion model on the fly without any additional training. We also provided experimental evidence that DeRaDiff enables precise and meaningful control of alignment strength and accelerates RLHF-style hyperparameter exploration while preserving downstream performance in terms of RLHF scores. We also demonstrated the substantial compute savings that DeRaDiff brings about. Thus, in conclu- sion, DeRaDiff yields an efficient way to search for the optimal regularization strength, eliminating the need for expensive alignment sweeps.
10

---

Published as a conference paper at ICLR 2026
8 ETHICAL STATEMENT
The authors have read the ICLR Code of Ethics and are committed to complying and upholding them. We only note two potential concerns: (1) pretrained image models and their training data might contain copyrighted content and also it may include societal biases, and (2) by lowering the computational costs involved in alignment, this can reduce the barrier to deployment and may increase the risk of misuse. The authors wish to inform that they strictly against such misuse and encourage responsible and safe use at all times without question. Furthermore, the authors only use pretrained models and datasets that are available the public and are committed strictly adhering to all model and dataset license restrictions.
REPRODUCIBILITY STATEMENT
The authors make every effort to make their work fully reproducible. To this end, the authors freely share the source code required run our experiments at github. wwmmitmsssst..eeee...e.ce htt.///tssaaaanDeeaaaa....
We also detail our experimental setup in ap- pendix A.7 Furthermore, we have used publicly available SDXL and SD1.5 checkpoints.
10 ACKNOWLEDGEMENT
Shahain Manujith would like to thank Teoh Tze Tzun for helping with proof reading, enhancing the flow of the paper with multiple additions, helping present mathematical results concisely, conducting final experiments and for refactoring the codebase. Shahain Manujith would also like to thank Dr. Yang Zhang sincerely for helping with research ideation, overseeing the progress of the research, providing weekly mentorship and feedback for manuscript drafts and providing access to compute.
Moreover, this material is based upon work supported by the Air Force Office of Scientific Research under award number 7A2386-24-1-4011 and this research is partially supported by the Singapore Ministry of Education Academic Research Fund Tier (Award No: T1 251RES2509).
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety, 2016. URL httphhss///axi...rrrrrr./s
606. 06565
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kemionn Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. htR.sttps:/// /ttss:///rxivvi www.vv.orrgaaaa.e...eee abs /ttpab////222222 2204. 05862
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion 13301
models with reinforcement learning, 2023.
Min Cheng, Fatemeh Doudi, Dileep Kalathil, Mohammad Ghavamzadeh, and Panganamala R. Ku- mar. Diffusion blend: Inference-time multi prefeeeee alignment for diffusion models, 2025. URL aatp::://ai....og//// wwtp////g/bb/aaeee http:///..o//////55......5 18547
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly tuning diffusion models on differentiable rewards, 2023. URL http:h:tttttt....oooo abs htt.b////299....
Prafulla Dhariwal and Alex Nichol. Diffusson models beat gans on image synthesis, 2021. URL htttpss//erxiv.orrr/b/////// wttpw/wwwwwwwwo/...oooo/ 05233
https://erxiis

---

Published as a conference paper at ICLR 2026
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Rein- forcement learning for fine-tuning text-to-image diffusion models. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Procesing Systems, volume 36, pp. 79858-79885. Curran Associates, Inc.,
2023.
https:://proceednnss.ee.... httpp//eeeeeeeeee//ppppeepeeepea /iles/paper//2223// File htl.e//ccooaass11443000Cooeeeee
cc Paper Conference. pdf
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In The Twelfth International Conference Learning Represenia- fions, 2024. URL peetps::/opperevviiee (eteenrerreeenee///oommdddddd-. htt/://rrmmmmooo000000
net
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. 2021. doi: 10.45550/ARXIV2104.057711 URL
abs///as//tt 2t1.aa////.1..//../ 08718
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc.,
2020. URL httpps://proceeddingg meweoeeeeedddnn....iiiio.....oo newrips wwwpeurips.ccppaaeeeeee cc/paper files/paper/2020. File/ #LL4/A/cccceeBB11000007100000002
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Han- nanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personal- ized large language model alignment via post-hoc parameter merging, 2023. URL https:
// //axxiv.com org/ abs////a/ 2310. 11564
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato Richard E. Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning sequence generation mod- els with kl-control. In Proceedings of the 34th International Conference Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 1645-1654. PMLR, Au- gust 2017. URL
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centrie dialog training via offline reinforcement
learning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3985-4000 Online, November 2020, Association for Computational Linguistics. doi: 10. 10.18853/v1220 1/2020. emnlp-main.327. URL https::t// httttpss//aalaatholooogoo
org/2020/ emn 1 ma i 327/
Tero Karras, Miika Aittala, Samuli Laine, and Timo Aila. Elucidating the design space of diffusion- based generative models. In Proceedings of the 36th International Conference Neural Infor- mation Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment of diffusion models without reward over-optimization. In The Thirteenth International Conference on Learning Representa- tions, 2025. URL http::/pppeneeeeeeeeeeeeeoeoeeee
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation, 2023. URL https: //arxiv. wwwpiiiorrg//baa abs///////// 2 2305. 01569
Tomasz Korbak, Ethan Perez, and Christopher L Buckley. RI with kl penalties is better viewed as bayesian inference, 2022. URL htttps://arxv..rrrrtr wttpg/g/ab///.aa/ 11275
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to text-to-image models using human feedback, 2023. ARI.tttss: /tt.avavvv///..////aa//..aaa/waa wrw. abtp://aa////..... 121 92.
12

---

Published as a conference paper at ICLR 2026
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmentte
generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1459-9474. Curran Associates, Inc., 2020. URL https::/prroeeedagg..eeee coppaer files/paper/2020/ E12.0000922220000000233333666663
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-tinge -ime realignment of language models, 2024. URLpttps: httttps://aaxiv.orrg.
wwwpx.xvvvvvo.o /bbs 220...2422..2 02992.
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. emulator for fine the-tuning large language models using small language models, 2023. URL
/tttp::/arraiv.orrr/b///3/3333 2310. 12962
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. hRtps:/:///aak http://wwwxwi.oo/ab/////aao
wttg http///a./222.... 01952
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-
image diffusion models with reward backpropagation, 2023. URL https://azxiv.org)
abs/ 0aa5/2310.0373
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023. URL wtt.:httts///axi...rrte abs 2230. 18290
Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, lean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by in- terpolating weights fine-tuned on diverse rewards, 2023. URL https::/arxxiv org/ abs/ 2306. 04488
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- resolution image synthesis with latent diffusion models. In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, IEEE, June 2022. doi: 00111099ccp15268.22222000000 2002201044
URL
doi
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image dif- fusion models with deep language understanding. 2022. URL httts:://arxi..er 2205. 1487
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.). Pro- ceedings of the 32nd International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL https:: htttps:://proceed.ngg. wwt.:mml....rr... press /vlee//v///// / html
Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2020. URL
13446
13456
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates. Inc., http://rr_f.eeeepppe//22222
2020. URL ht / wtwp:diiii.n.niuinueeeee fi Fooe
13

---

Published as a conference paper at ICLR 2026
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023. htth:tt:: / http://aavvvvvoooao org/ abs/2311.12900 12908
Luhuan Wu, Brian L. Trippe. Christian A Naesseth, John Patrick Cunningham, and David Blei. Practical and asymptotically exact conditional sampling in diffusion models. In Thirly-seventh Conference on Neural Information Processing Systems, 2023a. URU/ittps://ppeneiiie
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences text-to- image synthesis, 2023b. URL httph:t/////mm. //tpiv/aaivvvvviaa org/ abs /tbs/2336. 09341
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.
URL https: ptttpp://arxivi wwtpoogr/gggg.....////o 08593
14

---

Published as a conference paper at ICLR 2026
CONTENTS
Appendix
16
A.1
Re-expression of Realigned model
16
A.2
Unique Global Optimum for the Continuous Case
16
A.3
Proof of Denoising Time Realignment
17
A.4
Proof of Denoising Time Realignment when considering Linear Combination of Multiple Rewards]
18
A.5
An End-to-End process of finding the globally optimal X
20
A.5.1
Gaussian Process Surrogate
20
A.5.2
Acquisition Function
A.6
Additional Experiments
22
A.6.1
A fine grained examination
22
A.6.2
Undoing Reward Hacking
23
Detailed Experimental Seup
23
A.8
Metrics used for Detailed Statistical Analysis
25
A.9
Statistical analysis of DeRaDiff's performance on CLIP
26
A.9.1 SDXL
26
A.9.2 SDLS5
27
A. 10 Statistical analysis of DeRaDill's performance on HPS
28
A.10.1 SDXL
28
A.10,2 SDI.5
29
A.II Statistical analysis of DeRaDiff's performance on PickScore
30
A.I.I SDXL
30
1.2 SDL.5
31
A.12 LLM Usage
31
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 RE-EXPRESSION OF REALIGNED MODEL
This concerns the re-expression of the realigned model in terms of model that aligned from scratch. Thus, this shows a way to feasibly approximate an aligned model without training from scratch. From equation we see that e r(e,7o) Z(e)pp](((zolc)) p) and hence
Piet(aol) lc) (
Per (eeool))erraa
p Po13/\[(zolc)
S P (e)e )r(omb) deb
Prer(aale) (trlaz)) dxto
[ (100) S Parla' c) ( ( dxb
Pra((ra))) [P] (mo o) [[]]
(9)
Parlaal PP P (3(olle)/Paa(((())
da
A.2 UNIQUE GLOBAL OPTIMUM FOR THE CONTINUOUS CASE
This concerns the finding of the unique global optimum for equation This proof is seen in Rafailov et al. (2023), but where the partition function for the continuous case. Using the definition of the KL divergence, equation 2
simplifies to:
r(c, 20) Blog po(xol c) (rer(solc)
Pø max BewD Pa
E zxoo
log po(xole)
min E EEwDD
E Peoopol
B ((, Peet (zolc)
Pe
log po(wolc) 2(e) Prec Do Pre
min E E,ND,
E Bzo~Po(nale)
(2o)c)etr(c.o)) logg Z(c)
(10)
Pe
Here, the partition function is:
Z(c)= f Prer p drl
(11)
Now, define p"(oole) p" (xxooc) for all xo and p" (zale)dro 1.
1 Peollel which a valid probability distribution as
Then, since 7(c) is not a function of zo, bring the expectation inside:
po(zole)
P = min HowD. Ezompa(aolcc log
log Z(c)
Z(e) Prer (D ( ( tr(c,v)
min EawD [DKr.
(wolc)) log Z(c)]
(12)
Since the second term doesn't depend on ps. the minimum is achieved by the pe that minimizes the first term. Thus,
pø = pb = Z(c)
Pref (2o (ro
(13)
More specifically,
16

---

Published as a conference paper at ICLR 2026
po (3olc) = Prerl wo (eooeceedee le)
(14)
J Prer(z oo lcle (( day
a diffusion model that aligned with a regularization strength ( 0).
A.3 PROOF OF DENOISING TIME REALIGNMENT
This proof concerns the finding of a closed-form formula for
Dret(Tr- 120,, [1t,, - (x (xt- l0e, C)a
p = Pre ( (e, c
(15)
12,, 1 / dx
As noted previously, we have that
(r)( =N (M
o
oi
and
o
= =N ( ( c), o c oi
For ease of notation, denote Mo( t, c), P2 = ( t, and of = of t o
02.
Using the closed form of the isotropic multivariate gaussian distribution have that,
exp
1 x+--1 mill? 2n
P= = N(x of
(2TOT))122
and
exp
20
| - 12213
c) -2
D122
Define
1-1+ +
1-
1
Σ Enew
0
I.
(16)
of
(1-
(1
14 )
Pnew Enero
(17)
of
Now considering the numerator equation 15 we obtain
Pre((Er-12zz 12,, c)- P
c)*
exp
(18)
where through the application of equation 16 and equation 17
(1 ))
A | - 22l1
a
o
| - 11
+
022
X)
(1
+ 0.
||
f1 + 02 21122
x +
||
+
||
2
o
o
A)
(ft-1 Hmem
Emm (8e-1
[new)
P 5-1
Eneemnnee +
o
|m|
||
+
oz
|2م|
(19)
Pt. Tt-1, Mr, Pt-1 E RD
Note that need not be equal of our derivation handles this more general
17

---

Published as a conference paper at ICLR 2026
Recalling equation 18
now that,
exp {- (2
(1=00) ( 2D12
= (0 know) )r E (0e-1
Unew)
(20)
where
exp {
/ Snew
unll + o3 ||leal?) (1-X) (2Tot)(-A)D/1) ADD2
of
(21)
Note that expression yp a constant with respect to -- Similarly, we can also rewrite the denom- inator of equation T5 in the same way to arrive at
o exp
{ [( knew) T (F (new))
Po18/A((t-11 = [ exp {
1 (1/-11 - know) )2 D'ni 2 Mnew) dzt
D/2 1/2 cxp (at-1 - knew) E'm (I Hnew)
(22)
And thus we see that
p Po18/\\(e--1/2e c) = (xt- Uneww Enew)
(23)
A.4 PROOF OF DENOISING TIME REALIGNMENT WHENN CONSIDERING A LINEAR COMBINATION OF MULTIPLE REWARDS
We also consider the natural extension of decoding time realignment to DeRaDiff the case of multi-reward RLHF as proposed by Ramé et al. (0023). Jang et al. (2023), Mitchell et al. (2023).
Multi reward methods combine multiple models aligned independently using different rewards. Thus, consider the case of linear combination of rewards rx defined by
Tx(e, zo) Σ d 2c(
(24)
where we have K reward functions and where = Ak) E RK Then, considering the aligned model, Pš[B, X] under X,
Prer(zole) exp
J (he((afle)) exp [x(((, do'
Prer(zolc) exp 1 Σn Aeri
(25)
J (rella/le) exp ΣK Apro(e, dal Now, denoting pī.0 (xro(e) the model obtained by aligned reference model entirely from scratch using the hh reward, we have (as before) that,
exp 3 (c, 200 Z(c)pi,e (3)olc)) ((((oocc)/pr((zoc)).
(26)
Then, we note that,
ri vo)
exp X ri(c, 20) )
exp
B
(27)
pi (3
(zo cc) d:
Z(c)
Pret (20 c)
18

---

Published as a conference paper at ICLR 2026
In similar fashion,
ra m6)
pi [31(c/ I c)
d.
exp
II
Z(c)
(28)
8
Prof (2/ c)
Finally, we have that
P8 (solc)) Pret(2olc) II:=1 (Z(c) piol8((zale) d
Pror(
(29)
As Pi (
P [1 ((c)
drl
Peet
Then, letting X D Xs we have that,
Pb (((ooc)) = Pret(zole)** -., II: (Z())pt03)(200 c)) d
(30)
J Prer(sllcc)->) I1 ((c)pt, c)) dxb
dr (1 | e))
Prer(zol ()) >, IT
(31)
J (re( IT
(i,)1))<x o)) dx'
In similar fashion, due to the intractability of eq.
32 consider the stepwise approximation:
PāL3, (1 Prof(Ie- c)- II (pio(]](<) ( d d J P () 1)1 I (00 dry
(32)
Now, the DeRaDiff proof follows almost immediately noting that
1 || 1ill3
( (2, of (2nO?) D/2
exp
20
Now considering the numerator of equation 32 we have
Pref(ar-1 1>,,
II
k
exp
Di=1
K
20
2 1
(33)
II |T (2mo?)D12
Define
K
Enew
Σ
I,
(34)
K
A
Unew
Enew
H4
(35)
To simplify the exponent in the numerator of eq. (33) by defining = ΣK A
1
.s/12 and
applying equation 34 and equation 35 have
K
1 - 11l12
K
A Ui
1 -
-1-1 +
Σ
|
1=1
[ 1 [new) ) Ty- (xt-1 )new) The Mew
11u113
(36)
of
19

---

Published as a conference paper at ICLR 2026
Substituting the result above into equation 33
yield
Pref (122, 2-
II
1=1
exp -
[1: (2T0?) D/2
= xp exp {
( "TEm (xt-1 know }
(01-1 )mew
(37)
where
Mnew 5m "newr Unew (EK ae 1114112) |
exp
(2/0
Finally, we have that
1p exp { - (0 Mnew) ) E 5 know) }
p313, (613:7((211110000))
{ exp
exp f
(0/_1 Nnew )th Enen (oddnn(///20 (2/-1 knew)) dx
(/n)D/2|Enew/ 1// exp (0 )new) )T T (Fre-1 Mnew
(38)
which is the probability density of a Gaussian.
A.5 AN END-TO-END PROCESS FINDING THE GLOBALLY OPTIMAL X
One can even extend DeRaDiff by employing Bayesian optimization to find the globally optimum X" (and thus, the best regularization strength) that gives rise to the best downstream rewards. Here, constrain À [0, 1]. Here, outline the major ideas required to implement this.
We denote Px (x(x) as the generative distribution arising from using denoising time parameter X. Given reward function (eg.the downstream Pickscore by Kirstain al. 2023) ) our goal is:
X arg max J()), where J(A) (R()).
(39)
xc
Because (A) is expensive to evaluate (since this requires running the model on large batch of images and scoring using reward function), we treat it as a black-box function and use gaussian process optimization to find X in as few evaluations as possible.
A.5.1 GAUSSIAN PROCESS SURROGATE
We model the unkown objective f(A) = J(A) via gaussian process prior:
ʃ(A) ~ ((
(40)
For simplicity, we let mA) 0. Next, we use the RBF kernel k(A, = of exp (- (- with
2
I being the length-scale parameter and of being the signal variance. After n- many evaluations at points { yielding noisy estimates R. == J(A.), conditioning yields the exact posterior
(A) (AA, A) + o -R
(41)
aA k(A, A) k(A >) + -
k(),
(42)
where A [AIy...dAA] R [ RIn] and Ki k<>a;, Az)-
20

---

Published as a conference paper at ICLR 2026
A.5.2 ACQUISITION FUNCTION
To decide on which lambda value evaluate next, An+1, we maximize acquisition function a(A)
that balances exploration of the search space (high σx)) and exploitation (high pa):
1. Expected Improvement (El): Let f maxign Rj. Then
BI(A) (z) [ma f 03] (Jun - - (tt ((()) + on)(=))
(43)
where (Un /on. and Φ & are the standard Normal CDF/PDE.
2. Upper Confidence Bound (UCB): UCB(A) (nn (A),(();
with 3, chosen (e.g. B, v 2 2l0o((2222/6)) to guarantee sublinear regret.
We demonstrate run using SDXL aligned at B = 500 below. We leave this as an interesting avenue work in the future.
Bayesian Optimization: À vs. Score
0.2320
0.2312
PickScore
0.2304
0.2296
BO trials
0.2288
Best (2=0.312, Best PickScore=0.2321)
3rd-degree trend
0.2
0.4
0.6
0.8
1.0
Figure 8: DeRaDiff' + Bayesian optimization used to find the optimal regularization strength using an SDXL anchor model aligned at 3 500
21

---

Published as a conference paper at ICLR 2026
Algorithm 2 ID Bayesian Optimization for Global X Selection
Require: Domain A [0, 1], budget T. initial design size nu, reward evaluator R(-)
Ensure: Best weight X and estimate J(*)
1: Initial Design:
2: Sample {A, ] ~ Uniform({)
3: for L... do
4: Generate batch {xi.)) } from PA.
5: Compute R² [ Ej R(
6: end for
7:
Fit GP surrogate on {(A;, R=
8: for t= rig 1... do
9:
Compute posterior mean (A) and variance (A)
10:
Select next point via 1-D line search
X arg max EI(A 14-11 AEA
or UCB
11:
Generate batch from px., compute RS Append (Az, Rt)) to data and update GP
12:
end for
14:
return \ = max R
ST
A.6 ADDITIONAL EXPERIMENTS
A.6.1 A FINE GRAINED EXAMINATION
In this section, we train further 3 values the interesting region of 100 < 3 < 1500 where the human appeal rises fastest. Formally, sample the following 3 values: 250, 500, 750, 1000, 1250, 1500, 2000 and evaluate the performance of DeRaDiff on PickScore us- ing the experimental method detailed in section 5.1
0.24
0.24
1-28
0.23
0.23
PickSeare Moan
PickScare Mean
0.72
0.22
Original
Original
Approximation w/
Approximation w/
0.21
0.21
254
500
754
1000
1250
1500
1750
2000
250
300
750
1000
1250
1500
1750
2000
Figure 9: Line graphs for the average PickScore rewards gained by SDXL models realigned from scratch and also from DeRaDiff using anchor SDXL models with 750 (left plot) and = 1000 (right plot) regularization strengths.
22

---

Published as a conference paper at ICLR 2026
We see that this demonstrates consistent increase in the perceived human appeal (measured via PickScore) for both DeRaDiff approximations and as well as for models that were realigned entirely scratch showing that DeRaDiff can faithfully re-approximate models realigned entirely from scratch.
A.6.2 UNDOING REWARD HACKING
To demonstrate the capability of undoing reward hacking Namely, we use the SDXL. models aligned at B = 250 (severely reward hacked), / = 500 (moder- 2000 model as reference
three reward hacked models.
ately reward hacked), 9 750 (mildly reward hacked). We use model that is healthy (i.e. not reward hacked). Since reward hacking manifests itself as drastic dis- tributional changes contrast, vibrancy, colour, we use the Fréchet Inception Distance measure the degree of reward hacking and the extent to which DeRaDiff undoes reward hacking. To this end, we sample a batch of 1000 prompts from a combined dataset of HPSv2 and Partiprompts. Next, we measure the FID score between the outputs of the reward hacked models on these 1000 prompts giving rise to an average FID score for each reward-hacked model (in comparison 3 = 2000 model). Next, for each reward hacked model, we use it as an anchor to re-approximate the 2000 model and measure the average FID score. For each model, the difference in its respective FID scores will measure the extent of undoing reward hacking.
the healthy
Severnly Reward Hacked B-250
72.98
70
FTD (lower is better)
65
Moderataly Reward B-300
22.27
60-
55
14.48 (23.5%)
Mildy Reword
50
Hacked B=750
50.04
1.60 (3.2%)
46.43
50.71
47.13
Reward Hacked Models
DeRaDiff Approximation of 2000
Figure 10: DeRaDiff successfully undoes reward hacking
Table 3: DeRaDiff undoes reward- measured by FID (lower is better).
FID (reward hacked model
FID (DeRaDiff approx. against / 2000)
A
Model (reward-hacked)
against / 2000)
(abs)
(%)
Severely reward-hacked (3 = 250)
72.98
50.71
22.27
30.50
Moderately reward-hacked (33 500)
61.61
47.15 17
14.48
23.50
Mildly rewked-hacked (8 750)
50.04
48.43
1.61
3.20
Thus that DeRaDiff is capable of undoing reward hacking and this effeet of reward hacking much more pronounced in models that are severely reward hacked. fig. [1 we also provide more qualitative examples of how DeRaDiff can undo extreme reward hacking.
A.7 DETAILED EXPERIMENTAL SETUP
For experiments, we obtain public releases of Stable Diffusion 1.5 (SD1.5) from nuaxaymmmsstaaleedd.fffoooooooo
and Stable Diffusion XL 1.0 (SDXL) from We align them using DiffusionDPO (Wallace et al. 2023) on the Pick-a-Pic v1 dataset (Kirstain et al. 2023). Each alignment is per- formed on 2x NVIDIA A-100 80GB GPUs. The exact hyperparameters used follows:
23

---

Published as conference paper at ICLR 2026
Algond trn smrkc 2-050
Deikatiff 2von 2-120- 2000 Aligned fron scrich 2009
Alignd Ponn scriod d 250 Delbatiif Dvca d 2000
Allgted frves ecorch p 2000
1:
A whale breaching in frons of the Sydney Opera House.
A bookshelf.
hot air balloon landing in corn field.
beach with crvise ship passing by.
flook of geese fromt Alamo.
group of people walking walkway.
looking distant mountain.
Scottish castle loch.
Figure 11: DeRaDiff undoes reward hacking For each panel, left image from SDXL model aligned at B 250 (severely reward-hacked. center DeRaDiff approximating
SDXL model aligned at / 2000 using the reward hacked SDXL anchor aligned at 8 = 250, right = reference image from SDXL model aligned 2000. The image details and style successfully recovered by DeRaDiff.
Table 4: Aligning Hyperparameters for SDXL and SD1.5)
Parameter
SDXL
SDL.5
Pretrained VAE GPUs
madebyo 1 1 inn sdx1 wwwx--aee-fl6----
2
Per-device batch size
1
Gradient accumulation
64
64
Effective global batch size"
128
Dataloader workers Max train steps
16
16
30000
2000
LR scheduler
constant.xv th_warruppppppp
constant.Cd th_warmup
LR warmup steps Learning rate
200
500
x 10
x 10 8
Effective global batch size = NcPUs x train. batch x gradient accumulation steps. Thus SDXL: 2 x 1 x 64 = 128, SD1.5: x x 64 = 64.
Moreover, the Euler Ancestral Discrete scheduler for both SDXL and SD1.5. We use 50 denoising steps for both SDXL and SD1.5. Moreover, we use a guidance scale of 5 for SDXL and 7.5 for SD1.5
24

---

Published as a conference paper at ICLR 2026
A.8 METRICS USED FOR DETAILED STATISTICAL ANALYSIS
In this section, provide all numerical values of original and approximated metrics and also give detailed statistical analysis for each.
Metric
Formula
Meaning//nteen interpretatrnnnnonnn
Mean absolute error (MAE)
MAE
Zz
Average magnitude of the errors (un- signed). Provides a simple, easy-to- interpret measure of typical error size.
2=1
B
MAE mean)
(bootstrap
MAE
MAE
Average of MAE computed across bootstrap resamples; indicates stability of the MAE estimate under resampling. 2.5th and 97.5th percentiles of the boot- strap MAE distribution; 95% interval expressing sampling uncertainty.
B
b=
MAE 95% CI (boot- strap)
MAE2.5%
MAE97.5%
Root mean squared (RMSE)
RMSE
U xc
Similr to the MAE but squares errors first, it penalizes larger errors more strongly (sensitive to outliers).
Median absolute er-
MedAbs = median (19
The median absolute errors; a robust measure of "typical" error that is less sensitive to outliers.
æ=l)
1 71
Bland ManddAAlmman mean difference (bias)
d
di di yE
Mean signed difference between pre- diction and truth.
i=1
Blamd-Altman SD differences
$d =
(d: - d)2
Sample standard deviation of the differ- ences; this simply quantifies the vari- ability the errors.
11
Limits of agreement (LoA)
LoA d d 1.96 sd
Approximate interval containing ~95% of individual differences (under approximate normality of differences). Useful to see practical worst-case error bounds.
Relative to mean Rel(M) (original) (%)
100 X Expresses metric M (on the original scale) as a percentage of the mean of the original signal. This will help give n intuitive context for magnitude.
M Porig
m
Fori
m
Table 5: Definitions and their interpretations. Here denotes the truc/original value, the approximated value, n the number of pairs, bootstrap resamples, and the number of original observations whose mean used for scaling.
25

---

Published as a conference paper at ICLR 2026
A.9
STATISTICAL ANALYSIS OF DERADIFF'S PERFORMANCE ON CLIP
A.9.1 SDXL
Approiimened CLT Hears Fow Andher Cokura Tegert with Detta Li1
Dela S1 Setivenn Approdrrated CL Hestn Actual CL Heen
2373
CENON
ALIS
1809
020
Figure 12: Approximated CLIP Means with all
Figure 13: Delta (A) Between Approximated
(Row Anchor 3 - Column Target B) with Delta (A)
CLIP Mean and Actual PickScore Mean
Table Summary error metries for DeRaDiff approximations for CLIP scores when E [0,
Metric
Value
Relative to mean(original) (%)
Mean absolute error (MAF)
0.001 604
0.4300000
MAE (hootstrap mean)
0.001 608
0.431 000 %
MAE 95% CI (bootstrap)
0.001 038
0.002 226
0.278 000
0.596
Root mean squared error (RMSE)
0.001 994
0.534 000
Median absolute error
0.001 772
0.475 000
Bland-Altmal mcan difference (mean of x)
-0.001 018
-0.273 000 %
Bland- Altman SD of differences
0.001 775
0.475 000
Limits of agreement (mean 1.96 SD)
-0.004 496
0.002 461
-1.204 000
0.659 000
Notes: Value columns report absolute errors on the same scale as the original data. Relative column uses mean(original) 0.373 395. Limits agreement are computed as mean difference x
Table Summary emor metrics for DeRaDiff approximations for CLIP scores when
Metric
Value
Relative mcan(original) (%)
Mean absolute error (MAE)
0.005 014
1.342 690 %
MAE (bootstrap mean)
0.005 069
1.357 464
MAE 95% CI (bootstrap)
0.002 202
0.008 347
0.589 843
2.235
Root mean squared error (RMSE)
0.007 937
2.125 601
Median absolute error
0.001 593
0.426 633
Bland-Altman mean difference (mean of x)
-0.003 733
-0.999 878 %
Blama-Altman SD of differences
0.007 250
1.941 581
Limits agreement (mean 1.96 SD)
-0.017 943 07010476
-4.805 378
2.805 622
Notes: Value columns report absolute errors on the same scale as the original data. Relative column uses mean(original) = 0.373 395. Limits of agreement are computed as mean difference x1.
26

---

Published as a conference paper at ICLR 2026
A.9.2 SD1.5
Appawirarrdd C1 Hears Archor Cakere larpert Datto S01
Dnta 1d Retwess Aeproe rated Vean ArTAR
0.0
ALUS
-10
10:00
Figure 14: Approximated CLIP Means with all
Figure 15: Delta (A) Between Approximated
(Row Anchor 3 - Column Target 8) with Delta (A)
CLIP Mean and Actual CLIP Mean
Table 8: Summary metrics for DeRaDiff approximations for CLIP scores when E [0,
Metric
Value
Relative to mean(original) (%)
Mean absolute error (MAE)
0.001 557
0.447 532
MAE (bootstrap mean)
0.001 553
0,446 387 %
MAE 95% CI (bootstrap)
0.000 934
0,002 268
0.268 478
0,651 936
Root mean squared error (RMSE)
0.002 070
0.594 985
Median absolute error
0.001 137
0.326 711
Bland-Altman mcan difference (mean of y x)
-0.000 399
-0.1 14 728
Bland-Altman SD of differences
0.002 103
0.604310 %
Limits of agreement (mean 1.96 SD)
-0.004 520 0.003 722
-1.299 176 1.069 720
Notes: Value columns report absolute
the scale the original data. Relative column mean(original) 0.347 938. Limits agreement are computed mean difference SD.
Table Summary emor metrics for DeRaDiff approximations for CLIP scores when
Metric
Value
Relative mcan(original) (%)
Mean absolute error (MAE)
0.007 215
2,073 730 %
MAE (bootstrap mean)
0.007 302
2.098 770
MAE 95% CI (bootstrap)
0.002 903
0.013 009
0.834311
3,738 885
Root mean squared error (RMSE)
0.012 594
3.619 573
Median absolute error
0.002 391
0.687 314
Bland-Altman mean difference (mean of x)
-0.006 824
-1.961 343
Blama-Altman SD of differences
0.010956
3.148 884
Limits agreement (mean 1.96 SD)
-0.028 298
0.014650
-8.133 156 4.210469
Notes: Value columns report absolute errors on the same scale as the original data. Relative column uses mean(original) = 0.347 938. Limits of agreement are computed as mean difference x1.
27

---

Published as a conference paper at ICLR 2026
A.10
STATISTICAL ANALYSIS OF DERADIFF'S PERFORMANCE HPS
A.10.1 SDXL
Approdmoted 1155 Mears thox Ancher Coluen Targer Whth Dettre (i)
Cefe do Tetvewn Appnoorrend IPS Nesn Acsual IPS Mean
8000
DUTE
020
Figure 16: Approximated HPS Means with all
Figure 17: Delta (A) Between Approximated
(Row Anchor 3 - Column Target 3) with Delta (A)
HPS Mean and Actual HPS Mean
Table 10: Summary error metries for DeRaDiff approximations for HPS scores when 0,
Metric
Value
Relative to mean(original) (%)
Mean absolute error (MAF)
0.000 770
0.264 808
MAE (bootstrap mean)
0.000 770
0.265 098 %
MAE 95% CI (bootstrap)
0.000 501
0.001 053
0.172547
0.362 -.3352
Root mean squared error (RMSE)
0.000 949
0.326
Median absolute crror
0.000 680
0.233 938
Bland-Altma mcan difference (mean of x)
-0.000 640
-0.220 152 %
Bland Altman SD of differences
0.000 726
0.249 753 %
Limits of agreement (mean 1.96 SD)
-0.002 063 0.000 783
-0.709 668
0.269 364
Notes: Value columns report absolute the scale the original data. Relative column mean(original) 0.290 636. Limits agreement are computed as mean difference +: x
Table Summary error metrics DeRaDiff approximations HPS scores when
Metric
Value
Relative mcan(original) (%)
Mean absolute error (MAE)
0.004 041
1.390437 %
MAE (bootstrap mean)
0.00407 079
1.403 635
MAE 95% CI (bootstrap)
0.001 746
0.006 886
0.600 884
2.369 320
Root mean squared error (RMSE)
0.006 582
2.264
Median absolute error
0.001 539
0.529 484 %
Bland-Altman mean difference (mean of x)
-0.003 481
-1.197 693
Blama-Altman SD of differences
0.005 782
1.989 560
Limits agreement (mean 1.96 SD)
-0.014 80044814- 0.007 853
-5.097 232 2.701
Notes: Value columns report absolute errors on the same scale as the original data. Relative column uses mean(original) = 0.290 636. Limits of agreement are computed as mean difference 11.
28

---

Published as a conference paper at ICLR 2026
A.10.2 SD1.5
Agerodineted Measo Ancher Caaam larpet With Seeta
Deba Fetween Ap Franed Noon Actual
9374
Jone
5000
50000
12:00
2001
1940
Figure 18: Approximated HPS Means with all
Figure 19: Delta (A) Between Approximated
(Row Anchor 3 - Column Target 8) with Delta (A)
HPS Mean and Actual CLIP Mean
Table 12: Summary error metrics for DeRaDiff approximations for HPP when [0,
Metric
Value
Relative to mean(original) (%)
Mean absolute error (MAE)
0.001 175
0.425 179
MAE (bootstrap mean)
0.001 172
0.424 %
MAR 95% CT (bootstrap)
0.000 654
0.001 781
0.236 744
0.644 486 %
Root mean squared error (RMSE)
0.001 625
0.587 958
Median absolute error
0.000 744
0.269 095
Bland-Altman mean difference (mean of y - x)
-0.001 175
-0.425 179
Bland-Altman SD of differences
0.001 161
0.420 353 %
Limits of agreement (mean 1.96 SD)
00.003
0.001 102
-1.249
0.398 713
Notes: Value columns report absolute
the scale the original data. Relative column mcan(original) 0.276 300. Limits of agreement are computed mean difference SD.
Table Summary error metrics DeRaDiff approximations HPS scores when
Metric
Value
Relative mcan(original) (%)
Mean absolute error (MAE)
0.002 386
0.863 447 %
MAE (bootstrap mean)
0.002 415
0.873 887
MAE 95% CI (bootstrap)
0.000 986
0.004 310
0.356 733
1,560
Root mean squared error (RMSE)
0.004 179
1.512
Median absolute error
0.000 824
0.298 315
Bland-Altman mean difference (mean of x)
-0.001 569
-0.567 982 %
Blama-Altman SD of differences
0.004 009
1.450 790
Limits agreement (mean 1.96 SD)
-0.009 426
0.006 287
-3.411 531 2.275
Notes: Value columns report absolute errors on the same scale as the original data. Relative column uses mean(original) = 0.276 300. Limits of agreement are computed as mean difference xD.
29

---

Published as a conference paper at ICLR 2026
A.11
STATISTICAL ANALYSIS OF DERADIFF'S PERFORMANCE PICKSCORE
A.11.1 SDXL
aasnommeeee SolSor Nears thow Archor- Colure Targett tn Dertre Ib!
Data do Zetaeen Approd FroSCare Newn and Acsual ProGeon Mous
3/10
15003
020
Figure 20: Approximated PickScore Means with Figure 21: Delta (A) Between Approximated all (Row Anchor B + Column Target () with PickScore Mean and Actual PickScore Mean Delta (A)
Table 14: Summary error metries for DeRaDiff approximations for PickScore scores when À [0, 1]
Metric
Value
Relative to mean(original) (%)
Mean absolute error (MAF)
0.000 355
0.153 864
MAE (bootstrap mean)
0,000 353
0.153 119 %
MAE 95% CI (bootstrap)
0.000 238
0.000 498
0.103 446
0,216
Root mean squared error (RMSE)
0.000 441
0.191 478
Median absolute crror
0.000 283
0.122 888
ManndAAlman mcan difference (mean of x)
0.000
0.027 498 %
Bland- Altman SD of differences
0.000455
0,196 144 %
Limits of agreement (mean 1.96 SD)
-0.000 823
-..0000000 9.000
-0.356 944
0.411 939
Notes: Value columns report absolute the scale the original data. Relative column mean(original) 0.230 509. Limits agreement are computed as mean difference SD.
Table 15: Summary emor metrics for DeRaDiff approximations for PickScore scores when À
Metric
Value
Relative mcan(original) (%)
Mean absolute error (MAE)
0.004 903
2.126 836 %
MAE (bootstrap mean)
0.004955 939
507
MAE 95% CI (bootstrap)
0.002 544
0.007 666
1.103 610
3.325
Root mean squared error (RMSE)
0.007 102
3.080
Median absolute error
0.003 058
1.326.521
Bland-Altman mean difference (mean of x)
-0.004 644
2.014716 %
BlamanAltman SD of differences
0.005 562
2.412729
Limits agreement (mean 1.96 SD)
-0.015 545
0.006 257
-6.743 664
2.714232
Notes: Value columns report absolute errors on the same scale as the original data. Relative column uses mean(original) = 0.373 395. Limits of agreement are computed as mean difference 11.
30

---

Published as a conference paper at ICLR 2026
Rodsowe Noura Anchor Coture Tegett A.11.2 SD1.5
tetwoo Approomaned hoddore Achuol
8,000
-5.03
0238
50
2800
200
200
10:4
3000
5008
LO40
Figure 22: Approximated PickScore Means with Figure 23: Delta (A) Between Approximated all (Row Anchor 3 Column Target 3) with Delta (A)
PickScore Mean and Actual PickScore Mcan
Table 16: Summary error metrics for DeRaDiff approximations for PickScore scores when À [0,
Metric
Value
Relative to mcan(original) (%)
Mean absolute error (MAE)
0.000 718
0.331
MAE (bootstrap mean)
0.000 717
0.331 069
MAE 95% CI (bootstrap)
0.000 336
0.001 163
0.155 070
0.537 347 %
Root mean squared error (RMSE)
0.001 097
0,506 768
Median absolute
0.000
0.153 080
Bland-Altman mean difference (mean y x))
-0.000 709
-0.327
Bland-Altman SD of differences
0.000 867
0.400 259
Limits of agreement (mean + 1.96 SD)
-0.002 408
0.000
-1.112055
0.456 961 %
Notes: Value columns report absolute errors on the same scale as the original data. Relative column uses mean(original)) 0.216 494. Limits of agreement computed
difference 1SD
Table 17: Summary error metrics for DeRaDiff approximations for PickScore scores when À
Metric
Value
Relative to mean(original) (%
Mean absolute error (MAF)
0.002 255
1.041
MAE (bootstrap mean)
0.002 281
1.053 707 %
MAE 95% CI (bootstrap)
0.000
0.003 959
0.449 230
1.828 517 %
Root squared error (RMSE)
0.003 786
1.748 630
Median absolute error
0.000 787
0.363
Bland Altman mean difference (mean y x))
-0.001 573
-0.726613
Bland-Alrman SD of differences
0.003 564
1.646 340 %
Limits of agrecment (mean + 1.96 SD)
-0.008 559
0.005 413
-3.953 440
2.500 213
Notes: Value columns report absolute
the scale as the original data. Relative column uses
mean(original) 494. Limits of agreement computed mean difference 1D.
A.12 LLM USAGE
This research idea was conceived solely and only by the authors by identifying gaps in the existing research literature. LLMs were NOT used for any research ideation. LLMs were only used polish writing, help in plotting graphs, retrieve known mathematical facts and fix any grammatical emrors that the authors missed.
31


Reviews:
[{"review_id": "3YggVIb43a", "reviewer": "Reviewer_CgVt", "summary": "The paper proposes a novel method called DeRaDiff, which performs per-step denoising realignment. Aligning models to human preferences from scratch is often very time-consuming and computationally expensive. DeRaDiff, on the other hand performs the alignment on the fly during inference by modulating the alignment strength by a parameter $\\beta$. The experimental evaluation shows that the method has comparable results to models aligned from scratch.", "questions": "Q1: Did you observe whether biases are propagated from the reference model to the output of DeRaDiff?  \nQ2: In the experiments the reward-hacked model had a $\\beta$ of 500. Did you test it with even more reward-hacked models that have even a lower $\\beta$?  \nQ3: Does the method still work if the reference model and the aligned model have different architectures (e.g. DiT and U-Net models)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper shows that DeRaDiff has a closed-form solution.\n- The training-free inference-time alignment method saves computational costs\n- The method is able to undo reward hacking, eliminating the need for realignment", "weaknesses": "- When evaluating human alignment a real-world human study would have been nice\n- Since the pre-trained reference model is mixed with the aligned model, this could lead to biases reappearing in the output of DeRaDiff.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HRWmMQnTKz", "reviewer": "Reviewer_ziCu", "summary": "The author presents a classifier‑free guidance–like formulation designed for reward alignment in diffusion models (DDPM).\nThe experimental results support the formulation's validation.", "questions": "1. It seems odd to see the thermodynamic variable $\\beta$ placed in the denominator, since $\\beta$ \nis usually regarded as the inverse temperature $\\tau = 1/\\beta$.\n2. I was confused in Line 145 that you cited Song et al. (2020) without any SDE formulation in your paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper makes a worthwhile contribution with a clear presentation and credible theoretical \nsupport.\n\n## Presentation: ~95th percentile\n\nThis paper presents coherence, and most of the idea is clearly addressed. I would like to thank you for saving me a lot of time reviewing your work.\n\n## Soundness: ~75th percentile\n\nTheorem 1 underpins the soundness of the paper. Although I have not examined every minute detail, the derivation appears to be correct.\n\n## Contribution: 40th~70th percentile\nThis method seems novel to me, although I’m not sure if something similar already exists in the literature. It builds a scalable way to tune between the anchor model and the aligned model.\n\n## Note\nI hope the AC is aware that the rating is calibrated using percentiles to reduce evaluation noise effectively.", "weaknesses": "## Soundness\n\nI would have preferred to see additional comparisons between your method and other approaches applied to similar problems. Nonetheless, the absence of such comparisons does not undermine the validity of your \nclaim.\n\n## Presentation\n\n1. Presenting the denominator in Equations (3)–(6) as a partition function has both advantages and \ndisadvantages. While it clarifies the interpretation, the repeated form of the \nequations feels redundant. If the repetition is intentional, please justify it explicitly; otherwise, \nconsider consolidating the expressions to avoid unnecessary duplication.\n2. Algorithm 1 appears to be a verbatim transcription of your Python implementation. For readers who \nare more comfortable with Python than with pseudocode, it would be clearer to relocate the algorithm to the appendix and present the actual Python source code there. This approach preserves the practical \nrelevance of the code while keeping the main manuscript concise.\n\n## Contribution\nRegarding a [similar work](https://arxiv.org/abs/2505.18547) working on score-based SDE in the existing literature, it would be helpful to acknowledge it and clarify how your paper differs.\n\nIt is known that DDPM, DDIM, and any score‑based SDE can be reformulated as Karra’s SDE [1] in a \nbidirectional manner. Consequently, these paradigms are theoretically equivalent, although the conversion \nis not trivial. Thus, even if your work is considered concurrent or subsequent, there remains room for a \nmeaningful contribution. Besides, many contemporary studies were developed without awareness of this \nequivalence.\n\n[1] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" Advances in neural information processing systems 35 (2022): 26565-26577.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G3lliKWD8Q", "reviewer": "Reviewer_Jger", "summary": "The authors propose an inference-time realignment of diffusion models, to be able to emulate models trained under other regularization scenarios, without extra training or fine-tuning. Inspired by DeRa (from the realm of LLMs), the authors derive a closed-form approximate posterior sampling. The authors claim and briefly discuss the reduced computational overhead.", "questions": "- Fig 6 (b) is hard to read and interpret. Can't this be done differently? or at least elaborated better?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is reasonably well-written with a coherent narrative. \n- The idea of extending DeRa (from LLMs) to diffusion models is an interesting angle. \n- The results are rather promising, and align with the core claims.", "weaknesses": "- I believe only Fig. 7 for comparison across base, aligned and realigned models is too little, also not discussed in necessary level of detail. In my eyes this should be established with more qualitative results and further elaboration across the images and models.  \n- The paper would benefit from a thorough proof-read. Few typos, and styling inconsistencies can be seem across the document. \n- Maybe (pareto-front) reward vs divergence plots can help establish the core message from a different angle, that just looking CLIP or HPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5f5yZITqGl", "reviewer": "Reviewer_kfW9", "summary": "DeRaDiff introduces a denoising-time realignment procedure for diffusion models that allows practitioners to modulate KL regularization strength on the fly during sampling without any retraining or fine-tuning, by interpolating between a pretrained reference model and a single aligned anchor model through a closed-form Gaussian mixture applied at each denoising step, controlled by a scalar parameter lambda between 0 and 1. The method extends decoding-time realignment from language models to continuous latent diffusion processes, providing an efficient alternative to the expensive practice of training separate models for each regularization strength. Experiments on SDXL and Stable Diffusion 1.5 show that DeRaDiff accurately approximates models aligned from scratch across a wide range of regularization strengths, with mean absolute errors below 0.5% on metrics like PickScore, HPS v2, and CLIP, while reducing computational costs by up to 90% when exploring multiple regularization strengths. Additionally, DeRaDiff can undo reward hacking artifacts by increasing lambda to simulate stronger regularization, offering a practical and scalable solution for hyperparameter exploration in alignment of text-to-image diffusion models.", "questions": "The ReAlignment problem is a notion that has only recently surfaced in the LLM literature.  \nAs the first work that imports this term into the generative-Art domain, the current manuscript cannot assume that most readers are already familiar with it.  \n\nTo discover what “ReAlignment” actually means, one is forced to consult *Decoding-time ReAlignment of Language Models*.  \nTherefore, I believe the paper should be revised to:  \n\n- Give a concise, self-contained definition of ReAlignment up front,  \n- Explain how the generative-Art setting changes the problem.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "While the idea of decoding-time realignment was recently introduced for language models, the paper non-trivially lifts it to the continuous, iterative denoising process of diffusion: it derives a closed-form Gaussian mixture per timestep, handles scheduler-specific posteriors, and exposes a single scalar λ that maps to any effective KL strength β/λ. This extension is not straightforward—marginalizing over latent trajectories is intractable—so the authors provide a principled step-wise approximation backed by a new theoretical result. DeRaDiff is therefore the first inference-only knob for alignment strength in diffusion, removing the need for expensive sweeps.\n\nAlignment cost is a practical bottleneck for large diffusion models; DeRaDiff removes it by turning a multi-training sweep into a single pass plus sampling-time tuning. This lowers the barrier for researchers and practitioners to explore fine-grained alignment, potentially accelerating RLHF-style workflows, multi-reward composition, and personalised generation. The technique is architecture-agnostic (any scheduler admitting Gaussian posteriors) and complementary to existing alignment objectives (DPO, DDPO, etc.). By demonstrating that careful posterior interpolation can mimic full retraining, the work also hints at broader implications for efficient model merging and test-time adaptation beyond vision.\n\n\nThe submission is technically sound. The derivation (appendix) carefully justifies the Gaussian form, states regularity conditions (λ ∈ [0,1], positive variances), and warns against extrapolation (λ > 1). Empirical coverage is unusually broad: two model families (SDXL, SD-1.5), three human-centric metrics (PickScore, HPS v2, CLIP), 500 prompts spanning two datasets, and six regularization strengths. Error magnitudes are small (<0.5 % of metric means), confidence intervals are supplied, and ablations show stability inside the convex regime. Compute savings are measured in actual GPU-hours and EFLOPs rather than vague “speed-ups”. Minor gaps—no user study on λ control, limited diversity in prompts—do not weaken the overall thoroughness.", "weaknesses": "The paper trains anchors only at β ∈ {500, 1000, 2000, 5000, 8000, 10 000}.  \n   - Fig. 5 shows that PickScore saturates after β ≈ 2000; the interesting “knee” region where human appeal rises fastest (β ≈ 100–1500) is sampled very coarsely.  \n   - Because DeRaDiff can **interpolate** (λ < 1) but can only **weakly extrapolate** (λ > 1) before instability, a user who wants to explore β < 500 cannot do so with the supplied β = 500 anchor.  \n   \n\n\n   The authors never run a β-sweep **guided** by DeRaDiff. The experiment pipeline (§5.1) still assumes the user already knows which β-values to test.  \n   \n\n   Fig. 3 and Table 6 show visible degradation once λ ≳ 2.5. Many users will nevertheless try λ > 1 to obtain stronger alignment; at present the paper offers no guard-rail except a verbal warning.  \n   \n\n   PickScore, HPS v2 and CLIP give nearly identical curves (Fig. 5). The experiment therefore does **not** show that DeRaDiff can trade off **aesthetics vs. prompt fidelity**, a key reason one tunes β in practice.  \n   \n\n   All results use Euler-A, 50 steps. Many deployment pipelines use DPM-Solver-12 or 20 steps for speed. The closed-form update assumes the scheduler is **linear in the Gaussian sufficient statistics**; this is only approximately true for some solvers.  \n  \n   A simpler baseline is to take the aligned β = 500 and β = 2000 checkpoints and linearly interpolate their **weights** (θ-ref + λ(θ-2000 − θ-ref)). This costs zero extra inference time and is already used in “model soups” work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)