Neon: Negative Extrapolation From Self-Training Improves Image Generation
Authors: Sina Alemohammad
Zhangyang Wang
Richard Baraniuk
Venue: ICLR 2026 Oral (2026)
Keywords: Generative Models
Self-Improvement
Weight Merging
Image Generation
Primary area: generative models
TL;DR: Instead of simply fine-tuning a generative model on its own synthetic outputs, briefly fine-tune it to find the direction of model collapse, then apply the reverse of that update to the original model for a major performance boost.

Abstract: Scaling generative AI models is bottlenecked by the scarcity of high-quality training data. The ease of synthesizing from a generative model suggests using (unverified) synthetic data to augment a limited corpus of real data for the purpose of fine-tuning in the hope of improving performance. Unfortunately, however, the resulting positive feedback loop leads to model autophagy disorder (MAD, aka model collapse) that results in a rapid degradation in sample quality and/or diversity. In this paper, we introduce Neon (for Negative Extrapolation frOm self-traiNing), a new learning method that turns the degradation from self-training into a powerful signal for self-improvement. Given a base model, Neon first fine-tunes it on its own self-synthesized data but then, counterintuitively, reverses its gradient updates to extrapolate away from the degraded weights.  We prove that Neon works because typical inference samplers that favor high-probability regions create a predictable anti-alignment between the synthetic and real data population gradients, which negative extrapolation corrects to better align the model with the true data distribution. Neon is remarkably easy to implement via a simple post-hoc merge that requires no new real data, works effectively with as few as 1k synthetic samples, and typically uses less than 1\% additional training compute.  We demonstrate Neon’s universality across a range of architectures (diffusion, flow matching, autoregressive, and inductive moment matching models) and datasets (ImageNet, CIFAR-10, and FFHQ). In particular, on ImageNet 256x256, Neon elevates the xAR-L model to a new state-of-the-art FID of 1.02 with only 0.36\% additional training compute.

Published as a conference paper at ICLR 2026
NEON: NEGATIVE EXTRAPOLATION FROM SELF-TRAINING IMPROVES IMAGE GENERATION
Sina Alemohammad', Zhangyang Wang', Richard G. Baraniuk
ECE Department, The University Texas at Austin; *ECE Department, Rice University;
ABSTRACT
Scaling generative Al models is bottlenecked by the scarcity of high-quality training data. The ease of synthesizing from generative model suggests using (unveri- fied) synthetic data to augment a limited corpus of real data for the purpose of fine-tuning in the hope improving performance. Unfortunately, however, the resulting positive feedback loop leads to model autophagy disorder (MAD, aka model collapse) that results in a rapid degradation in sample quality and/or diversity. In this paper, introduce Neon (for Negative Extrapolation frOm self-traiNing), a new learning method that turns the degradation from self-training into a powerful signal for seff-improvement. Given base model, Neon first fine-tunes it on its own self-syyteesized data but then, counterintuitively. reverses its gradient updates to extrapolate away from the degraded weights. We prove that Neon works because typical inference samplers that favor high-probability regions create a predictable anti-alignment between the synthetic and real data population gradients, which neg- ative extrapolation corrects to better align the model with the true data distribution. Neon remarkably easy to implement via simple post-hoc merge that requires no real data, works effectively with as few as Ik synthetic samples, and typically uses less than 1% additional training compute. We demonstrate Neon's universality across range of architectures (diffusion, flow matching, autoregressive, and induc- tive moment matching models) and datasets (ImageNet, CIFAR-10, and FFHQ). In partieular, ImageNet 256x256, Neon elevates the xAR-L model to state-of-the-an FID of 1.02 with only 0.36% additional training compute. Code is available ht htsps: / githhb...b....ggggg con/VITA VITAA Group/Neon
XARL
xAR-L + Neon
Figure 1: Good to great: Neon's state-of-the-art performance on ImageNet-256. Ncon elevates powerful baseline generative model (xAR-I., top row) to new level of sharpness and realism (bottom row) with simple post-hoc merge. This leap quality. improving the Fréchet inception distance (FID) from 1.28 to breadd-breaking 1.02, is accomplished with only 0.36% extra training compute.
Corresponding author: anm.alllmoaamma@@@essiiiueeeee

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Modern generative models for images have achieved remarkable photorealism through continuous advances in architectures, training methods, and scale. Diffusion models (Ho et al., 2020; Song et al., 2021), flow matching approaches (Lipman et al., 2023; Liu et al., 2023), autoregressive architectures (Ding al., 2021; Yu et al., 2022b), and few-step generators (Song al., 2023; Zhou et al., 2025a) now form the backbone of large-scale image generation systems. Despite these advances, the most reliable path to state-of of-the-art performance remains scaling: ever more parameters, ever larger datasets, and ever increasing compute (Henighan et al., 2020; Kaplan et al., 2020).
Important energy sustainability issues aside, this scaling paradigm faces a fundamental bottleneck: high-quality training data. Curating diverse, rights-cleared image datasets is expensive and time- consuming, with diminishing returns as existing sources are exhausted (Muennighoff et al., 2023; Villalobos et al., 2022). As the gap between model capacity and available training data widens, the field must explore alternative paths to model improvement that do not rely on ever-larger real datasets.
The ease of synthesizing data from generative models has inspired range of model improvement approaches to augment a limited real data set. At the simplistic end, one can fine-tune a model on its own generated outputs. However, such naive self-training has been shown to lead to "model autophagy disorder" (MAD) (Alemohammad et al., 2024a) or model collapse (Shumailov et al., 2024), where diversity and/or quality degrades. At the complicated end, researchers have avoided collapse through sophisticated workarounds like external verifiers for synthetic data quality (Feng et al., 2024), auxiliary discriminator networks (Kim al., 2023), negative guidance during inference (Alemoham- mad et al., 2024b), and likelihood-basee While effective, these approaches add sigmificant computational overhead, are restricted specific architectures, or require complex iterative training.
discrimination between distributions (Zheng et al., 2025).
Neon. In this paper, we show that there hidden promise in directly fine-tuning a model on its own generated data. Our key insight is that the degradation due to self-training is not random noise but rather power signal that is anti-aligned with the real-data population gradient. Neon (Negative Extrapolation from self-traiNing) exploits this anti-alignment through simple parameter merge. Given a base model with parameters O, trained on real data, we first apply the naïve self-training approach: generate synthetic samples and briefly fine-tune obtain the parameters 0, that exhibit degraded performance. Then, rather than using 9, directly, we perform negative extrapolation:
N = 0, w 0,) (1 w)0, w0,,
0,
(1)
where w controls the extrapolation strength. The vector 82 - 6, corresponds to the synthetic gradient direction; because this direction is anti-aligned with the (infinite real data) population gradient, reversing it reduces the true data risk and redistributes probability mass to under-represented modes.
Contributions. [C1] We introduce Neon, a deceptively simple post-processing method that improves generative models by reversing their degradation on self-generated data (Section 3). In contrast to existing methods for synthetic data augmentation, Neon requires no additional real training data, no access to the original training data, no auxiliary models, no likelihood computation, and no inference modifications. [C2] We prove rigorously that mode-seekine mode-seeking inference samplers create predictable anti unti-alignment between the synthetic and population gradients that guarantees the effectiveness of negative extrapolation (Section 3.1). [C3] We demonstrate Neon's universality across diffusion, flow matching (Section 4.1,, autoregressive (Section 4.2), and few-step (Section 4.3) models on CIFAR-I0. 10, FFHQ. and ImageNet with < 1% additional compute and as few as Ik synthetic samples. For example, on ImageNet-256, Neon elevates xAR-L from an FID of 1.28 to the state-of-the- thrtr-aatt 1.02 using only 0.36% additional compute. [C4] We show that Neon's 's improvement mechanism operates through precision-reeal trade-off that redistributes probability mass from over- to under-represented modes (Section 4. 1). [C5] We demonstrate that the Neon degradation signal is transferable, which enables synthetic data from one model architecture to improve another (Section 4.4).
BACKGROUND
Notation and definitions. Let D be training data set drawn from Pdat- A training algorithm produces the generative model Gg, whose output is a score, velocity, or logit depending on the model family. The training budget B is the cumulative number of images seen (in millions):
2

---

Published as a conference paper at ICLR 2026
B = (global steps) x (global batch size). An inference routine I with hyperparameters' induces sampling distribution 90.x- Denote the idealized distribution without inference time modifications (e.g., guidance) by Po -0 .. We use dist(::-) for generic divergence, - for set cardinality, and the shorthand
|| == /212, (x. == x My.
|Allop.M = || A01//AM-111 1/2
for any rositive-definit matrix M, where || 1l.. (,, and || Ilap are the standard Euclidean norm, inner product, and operator norm. Let k denote 10a.
Visual generative models. Many image generators trace path from noise data via an affine inter- polation 2, = (t) + at)e for t E [0. 1]. with 2o Pdata <~ (0, I), and boundary conditions (0) = (0) 0, (11) O, ((1) L, inducing po Pdata and p1 NN I) (Lipman al., 2023; Song et al., 2021).
Diffusion models (Ho et al., 2020; Song et al., 2021) train Gz(z,t) t) to approximate the score V, lo (x) (or equivalently, predict noise). At inference, the learned score drives the reverse-time SDE or probability-flon ODE.
Flow matching (Lipman et al., 2023; Tong et al., 2024) learns the conditional velocity v" e, e, a' (0) (1) e by regressing Go(zt, with squared error; sampling integrates di = Go( I) from /. == 1 to 0.
Few-step generators reduce sampling cost by collapsing many steps. Consistency models (Song et al., 2023) predict zo directly from (Ir, IMM (Zhou al.. 2025a) learns direct transitions J, - Go +) with moment-matching, enabling quality with T 1-8 steps.
Autoregressive models (Ren et al., 2025; Tian et al., 2024) factorize images into tokens yI:N Tax)) and model p(vI.N) II p()+()) M=(<s)), where Go (y outputs next-token logits trained via cross-entropy. The ordering and decoding choices (temperature, top-k) form part of inference hyperparameters k.
Self-training and collapse. When models iteratively train on their own synthetic outputs, they exhibit what has been termed MADness or model collapse: E] Po.)] grows over time (Alemohammad et al., 2024a; Dohmatob et al., 2024; Shumailov et al., 2024). Pure self-training diverges, while mixing real and synthetic data converges to degraded equilibria (Bertrand al., 2023; Gerstgrasser et al., 2024). While external signals beyond the training data can prevent collapse (Alemohammad et al., 2024b; Feng et al., 2024), these methods require additional resources such as verifiers or fresh data.
Related work on synthetic data training. Several recent methods successfully leverage synthetic data for model improvement, but require significant architectural constraints or computational over- head. Discriminator Guidance (Kim et al., 2023) trains a post-hoe discriminator on real versus generated samples across diffusion timesteps, using its gradients to correct the score function dur- ing sampling. While effective, it adds inference overhead and remains diffusion-specific SIMS (Alemohammad et al., 2024b) employs self-generated data negative guidance steer diffusion trajectories away from degraded manifolds, but similarly requires inference-time modifications and is limited to diffusion models. Direct Discriminative Optimization (DDO) (Zheng et al., 2025) refor- mulates likelihood based models as implicit discriminators via log-likelihood ratios between target and reference models, enabling strong improvements for diffusion (via ELBO) and autoregressive models, but fundamentally cannot apply to likelihood-free architectures like flow matching (Lippman et al., 2023) or inductive moment matching (Zhou et al., 2025a). Self-Play Fine-Tuning (Yuan et al., 2024) iteratively pits models against earlier checkpoints, surpassing RLHF methods on human preference benchmarks but requiring multiple training rounds and substantial computational overhead. In contrast to these methods, Neon requires auxiliary models, inference modifications, likelihood computations, and works across all architectures with a simple post-hoc parameter merge.
3 NEON: NEGATIVE EXTRAPOLATION FROM SELF-TRAINING
When models train on synthetic samples produced by their inference procedure I (what we call "self-training " they predictably degrade. Neon exploits this: by reversing the degradation direction, we can improve a model without additional real data. Starting from a base generator Go, (typically trained real data), we: (i) generate the synthetic dataset S once using test-time inference I(Go,; R))
3

---

Published as a conference paper at ICLR 2026
Figure 2: Neon's key idea: synthetic degradation and real- data improvement point in opposite directions. This toy 2D Gaussian example plots a heat map the log Wasserstein distance to the true data distribution Pam from the generative model Getw..".)) We that updating the model's parameters in the reverse the direction they would updated by fine- tuning on self-syathesized data (increasing w.) achieves similar improvements to fine-tume fine-tuning the base model with 4x more real data (increasing wo).
0,
R.
*
(ii) briefly (e.g., using < 1% of the original training budget) fine-tune the generator on S to obtain the degraded Gg,, and (iii) negatively extrapolate via the parameter merge:
Neun == 0, w(O, - 0,) = (1 + w)0, - -ww,,
(2)
where w > controls the extrapolation strength. Algorithm provides the full details.
Algorithm 1 Neon: Negative Extrapolation from Self-Training Training
Require: Base model Go.,, inference routine with hyperparameters &. Hyperparameters: Synthetic dataset size n, = [S], extrapolation strength training budget B
1: S + )= where T; ~ q..... induced by I(Ga,; 2: Go, FineTune(Ga,,,5
P sample using test-time inference briefly fine-tune on synthetic data > reverse the degradation
3: 8Nen < (1 + w) - ww8
Output: Final generator Gess.
3.1 WIYY NEON WORKS
Geometric intuition via a toy study. To visualize why negative extrapolation from degradation succeeds, consider 2D Gaussian example where Pdata =N(uue Σmue). We train base model Ge. on 1k real samples and then define two directions in paramcter space: the degradation direction from fine-tuning the base model 10 synthetic samples from Go, obtain Go.- and oracle improvement direction from fine-tuning on 5k real samples (the original 1 real data points plus 4k new ones) obtain Gg,. We evaluate models in the 2D span of these directions:
0 (w,, Wo) 0r + W
0, 0.)
wo
(0. 0r)
degradation direction (Neon)
improvement
where wA controls the amount of negative extrapolation (Neon) and w. adds real-data improvement (oracle baseline). Figure 2 visualizes our key finding: moving backwards along the Neon direction alone (w, = 0) yields substantial improvement, indicating that the opposite of degradation direction and additional real-data improvement direction both point towards a better approximation the true data distribution.
Theoretical analysis. We now formalize the intuition provided by the toy example. We prove that typical inference samplers cause the synthetic and real data gradients to point in opposite directions, enabling negative extrapolation to reduce the true data risk.
Set-up. Let lo (x) be differentiable loss function and (0ata(() :- 1 the corresponding risk. Let 0* E arg minş (f) and write 0,- 0" + with ||=| =ET HaE. Let ge... denote the fixed sampler constructed once at 0,. Define
(1(2)= = Vola(z),
Ha = V2Raataa (0*) = [On00 A=9-
Royn(0 (0) = (
= VoRdata (0) (0)
r=:= (o)
Let P > 0 be a preconditioner and set K == H1 PH H with mn 3 3 MI.
We say the synthetic and real data gradients are anti-aligned 0, if their preconditioned inner product is negative
(ra. Prs)
4

---

Published as a conference paper at ICLR 2026
Neon improves under anti-alignment. Short synthetic fine- fine-tuning yields 9. 0, - Pr, +a(a))
which Neon reverses: NN = 8, + wa Pr. + O(wa²). A Taylor expansion of the risk yields
datata (fNoon) = Raaaa (8,) + wa s (wa)²
T. P.
T Rdatta Pr, )(wa)))
(4)
2
When 8 < O, the negative linear term dominates for small w > 0, ensuring that Retata (ONcon) < R (0,). When Rettta is locally convex at 0, (i.e., R (0,) = 0), the optimal w" ss((a)) >
O, where 2 == == r. P RaRdaca Pr.. See Appendix B.4 for the proof.
Sampler-induced anti-alignment Let
b := Lm (X)], A := [ [X)]
E [o (X)];
Jo- Ja-(x) @poo(c))lr. (5)
and measure their sizes the Ha-geometry by
Yo = ||b||
mh = ا Hr
Define the angle between the model error and the sampler bias b in the H geommeeyy by
(c, Hz b)
cOs o || | blll,
E[-1,1]
(6)
Intuitively, cos ạ < 0 means that the sampler's bias points is in direction opposing the current emor, favoring anti-alignment.
Theorem 1 (Anti-alignment under inference mismatch). Let H/ "H2/ with spectral bounds
m/ 3 K 3 MT. Then the alignment (rd, Pr.) obeys
< (1 + m)) ||
||e|| cos +p+ O 1 (Hg).
Consequenily, a sufficient condition for 8 < 0 is that the leading two terms on the right-hand side be negative. In particular, for cos yp and sufficiently small ||e||
|E
1)0
(- (-cos
S<0.
M m)
See Appendices B.4-B.5 for the proof.
Mode-seeking samplers enable Neon. Many deployed inference routines can be written as monotone reweighting of the reference model,
((x) x S( log Po. (I)) po. (x), with f nondecreasing and not a.e. constant.
Such mode-seeking samplers emphasize high-density regions. The following theorem shows that this structure guarantees anti-alignment.
Theorem 2 (Mode-seeking samplers induce cos < 0). If the sampler is mode-seeking (monotone reweighting with f nondecreasing). then cos & < to first order in [| guaranteeing anti- alignment near good models.
See Appendices B.6-B.7 for the proof.
Concrete instances. Combining Theorems 1 and 2, we obtain that Neon reduces risk for the following standard configurations:
Autoregressive models: Temperature 1, top-k, and top-p sampling all produce nondecreas- ing reweighting of log ps, (App. B.6).
(ii) Diffusion and flow models: Finite-step ODE solvers (including classifier-free guidance) induce monotone terminal reweighting first order in step size (App. B.7).
I Local convexity is sufficient but necessary. The result holds under the weaker condition of directional smoothness along the step direction = Pr.. See Appendix B.4 details.
coupling (A-MONO): the conditional E [ to ) 2 Xo 2o] increases with log (#a). See App. B.7.
Assumes curvature-density
expectation
5

---

Published as a conference paper at ICLR 2026
EDM CCIARR 10
EDM-VPI FFHQ 64
Flow Matching CCFARR
2.5
16A
1.6
FTD
1.5
1.4
1.2
0.5
15 (Mi)
23 (Mi)
8 (Mi)
Figure 3: Neon consistently improves FID with minimal self-training overhead. Minimum FID (optimized over extrapolation strength w) vs. self-training budget B (millions of images seen during fine-tuning on S) for varying synthetic dataset sizes [S], on EDM-VP (CIFAR-I0/FFHQ-6)) and flow matching (CIFAR-10). Optimal gains use < 3Mi (< 2% base model training compute for EDM: < 3% for flow), confirming Neon's 's efficiency. At B = 0, FID reflects the base model (no Neon).
When interpolation (not extrapolation) helps. The theory also identifies complementary regime: diversity-seeking samplers (f nonincreasing) reverse the angle to cos Φ 2 O, making 8 > 0 and thus favoring interpolation (-1 < < 0) over Neon's negative extrapolation (w > 0). However, such samplers are rare in practice: autoregressive models would require high temperature T > 1 or anti-mode truncations; diffusion/flow models would need samplers that decrease contraction near modes, which is atypical. In these cases, the Neon formula Neon (1 + w)O, wd, still applies, but the optimal merge weight satisfies -1 < w < 0 (moving toward 0.) rather than w > 0 (moving away from 0,). See App. B.9 for formal analysis.
Finite S effects. Our analysis assumes that the population synthetic gradients rs (0,). but in practice we use finite S with brief fine-tuning from 0,. For checkpoint 0₂ after 7 steps with step size ax the sufficiently large while aT remains small, yielding stable, low-variance Neon directions despite limited [S]. This produces a U-shaped performance in |S] very small sets are variance-limited, very large sets amplify curvature effects (inflating the quadratic term in our Taylor expansion), while moderate sizes optimally balance these competing factors. See Appendix B. 10 for formal bounds and parameter selection guidance.
displacement dr- == (0, 6,) 8.)((aT) concentrates on -r. (6,) when
4 EXPERIMENTS
We evaluate Neon across four model families - diffusion (EDM (Karras et al., 2022)), flow match- ing (Tong et al., 2023; 2024), autoregressive (VAR (Tian al., 2024), xAR (Ren et al., 2025)), and few-step (IMM (Zhou et al., 2025a)) on ImageNet (Deng et al., 2009), CIFAR-10 (Krizhevsky & Hinton, 2009), and FFHQ (Karras et al., 2019).
For each model, starting from a public checkpoint Ga., we generate synthetic datasets S using the FID-optimal inference settings к from each paper. We fine-tune on S with the original training recipe at reduced learning rate (see Appendix C for details). We report FID as our primary metric using 10k/50k samples for hyperparameter search/final evaluation (Heusel et al., 2017), with Preci- sion/Recall (Kynkäänniemi et al., 2019) at k = 5 nearest neighbors. For comprehensive comparison of Neon against state-of-the-art generative models across all benchmarks, please see Table A.
4.1 DIFFUSION AND FLOW MATCHING MODELS
We evaluate Neon with the EDM-VE VP (Karras et al., 2022) (CIFAR-10 conditional, FFHQ-64 64 uncondi- tional) and flow matching (Tong et al., 2023; 2024) (CIFAR-10 10 unconditional) models using public checkpoints. The synthetic datasets S were generated with default inference settings.
Results. Figure 3 plots the FID vs. the fine-tuning budget B for various S). Neon achieves substantial gains with minimal overhead: Neon+EDM- VP trained on CIFAR-10 improves the FID from 1.78 to 1.38 using only 6k synthetic samples and 1.75% extra compute compared to training the base model. Neon+EDM-VV VP trained on FFHQ-64 improves the FID from 2.39 to 1.12 using only 18k samples and 0.85% additional compute. Neon+Flow matching CIFAR-10 improves the FID from 3.5 to 2.32 using only 25k samples and 3.2% additional compute. Neon's performance shows a non-monotonic relationship with the synthetic dataset size |8], with optimal performance in the range

---

Published as a conference paper at ICLR 2026
3
0.75
2.6
Precision
0.7
Recall
RD
1.5
1.00
Figure 4: Neon trades precision for recall, yielding net FID improvement. For the EDM-VP model trained on CIFAR- 10, we plot the FID, precision, and recall vs. negative extrapolation strength w for various training
budgets B. w -1 corresponds to the model directly trained synthetic data, ie.. Owean 6.. w 0
corresponds the base model. ie.. 0xoes 0,. 0 corresponds to the negative extrapolation regime where Neon demonstrates its improvement capability. In each case, |S]
xAR-B ImageNet-256
xAR-L ImageNet-256
VAR-d16 ImageNet-256
VAR-d30 ImagaNet-512
1.3
2.5
1.6
FIP
2.5
1.4
8 (Mi)
B
(Mi)
(MD)
Figure 5: Neon consistently improves autoregressive models across architectures and resolutions. We plot the minimum FID (optimized merge weight and CFG scale the fine-tuning budget B for various synthetic dataset sizes (S]. From left: xAR-B xAR-L on ImageNet-256 (with xAR-L achieving state-of-the-art 1.02 FID), VAR-d16 on ImageNet-256,, and VAR-d30 on ImageNet-512.
6k-25k samples. Smaller [S] require more precise w tuning but converge rapidly; larger [S] support a wider range of w's but slower convergence.
Figure dissects Neon's effect on EDM-VP trained on CIFAR- 10 using precision- metrics with |S = 6k. The FID vs. weight relationship (left panel) exhibits the unimodal shape predicted by our Taylor series analysis. As fine-tuning progresses, the optimal w* decreases, which consistent with w" =az)) where increases with training steps. The precision-recall trade-off (middle/right panels) reveals Neon's mechanism: precision monotonically decreases with w, while recall follows inverted-U peaking near the FID-optimal weight. This aligns with analysis: fine-tuning on synthetic data concentrates probability mass on well-captured modes, degrading coverage. By reversing this direction, Neon redistributes mass from over-represented to under-represennte regions, trading precision for improved recall and yielding net FID improvement. These dynamics intensify with longer fine-tuning, with later checkpoints showing sharper recall peaks and steeper precision drops. (See Appendix D for all models.)
4.2 AUTOREGRESSIVE MODELS
We evaluate Neon's impact on xAR-B and XAR-L (Ren et al., 2025) (ImageNet-256) VAR-d16 (Tian al., 2024) (1m2g)Net-256), and VAR-d30 (ImageNet-512), Both model families CFG, with VAR adding top-k/top-p sampling; these are mode-seeking samplers, and so our theory predicts Neon benefits. At evaluation, we jointly optimize both the merge weight w and CFG scale y. Co- optimization is crucial to reaching the best FID: increases recall at precision's expense, while does the opposite.
Results. Figure depicts the best FID after (7, w) grid search versus fine-tuning budget B, testing up to ( = 750k synthetic samples. The xAR family FID improves monotonically: xAR-B from 1.72 to 1.31 (750k synthetic samples, 0.41% additional compute); xAR-L from 1.28 to the state-of-the-art
FID 1.02 (750k samples, 0.36% additional compute), surpassing UCGM's s 1.06 (Sun et al., 2025). Even with just 1k samples, the xAR models achieve netrmaltimal performance (xAR-L: 1.05, xAR-B: 1.36), indicating that the degradation direction stabilizes quickly and requires minimal synthetic data to identify. VAR-d16 improves from 3.30 to 2.01 (750k samples, 0.64% additional compute) but
7

---

Published as a conference paper at ICLR 2026
FID
Precision
Recall
Asymplotic Recall-Precision
Figure 6: Optimal precision-recall trade-off's for R- as function and % Left: Heatmaps for FID, precision, and recall on ImageNet-256 (|5k= B B-1.25Mi) from grid search over w and 7. The star marks the best FID Asymptotic precision-recall curves showing expanded behavioral range through joint tuning.
achieving FID 2.01, unreachable by either parameter alone. Right:
IMM TT
IMM T
IMM T 4
IMM TT=8
2.4
7.5
1.8
3.5
1.6
1.8
1.4
0
B (Mi)
B (Mi)
B (Mi)
B (Mi)
Figure 7: Neon dramatically improves few-step inference for IMM on ImageNet-256. Minimum FID (optimized over and 7) vs. finc-tuning budget B for different [S]. Synthetic data were generated using 7=8, 1.1.5. From left: T=1,2 2, 4, inference steps. Ncon achieves substantial FID reductions with near-zero additional compute 0.005% IMM's training), with Ncon improved model with 4-step nearly matching base model with 8-step generation quality.
requires larger synthetic datasets datasets--performacc degrades with [S] < 90k. VAR-d30 achieves its best FID of 1.69 with just 90k samples; adding synthetic data provides further meaningful improvement, suggesting the model has reached its capacity for Neon-based enhancement at this checkpoint.
Figure 6 visualizes the (w, 7) interaction for VAR-d16. The FID landscape's is diagonal valley with yields FID 2.01. Independent optimization (y=1.25) yields FID 3.01 far worse. Joint tuning enables precision-recal trade-offs unreachable by either parameter alone: at the optimum, precision drops ~0.87 while recall rises ~0.63. The rightmost panel reveals the asymptotic behavior:
optimum (w*=11 =1.0, *==2777 >=0,,""002,
increases, the models converge to high precision (>> 0.95) but severely degraded recall (< 0.45), leading mode collapse. Higher w values provide partial protection at = 2, the low-recall limit rises to ~0.55 ~0.40
0, demonstrating how negative extrapolation counteracts CFG's mode-seeking tendency even at extreme guidance scales.
4.3 FEW-STEP GENERATORS
We investigate Neon paired with Inductive Moment Matching (IMM) (Zhou et al., 2025a) on evaluation, we tested
ImageNet-256. We generated S using T-8 steps with CFG scale 1=1.5. 7=1.5. the models across inference steps TE 2,, 2, and jointly searched over (w,7).
Results. Figure plots the FID vs. the linettunig budget B. Neon delivers dramatic improve- ments across all step counts with minimal overhead relative to IMM's 40,960Mi training budget. Performance scales inversely with the number of inference steps. Neon improves 7'=1 (single-step) inference to FID of 6.67. T-2 reaches 2.89: T-4 reaches 1.69; and T reaches 1.46. Re- markably, 4tstep inference nearly matches base model with 8-step quality (1.69 vs. 1.98), effectively halving the inference cost. Unlike IMM's tens of thousands of million-image image steps, Neon achieves optimal performance within 2Mi in all experiments for different (S], demonstrating rapid degradation direction stabilization for few-step models. The 30k sample sweet spot across all 7' suggests that
8

---

Published as a conference paper at ICLR 2026
few-step generators are particularly well-suited for Neon, as their training already distills mult dynamics into compact transitions, making the synthetic degradation signal especially informative.
4.4 ABLATION STUDIES
Neon is transferable across different architectures. A key advantage of Neon is that the degradation signal is transferable across different model architectures. We con- firm this empirically in Figure 8, by improving a baseline unconditional EDM-VP model (FID = 1.97) using syn- thetic data from different sources. While data from the model itself yields the strongest improvement (FID= 1.38), vross-architectuue
1.8
1.6
transfer is highly effective. Data from a flow matching model achieves FID of 1.59, and from an IMM model reaches 1.80. The theory expounded in Ap- pendix B.8 formalizes why Neon is transferable. Consider models A and B that minimize the same objective with Hessians H((A) and ( If these Hessians are spectrally close (equivalent norms up to constants e, C) and the ar- chitectures induce similar sampler biases (small mismatch
Fboe
1.4
10
25 (Mi)
Figure 8: Neon supports cross- architecture synthetic data transfer. We illustrate by using synthetic data from an IMM and a Flow model to improve EDM-VP on CIFAR-10.
Ç in the terms b, defined in (5), then anti-alignment transfers from one model to the other. That is, when model (A) satisfies g(A < - < O, any nearby model (B) inherits ,(B) < -u/2 < O. Intuitively, models learning similar representations exhibit similar overconfidence patterns, and so one model's 's degradation direction corrects another's 's biases. This makes Neon practical when generating samples from the target model is costly.
To test if any signal, we replaced the synthetic data with CIFAR-10C (Hendrycks & Dietterich, 2019), dataset of corrupted real images. Neon resulted in no FID improvement. This null result confirms that Neon specifically leverages the anti- alignment from a model overemphasizing its own modes bias absent in structured corruptions like CIFAR- 10C.
dataset provides a useful
10
109.5
How good must the base model be? key question is whether Neon's 's benefits are limited to nearly optimal mod- els, since our theory guarantees anti- anti-alignment only when the model error ||<|||= is small. To test this condition's 's robustness, we applied Neon to spectrum of EDM-VP base models trained on CIFAR-I0 subsets of varying sizes, Figure shows that Neon offers substantial improvements across the entire quality spectrum. Strikingly, a model trained on only 30k real samples (FID 1.87) and improved with Neon nearly matches the baseline model trained on the full 50k dataset (FID 1.85). This demonstrates that Neon can compensate for a 40% reduction in real training data, confirming the anti-alignment condition (s < 0) is not fragile but holds across a wide range of model qualities. This bodes well for data-scarce applications. In all of the experiments, we used |S| 6k.
10
[D]
Figure 9: Neon does not require a near- optimal base model to succeed.
Sensitivity to synthetic data quality. Our main experi- ments generated synthetic datasets using optimal inference settings for FID (e.g., = 2.7 for xAR-B). To test the sensitivity to the quality of S. we trained Ncon+xAR-B on ImageNet-256 with |S| = 90k and varied the CFG scale used during generation. We generated synthetic datasets with J0, 6.2). fine-tuned on each S, and then optimized the final Neon model. Figure 10 demonstrates Neon's re- markable robustness: despite training on synthetic data of varying quality, the final FID remains near-optimal (1.30-1.31) for any 7 E [1,3], Even suboptimal synthetic datasets yield performance within 3% of optimal. This sug- gests that Neon captures the fundamental mode-seeking
FID
1.35
1.3
Figure 10: Ncon does not require high- quality synthetic data to succeed.

---

Published as a conference paper at ICLR 2026
bias rather than requiring precisely tuned synthetic data. Only at extreme values (e.g., 2 6) does performance degrade significantly, likely due to excessive mode collapse in S.
5 CONCLUSIONS
We have introduced Neon, simple and efficient post-processing method that improves generative models by inverting the degradation caused by self-training. Neon grounded in key insight: common mode-secking inference samplers induce a predictable anti-alignment between gradients from synthetic and population data, explaining both the failure of naïve self-training and Neon's success. By extrapolating away from this degradation direction, Neon corrects the sampler's inherent bias, redistributing probability mass from over-represented modes to reder-represented ones, thereby enhancing recall and overall generation fidelity. Neon's 's effectiveness across diverse model architec- tures and training datasets suggests that we can reframe model degradation not as a failure, but as structured, harnessable signal for improvement in increasingly sata-scarce field. Our work also positions inference samplers as valuable diagnostic tools for uncovering and remedying model's distributional flaws.
Neon opens several promising avenues for future research. Can we identify ppromtiigg samplers that foster positive alignment between synthetic and real data, thereby enabling direct self-improvement without inversion? Can we actively synthesize optimal "bad" datasets that elicit degradation direction that maximizes the corrective signal?
As the demand for more capable models outpaces the availability of high-quality data, methods that extract more value from the models themselves will be crucial. Neon's success establishes a practical principle for self-correction: by identifying and inverting the predictable biases introduced by model's s own processes, we can achieve substantial performance gains. This work shows that even seemingly harmful signals can be harnessed for improvement, a finding that shows sometimes to move forward, we have go backward.
ACKNOWLEDGMENTS
This work was supported in part by NSF Awards 2145346 (CAREER), 02133861 (DMS). 2113904 (CCSS), and the NSF AI Institute for Foundations of Machine Learning (IFML): ONR N00014-23-1- 2714; ONR MURI 100011-20-1-221
DOE DE-SC0020345; and DOI 140D0423C0076. Thanks to Predrag Neskovic for pushing us down the path towards understanding negative extrapolation and Ahmed Imtiaz Humayun for early discussions and for suggesting Algorithm for model self-improvement with synthetic data.
ETHICS STATEMENT
This research is methodological contribution evaluated standard public datasets (ImageNet, CIFAR-10 10, FFHQ) with no private or identifying data. We do not anticipate direct harms from our experiments; however, with any image-synthesis method, potential misuse exists.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sina Alemohammad, Josue Casco- Masco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go MAD. In International Conference on Learning Representations, 2024a. URL https: / //openreview fert/ffoum?? ji
Sina Alemohammad, Ahmed Intiaz Humayun, Shruti Agarwal, John Collomosse, and Richard Baraniuk. Self-improving diffusion models with synthetic data, arXiv preprint arXiv:2408.16333 2024b.
Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on 2023.
computer vision and pattern recognition, pp.
Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong. and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. arXiv preprint arXiv:23 004999 2023.
Richard P. Brent. Algorithms for Minimization without Derivatives. Prentice-Hall, 1973.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1 1315-11325, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1109/CVPR.2000 2200..008....
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021, URL https: /tpp//ppeevviieeeeeeeee
net/ / forum? i d=AAAuCvzaVV.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 12211- 19835, 2021.
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. tale of tails: Model collapse as change of scaling laws. In International Conference on Machine Learning, 2024.
URL https::/p//..c....tt. /wwpnreveieeeeeeeeeeeeeeeeeeeeee net/ forum? dd=Kvvuu 4hhh..
Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model col- lapse: Scaling up with synthesized data requires reinforcement, arXiv preprint arXiv:2406.075// 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /cppnreviieeeeeeeeeeeeeeeeeeeeee net www.mmodoommmmseeeeeeeeee
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. arXiv preprini arXiv:2303. 14389, 2023.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. Is model collapse in- evitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413, 2024.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL https: htt.httts/:/ppereeiiiiii /wwpneevieeeeeeeeeoooeeeeee.e..e nete:/iwwweeeeeeeeeeeeeeeeeeeeee wwt.:/ntttoootee/oee

---

Published as a conference paper at ICLR 2026
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701. 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Nick Huang, Aaron Gokaslan, Volodymyr Kuleshov, and James Tompkin. The gan is dead; long live the gan! modern gan baseline. Advances in Neural Information Processing Systems, 37: 44177-44215. 2024.
Allan Jabri, David J. Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 5941-59961 2023.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 10124-10134 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 .08361, 2020.
Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7799-7808, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401440444 -4110, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33: 1111211412221 14, 2020.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 26565-26577. Curran Associates, Inc., 2022.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. arXiv preprint arXiv:2406.022
2024a.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24174-24184, 2024bb
Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. In International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 16567- 16598. PMLR, 2023. URL
http::///ccceeeeeeeeeeeo mlr. www..........................e.. v222 10/22/kmmd /immmmmm333 31 htmll.
Dongjun Kim, Chich-Hare Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning prob- ability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Representations, 2024. URL tettts::/openreeee
n wweeeeettreeeeeeeeeeeeeeeeeeeeee (iderummmmiimmyi BeDTD.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives the ELBO with simple data augmentation. In Thirth-eeventh Conference on Neural Information Processing Systems, 2023. URL https: / htttlps:///ppeeeee /ttp//eweeeeeeeeeeeeeeeeeeeeeeee n forum d=NnMEadcdyD.
12

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp. 3927-3936, 2019.
Tuomas Kynkäääniemii, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724 2024.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 16424-56445 2024.
Lin, Richard Zhangg Frieder Ganz, Song Han, and Jun-Yan Zhu, Anycost gans for interactive image synthesis and editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14986-14996, 2021.
Yaron Lipman, Ricky Q. Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling, In International Conference on Learning Representations, 2023,
URL https: wwtp//ppweweeeeeeeeeeeeeeeeeeeep net forum? Pd-QqMMDDC.....
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances neural information processing systems, 015775-57877 2022.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23 40. Springer, 2024.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pp. 8162-8171. PMLR, 2021.
Dogyun Park, Sojin Lee, Sihyeon Kim, Taehoon Lee, Youngjoon Hong, and Hyunwoo Kim.
Constant acceleration flow. Advances in Neural Information Processing Systems, 37:90030-90060, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 2023.
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang- Chieh Chen. Beyond next- token: Next-x prediction for autoregressive visual generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15781-15791, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings. SIGGRAPH '22, pp. 1-10, New York, NY, USA, 2022.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained recursively generated data. Nature, (802)))77555 2024.
13

---

Published as a conference paper at ICLR 2026
Yang Song and Stefano Ermon. Improved techniques for training coore-based generative models. Advances in Neural Information Processing Systems, 33:12438- 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Πyy Sutskever. Consistency models. International Conference Machine Learning, pp. 32211 1-22522. PMLR, 2023.
Peng Sun, Yi Jiang, and Tao Lin. Unified continuous generative models. arXiv preprint arXiv:2505.07447, 2025. URL https::////// https:://axxx...tt http:/bb//255574/5
Yuhta Takida, Masaaki Imaizumi, Takashi Shibuya, Chieh-Hsie Lai, Toshimitsu Uesaka, Naoki Murata, and Yuki Mitsufuji. SAN: Inducing metrizability of GAN with discriminative normalized linear layer. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ / petts::/cpennee /pwpeeeeevvvieeeeeeeeeeeeeeeeeee
net
Yi Tang, Peng Sun, Zhenglin Cheng, and Tao Lin. Gmem: A modular approach for ultra-efficient generative models. arXiv preprint arXiv:2412.0878// 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In Advances in Neural Information Processing Systems, 2024. URL https: opwprevieneeeeeeeeeeeeeeee wwtpeeewwwwwweeeeeeeeeeeeeeeeeee www./eee///iddeeeeeeeeeee...eeee gooooooooooooooo..
Alexander Tong, Nikolay Malkin, Kilian Fatras, Lazar Atanackovic, Yanlei Zhang, Guillaume Huguet, Guy Wolf, and Yoshua Bengio. Simulation-frr schrödinger bridges via score and flow matching. arXiv preprint arXiv:2307.0380 007.0662. 2023.
Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024. URL ht https: htttp:://pperreviee
net /tt///oodddddnsneeeeeee 73A..
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems, 2021. URL https: / /opnreevin. net/ pet/fozze Eorum? ww..d=ssTYYG000
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.0433 2022.
Mark Weber, Lijun Yu, Qihang Yu, Xiang Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chich Chen. Maskbit: Embedding-fiee free image generation via bit tokens. arXiv preprint arXiv:2409. 162 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In International Conference on Learning Representations, 2022a. URL https: /ttp::::////eeeellee net/fo htte:/Eoummiiddffypp.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022b.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta. Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming- Yang. Irfan Essa. David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. The Twelfth International Conference on Learning Representations, 2024. URL httph:tt:////c... httpt:://ppennnteee /epenrev new. n forum i A=C2grANCF44.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. In Advances in Neural Information Processing Systems, volume 37, 2024.
14

---

Published as a conference paper at ICLR 2026
Bowen Zheng and Tianming Yang. Revisiting diffusion models: From generative pre-training to one-step generation. arXiv preprint arXiv:2506.09377 2025.
Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming-Yu Yu Liu, Jun Zhu, and Qinsheng Zhang. Direct discriminative optimization: Your likelihood-based visual generative model is
secretly a GAN discriminator. In International Conference on Machine Learning, 2025. Spotlight.
Lingi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching. In International Conference on Machine Learning, 2025a. URL https:/ /ttp://////eeeeeeeeeeeeee net / forum?id- pwNSUo7yUb.
Mingyuan Zhou, Huangjie Zheng. Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. In International Conference on Learning Representations, 2025b. URL https: http///pppewe.e.eeee/eeee http://wwwwww...e.eeeeeeeeeeee weet/forrmmii=1112201
15

---

Published as a conference paper at ICLR 2026
A STATE OF THE ART COMPARISON
Table A.1: Comprehensive comparison of generative models across four standard benchmarks. Best
results are highlighted in blue
(a) Results on CIFAR-10 10.
(b) Results on FFHQ-64x64
Type
Cond
Type
Model
NVE
StyleGAN2-ADD (Karras al., 2020)
2.42
R3GAN (Huang et al.,
GAN
StyleGAN-XL (Sauer 2022)
(Takida 2024)
1.36
1.85
GAN
MSG-GAN (Karewarr Wang. 2020)
Anyoost GAN (Lin et al., 2024)
1 2.52
CAF (Park et al., 2024)
1.48
1.39
StyleGAN2 (Karras et al., 2019)
1 3.32
DDPM (HP et 2020)
1000
3.17
EDM-Ge (Karras al,, 2024h)
71 1.98
Dill &How
(Nichol Dhuriwal, 2021)
2.90
(Karras al., 2024b)
NCSN+ (Song & Ermon, 2020)
2000
2.20
EDM-VE VP (Karras et al., 2024b)
79
2.39
DPM-Solver (Lu
2022)
10
4.70
LSGM (Vahdat et al., 2021)
138
2.10
SiD (Zhou
2025b)
1.04
EDM. VP (Karras al., 2024b)
35
1.97
1.79
Post-hoc.
EDM + SIMS (Alemohammad et al., 2024b)
158
1.04
GMMe-XLL (Ting et al., 2024)
35
1.22
EDM D20 (Zheng & Yang. 2025)
1 1.08
Flow Matching (Lipman et al., 2023)
100
3.50
EDM + D2O-F (Zheng & Yang, 2025)
0.85
Rectified Flow (Liu al., 2023)
2.58
Ours
EDM + Neon
79 1.12
Post-boe Few Tow-step step
CTM (Kim et al., 2024)
1.87
sCT (Song 2023)
2.06
IMM (Zhou et al., 2025a)
3.20
EDM DG (Kynkäämiieni et 2024)
53
1.77
1.64
EDM DDO (Zheng al., 2025)
35
1.38
1.30
EDM SIMS (Alemohaminad et al., 2024b)
70
1.33
EDM SID"
2025b)
Ours
EDM Neon
35
1.38
Flow Neon
100
2.32
(c) Results on ImageNet-256x2560
(d) Results on ImageNct-512xx52.
Type
Model
NFE
FID
Type
Model
NFE
FID
GAN
GigaGAN (Kang al., 2023)
3.45
BigGAN-docp (Brock et al., 2019)
8.43
StyleGAN-NI (Samee al., 2021)
2.30
GAN
StyleGAN-XT. (Samer (Saneer SID A (Zhou al., 2025b)
2022)
1
1.37
ADM (Dhariwal Nichol, 2021)
250
10.94
Dillusion
LDM-4 (Romhach et al., 2022)
250
10.56
ADM (Dhariwal & Nichol, 2021)
(Pechles
2023)
250
9.62
ADM-U (Dhariwal Nichol, 2021)
500
9.96
U-VI (Bao et al., 2023)
50
2.29
DIT-XT.2 (Peehles Xie, 2023)
250
12.03
MDT (Gao al. 2023)
250
6.23
Difusson
STT-XL (Ma et al., 2024)
250
8.30
REPA-UCGM (Sun et al., 2025)
80
1.06
RIN (Jabri al., 2023)
1000
3.95
U-ViT-L- et al., 2023)
512
3.54
MaskGTT (Chang et al., 2022)
8
6.18
VDM++ (Kingma & Gao, 2023)
512
2.99
Masked
MAR al., 2024)
100
1.98
2024h)
MaskBit ( (Wber et al., 2024)
256
1.52
EDM2-XXL (Karras et at., 2024b)
63
1.91
VOGAN (Yu et al., 2022a)
256
15.78
MAGVIT-v2 (Yu al., 2024)
64
3.07
VAR-d16 (Tian
2024)
3.30
Masked
MAR-L al., 2024)
1024
AR
VAR-d30 (Tian ct al., 2024)
10
1.92
xAR-B (Ren al., 2025)
1.72
VAR-d36-s (Tian et al., 20244 XAR-L (Ren al., 2025)
2.63
xAR 1. (Ren el al., 2025)
50
1.28
AR
50
1.70
Few step
Shortcut (Frans et al., 2025)
10.60
EDM2-S STMS (Alemohammad et 2024h)
63
1.73
IMM (T=1) (Zhou al., 2025a)
IMM (T=8) (Thou ol., 2025a)
7.77
1.99
Post-boe
HDM2-L + DDO (Zheng el al., 2025)
EDM2 AG (Karras al.. 2024a)
63
63
1.25
1.21
Post-hoc
VAR d16 DDO (Zheng al., 2025)
10
2.54
EDM2 + SiD (Zhou et al., 2025b)
1
1.37
VAR-d30 DDD (Zheng 2025)
1.79
Ours
VAR-d30.s Neon
10 1.70
Neon
2.01
XAR-B B Neon
30
1.31
XAR-L Ncon
1.02
IMM (T=8)- Neon
1.46
IMM (T=4) Neon
1.68.
IMM (T=2)+ Neon
2.88
IMM (T=1) Neon
6.67
16

---

Published as a conference paper at ICLR 2026
We summarize results and provide comprehensive comparison with state-of-the generative models in Table A.1. The following section discusses Neon's 's performance on each benchmark in more detail, highlighting its standing relative to pop-performing models and other post-hoe methods.
CIFAR-10 On both conditional and unconditional CIFAR-IO, Neon improves the EDM-VP baseline to 1.38 FID while maintaining its 35 NFE (Karras et al., 2024b). the conditional setting, this is competitive with DDO, which achieves 1.30 FID from the same base model but requires significantly more training compute 12% extra vs. Neon's 1.75%) (Zheng et al., 2025). In the unconditional setting, Neon's 's 1.38 FID is identical to DDO's and close to the SOTA held by SIMS at 1.33 FID (Alemohammad et al., 2024b). Notably, SIMS requires doubling the NFE to 70, making Neon more sampling-efficiee alternative. Ncon also demonstrates versatility by improving Flow Matching model to 2.32 FID (Lipman al., 2023).
FFHQ-64x64 On FFHQ. Neon significantly enhances the unconditional EDM-VP model, lowering its FID from 2.39 to 1.12 with 79 NFF. While the state-of-the-art is held by the one-step D20-F at 0.85 FID (Zheng Yang, 2025), Neon's performance is highly competitive. It stands against other post-hoc methods like SIMS (1.04 FID, 158 NFE) (Alemohammad et al., 2024b) and the one-step distilled SiD A (1.04 FID, NFE) (Zhou et al., 2025b). Neon achieves its strong result with simple parameter merge that preserves the base sampler's s structure, offering a distinct trade-off between FID and NFE.
ImageNet- 2566222 On ImageNet-256, Neon sets new state-of-the-art, improving the xAR-L model from an already strong 1.28 FID to 1.02 FID (Ren et al., 2025). This surpasses the previous best result of 1.06 FID from REPA-UCGM (Sun et al., 2025). Neon also demonstrates its superiority over DDO on this benchmark; when applied to the same VAR-d16 base model (Tian et al., 2024), Neon achieves a 2.01 FID, which is significant improvement over DDO's 2.54 FID (Zheng et al., 2025). Furthermore, Neon consistently improves other architectures, including xAR-B (1.31 FID) and IMM (1.46 FID).
ImageNet-512x512 On ImageNet-5122 12, Neon improves the VAR-d30 model to 1.70 FID with 10 NFE (Tian et al., 2024). While the state-of-the-art belongs to EDM2-L+DDO at 1.21 FID (Zheng et al., 2025), Neon's result is competitive with other post-hoc methods applied to different base models, such as EDM2- SMM2-++SIMS (1.73 FID) (Alemohammad et al., 2024b). showcases Neon's 's ability to enhance autoregressive models at higher resolutions with its characteristic low compute overhead.
Summary Across all benchmarks, Neon proves to be simple, efficient, and broadly applicable post-hoc method for improving generative models. It achieves new ttate-of-the--e on ImageNet-255 256 and delivers highly competitive results elsewhere, often with superior sampling efficiency compared to other post-hoc techniques. A key finding is that Neon's effectiveness corresponds directly to the quality of the base model it enhances; applying to a stronger foundation like xAR-L. yields a greater improvement and the best overall performance. This positions Neon reliable tool for adding a final layer of polish to strong, pre-existing generative models with minimal computational effort. Crucially, since Neon improves the base diffusion model itself, its benefits potentially orthogonal to distillation methods; one could apply SiD? A or D20-F to the Meon-enhances model for further gains.
17

---

Published as a conference paper at ICLR 2026
B PROOFS AND DETAILED EXPLANATIONS
This appendix provides the complete theoretical foundation for Neon. We begin with a simple Gaussian warmup example that demonstrates Neon's core mechanism with minimal mathematical machinery. We then present roadmap outlining the logical structure of our proofs, followed by the detailed technical derivations.
B.I WARMUP: GAUSSIAN ESTIMATION WITH FINITE DATA
This subsection gives self-contained, concrete instance of Neon in the simplest setting: estimating the mean and covariance of a Gaussian distribution from limited data. The goal is to show-with minimal machinery-wih tiny Neon step reduces the population (test) loss in expectation. The calculation also clarifies the role of mode-seeking synthesis and the sign appearing in the main theorems.
Population objective and finite-sample MLE. Let the ground-truth law be (x)x) M( Σ.) RR.. We fit the model family (u.z(z) = N( μ4 Σ) by minimizing the population negative log-likelihood
[p Σ) Ex-nal - log (n,
(logdet E + tr tr '2, + (n pm) ))) + D log((2T) (E.
which is minimized at (1,, E) = (M,, E.).
In practice we observe n samples &1,,... Tn ~ Pa and use the finite-sample MLEE
Σ= Σ (I. p))x, T.
For Gaussian data, f and are independent; moreover n Wishart(E.,
Synthetic objective. To model the "mode-seeking" behavior of common samplers in a controlled way, define the synthetic law
Psyn (,, = N(x i, -2)),
T>0, 0,
so 0 < r <1 shrinks variability (mode-seeking) and expands variability (diversity-seeking) Given (j, E), the synthetic negative log-likelihood is
(yy Pr: Er: E2 log qu [x)]
= ( 14) -
(μ ur) tx(2)))7E,,) + logdet + D log (2T)
Equivalently, with Csyu = -- (11 A)(H T)
Layn (14, Σ; ft, E,
(logdet E tr(E= + Dlog(2r))
Taking derivatives gives the standard Gaussian gradients
V Pptgyynle (H,, Σ) = Σ ( -A), VzLnya(p, E) = { - E-'CayuE-1)
Evaluated at the MLE (f, E))
V LyyLnn Σ)= =,
V (ALLeyn(1,555 Σ)
= = (1- --(-----1)))))
(B.1)
Thus the synthetic step and Ncon leave the mean unchanged to first order: μ4 = P,, PNeon = f (up to o(1)).
18

---

Published as a conference paper at ICLR 2026
A tiny synthetic step and the Neon flip. Starting from (ft, Σ), take infinitesimal gradient step on Leyu with rate 17:
-- Vr (A, Σ) + d(n) = E [1 - -1)))
i.e, the synthetic self-train moves inward when 0 < < 1. Neon flips that tiny move via a linear
merge with weight w > 0:
Neon (1 +)) -. = (1 Puwn Jun Hー1 o(1)) 5 = o(n)
a>0
To first order, Neon applies AEnoon E-1 with small a > 0x (1 -2)
First-order change in population loss. Compare pop (i. E) and (f,, ENoon. A first-order
expansion yields
ALpop == Ppop (A, Neonn (R, En s (Vy (f, E),, ALiNoon)
(A, = tr(AA B).
We now average over the randomness of (fi, 5. from finite sampling. Two classical facts for Gaussian data (valid for n D + 4) are sufficient:
(i) A and Σ are independent with [E[( - ) - ] = E./n;
(ii) n E is Wishart with scale Σ. and v = n 1,11,0 so
n²
EY--1 D-D 2T
B E-1]
(n D - 2) D 4)
Using Vy.Lpop = -1-------
with C. = E. + (p - -a HO.T .independence gives
E BV hppp (1i, E3} = (R|E-1) - (1 + ) 1
أΣ, 5 1 = -7MLE E,
with
n(D + 5)
7MLE 2 D 2) D - 4) > 0.
Combining with AENeun a + o())
E E[pLpopl x (- a 1EL--11 = = =-aTMLE n n- - tr(2. + o(a) < 0.
71
Conclusion: for sufficiently small O, E Lpop (fe, ENeou) - Lp }
0.
Geometric intuition. The synthetic step systematically identifies the local "tightening" direction (because 0 < r < 1 shrinks variability); Neon flips that direction by tiny amount. On average, the population gradient at (já, Σ. prefers small de-tightening, so the inner product negative and the population loss drops. In this toy case, the mean does not move: (Neon u to first order.
Remarks on T >>> and on not optimizing Layn to convergence. If the synthetic law is diversity- seeking (r 1). the sign in Eq. equation B.1 reverses and the first-order Neon update becomes AE E-1 with a (1 - >rm < 0 for w, >> 0. In practice, one simply tunes the signed merge strength a (equivalently, the sign of w) so that a > O; after this tuning, the same conclusion holds, and the correction induced by diversity-secking synthesis coincides with the mode-seeking case at the level of the final Neon update.
Finally, are not interested in minimizing the synthetie objective Layn to its exact optimum. In this Gaussian toy, the minimizer is (μ, Σ) = (a. ""))) If one trained to that point and then merged, the result would be pure scalar rescaling
Exou = (1+ + w) --(w(((2)) = (1+
which carries additional directional information beyond Σ. What Neon needs is only robust estimate of the local gradient direction; hence we use a very small synthetic training budget so the direction is measured in the local regime and the negative extrapolation is reliable.
19

---

Published as a conference paper at ICLR 2026
Lop (ā, I(a))
Loop (e,, ).) (global mia)
Population loss Lpop
22.5
Neon strength in (aa 3 +a5
Figure B.I: Population loss Lpop (ū. E(a)) along the Neon path for a Gaussian with = 10, 100. The dashed line shows the global minimum (requires true parameters). The optimal a (blue dot) substantially improves over the baseline empirical estimator.
Finally, validate theoretical framework controlled synthetic task where ground truth known. We generate n = 100 samples from a D = 10 dimensional Gaussian N N(u,, Σ.) and evaluate the population loss Lpp (7, E(a)) along the Neon path Σ(a) = a5-1 for E [0. 1]. Figure B. shows that Neon improves over empirical baseline (ar = 0), with the optimal a achieving reduction in population loss. The dashed line indicates the global minimum popp(1us E,), unattainable without knowledge of true parameters. The gap between the optimal Neon loss and this theoretical minimum reflects the fundamental constraint of finite-sample estimation within our parameter family, consistent with our theoretical predictions.
B.2 ROADMAP: LOGICAL STRUCTURE OF THE PROOFS
The Gaussian warmup illustrates Neon's mechanism transparent setting. We now outline how the general theory extends these ideas to arbitrary generative models. The proofs are organized in five conceptual stages:
Stage 1:
Framework and first-order analysis (Sections B.3-B.4). We establish notation, define the alignment scalar (ra: Pra) (where Td; Y are the real-data and synthetic gradients), and prove that Ncon reduces population risk when < O.
Key result: Theorem B.1 shows
Rdata Rdata(ONeoul) Rdats (0,) was (wa)² rg TT HTTHaPr + ((wa))),
establishing that the sign of s determines whether Neon helps (s < 0) or harms (s 0).
Stage 2: Geometric characterization of anti-alignment (Section B.5). We decompose the syn- thetic gradient into signal and bias components, introducing the angle between the model error E = 0, and the sampler-induced bias b = E,,.,. [dg- (X)]-
Key result: Theorem B.4 provides a directional upper bound
$ $ M(1 + m)) m))]le[[?,, mnolle HH, cos cos pl+ + O(I
showing that when cos < (obtuse angle) and ||s|| small, the linear term dominates and < 0.
Stage 3: Mode-seeking samplers induce anti-alignment (Sections B.6-B.7). We prove that all commonly-use used inference procedures mode-seeking and therefore guarantee cos op near good models.
Key results:
20

---

Published as a conference paper at ICLR 2026
Autoregressive models (Lemmaa B.6): Temperature 1, top-k, and top-p sampling all induce ((>) x f(log Pe, Pe, with f nondecreasing, which implies cos (p < to first order in |=||e-
Diffusion/flow models (Section B.7): Any finite-step ODE solver introduces discretization error that is mode-seeking (under mild curvature-density coupling assumptions), yielding cos sp 0 even at the neutral point (no explicit guidance).
Together, these results establish that & < 0 holds for all standard inference configurations, making Neon universally applicable.
Stage 4: Complete characterization paaraattriaatonddddvesitt--essie samplers (Section B.9). We complete
the theory by characterizing diversity-seeking samplers (r for AR, or inverse truncation), which reverse the gradient alignment to cos y 0, yiclding > 0.
Key result: Theorem B.13 provides a lower bound showing that diversity-secking configurations yield 2 (m- 1 }l| lir, + H.)) In this regime, the Neon formula N (1+w)0, w0, improves performance for - < w < 0 (interpolation toward f,) rather than w > (extrapolation away from 02). Thus the sign of determines the sign of the optimal merge weight, and Neon's 's framework applies universally by tuning w appropriately.
Stage 5: Finite-sample effects and hyperparameter guidance (Sections B.10-B.11). We analyze the impact of finite synthetic datasets S and provide theoretical guidance for choosing the training budget B and merge weight w.
Key results:
Lemma B.17 shows that short fine-tuning (small a')) with moderate |S] produces low-variance, reliable estimate the degradation direction - ( (0,).
Very small [S] suffers from high variance; very large |S| amplifies curvature effects via the quadratic term. This predicts the U-shaped performance observed empirically in Figure 3.
Controlled toy experiments (Section B. Figures B.2 and B.3) directly validate the theory by measuring gradient alignment as function of sampler type.
Summary. The complete proof proceeds follows:
Show that Neon helps when s < 0 (Stage 1)
2. Express in terms of the geometric angle yp between model enror and sampler bias (Stage 2)
3. Prove that all standard samplers mode-seeking, yielding cos y 0 0 (Stage
4. Characterize Diversity-secking samplers where > 0 and optimal w changes sign (Stage
5. Provide finite-sample guidance and empirical validation (Stage 5)
This roadmap should guide the reader through the technical details that follow. Each subsection refines and formalizes one step in the logical chain, culminating complete characterization of when and why Neon works.
B.3 ASSUMPTIONS, NOTATION, AND IDENTITIES
This subsection establishes the mathematical framework for analysis. We introduce the basic objects (loss, risk, gradients), state our standing assumptions, define the geometric structures used throughout the proofs, and provide comprehensive notation reference table.
Basic setup. Let (z) be differentiable per-example loss. The population risk (or data risk) is
Rdata (f) Epan [X)).
The per-example gradient is denoted
Φo(x) Volo(z) E IR?
21

---

Published as a conference paper at ICLR 2026
and its Jacobian (with respect to 8) is
Hœ(æ) (x))(() Vzlo(z) RPxP.
The data Hessian (population Hessian the optimum) is Ho (X)).
Ha RaRaata((")) E
Standing assumptions.
Throughout the appendix, we maintain the following assumptions:
(A1) Optimality. There exists population risk minimizer 0 E arg ming Rdata (0), which satisfies the first-order condition
E 1oo. (X)) 0.
(A2) Regularity. The data distribution Pdata and model family satisfy:
Common support between Pdata and the model distributions;
Dominated convergence holds, allowing interchange of limits and expectations;
The gradient map @9 and Jacobian map (x) are locally Lipschitz continuous near 0".
(A3) Local analysis. Our base model 0, lies in small neighborhood of the optimum:
0, 0* + with || small,
and all remainder terms O(I|= smaller.
(A4) Rank condition. If Ha is not full rank, all statements are understood to hold on the image space (m(Ha).
Geometric structures. try:
For any positive-definite matrix M > O, we define the M-induced geome-
Inner product:
(v. = x. My,
Norm:
| las Vx Mr A/
Operator norm: ||Aloo.MM = 12 AM- 11/2 1/2 |lop-
When M = Ha. we write || I|IH and (-.))., Ha for brevity.
For any preconditioner P 0 (e.g., the inverse Hessian, an Adam-like diagonal, or identity), define the whitened preconditioner
K
which satisfics spectral bounds
mī = K 2 MT for some 0 m S M < X.
These constants m and M control the condition number the preconditioned system and appear throughout our bounds.
Notation reference. Table B.1 provides a comprehensive reference for all notation used in this appendix. Symbols are grouped thematically for easier lookup during proofs.
Conventions.
Scalars: lowercase (s, a)); vectors: lowercase bold or Greek (0,,,ra); matrices: uppercase (Ha, P,K).
Subseripts: (real/base), (synthetic). (optimal).
All norms default to Euclidean unless subscript specifies otherwise (e.g., || ml
B.4 NEON IMPROVES UNDER ANTI-ALIGNMENT
Having established our framework and notation, we now prove the central result: Neon reduces population risk when the synthetic and real-data gradients anti-aligned. We begin by defining the key quantity that governs Neon's behavior the alignment scalar s and then show how its sign determines whether Neon helps harms.
22

---

Published as a conference paper at ICLR 2026
Table B.I: Notation reference table
Symbol
Description
Symbol
Description
Data, models, and parameters
Pias
Ground-truth data distribution
0"
Population risk minimizer
Go
Generative model
Base model, 0, + Self-trained model
pe
Model distribution
q
Inference distribution
O
Neon model, (1 w)0, w0,
Model error, 8, 8"
Inference hyperparameters
Losses, gradients, and Hessians (z())
Per-example loss
Real-data gradient
Raata (0)
Population risk
na
Synthetic gradient
Rayn(O)
Synthetic risk
Pa
(S)
Empirical synthetic gradient
(5(x)
Per-example gradient
8
Alignment scalar, (ra, Pr,)
Ho(x)
Per Per-example Hessian
b
Sampler bias
Ha
Data Hessian
A
Curvature tilt
Geometric structures
(z, y)
M-inner product, x My
7
Linear bias, ||b|l
M-norm, (x Ma)
Curvature tilt, | AAllop..ooo. Hp. Angle between and H b
||Allop.ax
M-operator norm
Optimization
P
Preconditioner, P> x
m,
Spectral bounds, / 1 K N MI
K
Whitened preconditioner, HHH Merge weight
Step size (learning rate) Number of fine-tuning steps Training budget (millions of images)
w
Optimal merge weight
B
Synthetic data and sampling
S
Synthetic datasct
7
Temperature (AR sampling)
|
Synthetic dataset size
S
Score scale (toy experiments) Fréchet Inception Distance
CFG scale
FID
Model- Model-specific notation
(x)))
AR score, V log pe (1)
80 (Zz, t)
Diffusion score function
F
Fisher matrix for AR
Ue (Zi, 1)
Flow matching velocity
Operators and conventions
Expectation
||
Frohenius norm Positive part, max(-, 0) PSD matrix
tr
Trace Big-O, little-o
o0.0()
Alignment scalar and synthetic objective. We work with two gradient directions evaluated at the base model 0,. The real-data gradient points toward improving performance the true data distribution:
rd Rastaa
The synthetic gradient is computed with respect to the synthetic risk, which measures performance on samples drawn from the model's own inference procedure:
Reyn (0) == Eyg. (X)],
T. VoRayn(0)l..-
The central quantity controlling Neon's effectiveness is their preconditioned inner product, which we call the alignment scalar:
(ra Pr.)-
(B.2)
When s < the gradients are anti-aligned: moving in the direction - Pr, (away from synthetic fine-tuning) actually improves the real -aal risk. This anti-alignment is the key insight that makes Neon work.
Main result: Neon reduces risk when < O. We now formalize how Neon exploits anti aligneent.. The following theorem provides a Taylor expansion of the risk after Neon's parameter merge, showing that the sign of determines whether performance improves.
23

---

Published as a conference paper at ICLR 2026
Theorem B.1 (One-ster step Neon improvement). A short synthetic mrntheiicffie--ieee fine-tune produces 0₂ = = Pr, (aa²) { for some a > O. For w >> 0, the Neon merge is
ONoon
(1 w),, w0, 0, wa Prs + O(wa²).
Let Ha- = R2Rdata(B,, Then
(O)) + was + (wa)²
Rdata (ONcon) =
2 rg TT AaPrs + ((10a))))
(B.3)
In particular, if then for all sufficiently small w > we have Rdata (ONeon) < (d,t If moreover Ha = 0. writing q = r. P HaPr, 20, any
0 < w
guarantees
Rduza (ONoo)) & (data(Or) (up to )((wa)))),,
and the quadratic proxy is minimized at w² = - s//(a))) > O.
Proof. The proof proceeds in three steps: (i) we express the Neon update as step in the synthetic gradient direction, (ii) we perform a Taylor expansion the risk along this direction, and (iii) we analyze when the expansion is negative.
Step I: Neon as directional step. From the short synthetic fine-tune we have
8, = 0r aPrr + O(a²).
Therefore
0Nom = (1 w) w0, = 0, + wa Pr. + Owwa)).
Thus Neon moves in the direction Pr. opposite of the synthetic gradient direction.
Step 2: Taylor expansion along the Neon direction. Define the univariate function
(T)) = Retta (0, + Pra), and set wa.
Taylor expansion of & at 0 gives
(bT) (0) + T0)(0) + 72 (((0) + 0(7²) 2
By the chain rule,
(0))) = (ra, Pr.) = 8, w²(0) r3 PT Å. Pr.
Substituting wa yields
Retata (0 Retata (0r) + wac s + (wa)²
2 r, T HaPr. + O(wwa)3),
which is equation B.3.
Step 3: When does the risk decrease? If < O, the linear term was is negative and dominates for sufficiently small w > 0. giving Raata (fNeen) Raata (fr).
If. in addition, A. = 0, then (0) 2 and the quadratic proxy + xb(0) + T + ((()) is minimized at
*0( > 0.
Since т = wwa this gives the safe window < w <
2s and the minimizer w" (a)"()))
a (0) ar. HaPP.
24

---

Published as a conference paper at ICLR 2026
Interpretation. Theorem B.1 reveals the fundamental trade-off in Neon. The expansion equa- tion B.3 is a balance between:
Linear term (was): Negative when < O, this drives improvement.
Quadratic term (wa)) Q): Always non-negative (when A č 0), this penalizes moving too far.
For small w, the linear term dominates, so Neon helps. For large w, the quadratic term dominates, so performance degrades. The optimal wt balances these forces, and depends on both the alignment strength [s] and the curvature q.
Remark B.2 (No convexity needed: directional smoothness). The PSD requirement on Ha can be replaced by an upper curvature bound along the step direction d == Pr.. If there is Ldie 2 with d' Radaa (0, + rd)d s Lailld]]e for т near O, then the same conclusion holds whenever a 1 This weaker condition is sufficient because Neon only moves along a single <
direction, so we only need smoothness along that rather than global convexity.
B.5 AN UPPER BOUND ON 8 AND SUFFICIENT CONDITIONS FOR ANTI-ALIGNMENT
We now pursue the question posed at the end of the previous subsection: when does 8 < 0 hold? The key is to decompose the synthetic gradient into two parts-a "signal" component that matches the real-data gradient, and a "bias" component induced by the sampler. We will show that the sign of depends on geometric angle between the model error and this sampler-induced bias. This geometric characterization then allows us to identify simple, verifiable conditions that guarantee anti-alignment.
Local expansion at 0,.. We begin by establishing first-order expansions of both gradients. The following lemma isolates the sampler-induced bias terms that drive anti-alignment.
Lemma B.3 (First-order expansions of real and synthetic gradients). Let 0, 0 + with || small and assume (A4). Then
ra = VaRaata (0) Has + ((Ielli))
(B.4)
and, with
b = Bo [00. (X)],
A = E Eqo. Ha- (X)] - Bpam He- (X)].
rs = VoRay((0)
= Has + (b+ ((11)
(B.5)
=:B
Proof. The proof proceeds by Taylor-expanding the per-example gradient do around the optimum 0* then taking expectations under the real and synthetic distributions.
First-order expansion of the per-example gradient. By (A2) (regularity) and a first-order Taylor expansion 0*,
(x)(x)) (ho- (I)) Ho- (x) p(x). where the remainder satisfies Eppn[1p((X)] = O([=])?r,) and similarly (llexxll
Real-risk gradient. Taking expectation under Pdata and using (A1)-(A3),
ra Epp.nooo (X)] E
pase [dg- (X)
+ +EEu. [Ho- (X)] + E (p)o)((())) HaE + O(1:)ir).
=0
The first term vanishes by optimality (A1), leaving only the Hessian-weighted error. error.
Synthetic-risk gradient. Taking expectation under ge..x.
Ts = Egg.... (X))
LE Ege, -o (X)
+ E Ey Ho (X)] E E [p(X))
Ha+A
25

---

Published as a conference paper at ICLR 2026
Hence
Ts b- (Ha+ +) + O(I
Crucially, the synthetic gradient has two extra terms compared r.z the linear bias b (because the sampler does not satisfy the optimality condition), and the curvature tilt Ae (because the sampler changes the Hessian).
Equivalent residual form used later. is convenient (and used in subsequent bounds) to rewrite this as
rs E RR + O(1 where R₂ (b + Ae). Both expressions are identical up the first-order terms, and the latter isolates the "useful" Has part from the sampler-induced mismatch Rx.
Geometric interpretation via angle and magnitudes. The lemma shows that rs and ra share common "signal" Has, but differ by the sampler-induced residual R. - (b + As). To understand when 8 < 0, we characterize the geometry of this residual. Define the whittened magnitudes
== ||6|| ||!|
== | MJ'
which measure the strength of the linear bias and curvature tilt, respectively. The key geometric quantity is the angle between the model error and the bias direction H b:
(E, H 6) Ha
E [--1]]]
(B.6)
|| Ha [ blIn
Equivalently, ip the Euclidean angle between H²/2 and Н 1/36. Recall that the whitened preconditioner is K = - -= PH2 with spectral bounds mI 3 K MI.
when does < We now express terms of the angle yp and the bias magnitudes. The following theorem provides an upper bound showing that < O whenever is obtuse and the model is sufficiently good.
Main geometric result:
Theorem B.4 (Directional upper bound for s). With 0,- = 0" + and ||e|| small,
((1 + m)) || mo || sr, cos +1+ O(Ieella.)
Consequently, a sufficient condition for < is
||
170
(-cos cos s) with cos yp 0.
M(1 m)
Proof. We compute (ra, Pr.) using the local expansions from Lemma B.3, then bound each term by whitening to Fuclidean geometry.
Using Lemma B.3, write
== HaPH E HqPb = HHAPA + O(IE=I)aa)
Whiten with a 11//2 e, b b H 1 Â = 1/ AH 1/2
and K 11 p/ to get
a Ka a Kか a d'Kå + O(1aal)).
Now bound the three pieces using the spectral bounds on K: Kйa M Mm m ||
Ta I M||all2 2 M|
For the linear term, write KT || || cos 0, with the angle between K1/2 and K1/25. Since 1/2 2 vm|x=11
Kか 2 ||a||2 bl|2 [cos 0 m ||e||| Hr, no [cos sp+
Thus - TT < m: ||,]|и, [cos 4 Since [oos yp]+ 2 0 and | cos vp) > [cos sp]-, we can replace - -(cos yp) by the slightly looser but sign-robust term -[ cos 4+ yielding the stated bound after collecting terms and absorbing ({]aa]))
Corollary B.5 (Natural- -Natural-gradient geometry). P HT then K I (so M and < (1+ m) [|=||? no |E cos + 0(11=111..)
Thus it suffices that ||||| 1 m
770 (-
(-cos cos with cos < 0 to guarantee s < 0.
26

---

Published as a conference paper at ICLR 2026
Interpretation: why obtuse angles guarantee anti-alignment. Theorem B.4 reveals the structure of s as balance between two competing forces:
Quadratic terms (M (1 + m))l==ll ): Always positive, these tend to make > O.
Linear term (-mno |=|| I cos x]+); Negative when cos ip O, this drives negative.
The key insight is that oo captures the sampler's linear bias (whitened by Ha), while 7/ measures its curvature tilt. From Theorem B.4, the leading terms obey
x (( + 7)) l24 - m T/ | (-cos
so whenever the angle is obtuse (cos yp < 0, i.e., H points mostly against e), the subtractive linear term eventually dominates as || + O. Equivalently: there exists threshold Eo 0 (depending on m, M, 7o: 7/ and cos sp) such that if the model is sufficiently close to optimal, || < 6o. then 0. In this small-eroor regime, Neon reduces the real-data risk by Theorem B.1.
B.6 ACULE-ANGLE CONDITIONS THAT IMPLY < 0 (AR MODELS)
We now verify that the geometric condition cos ip holds for autoregressive models under standard sampling procedures. The key insight that common inference methods methods-temperalee temperature scaling, top-k, and top-p sampling-are all mode-seeking' they systematically reweight the model distribution to favor high-probability regions. We prove that this mode-secking behavior guarantees the obtuse angle condition, and hence anti-alignment.
Loss and geometry (AR). For autoregressive (AR) models negative log- ligllik:
(x log log
Φ0(x=) Volo(x) (1))
the data Hessian is the Fisher, Ha F= Eyas (ug- For sampler & let
b = Eg 00- (X)l = -- - ug- (X))
Our global angle is
(s, F-1b)e
008 y = |||||r r | F-bbll
E 11-1,1],
so anti-alignment corresponds to cos < O.
Definition (mode-seeking samplers). Fix 0, = 0" + E. We call mode-seeking if it is monotone
reweighting of the reference model:
((x) w(x) pa, (x),
w(x) [loggo,(x))
with f - R>0 nondecreasing and not a.e. constant. (For AR decoding applied tokenwise, the overall sequence law inherits product of such nondecreasing reweights; we write it as floog Pe, (x)) for brevity.)
Intuitively, nondecreasing f means that higher log- probaabbbyyy samples receive higher weight, thus concentrating mass on modes. The "not a.e. constant" condition excludes the trivial neutral case where Po..
Common AR samplers are mode-secking. All widely-used inference procedures fall into this category:
1/7 Temperature < 1. The sampler draws from & Pa. so f(2) exp( 2} with 1/r-12 0, hence is strictly increasing (neutral only at 7 1).
Top-k. Keep only the k largest probabilities: there exists threshold 2 such that f(2) K 2x). nondecreasing step function (neutral only at & vocabulary size).
Top-p (nucleus). Keep the smallest set whose cumulative mass exceeds p; this induces a (context- dependent) threshold 2, and f(z) {{ }p}. again nondecreasing (neutral only p 1).
27

---

Published as a conference paper at ICLR 2026
Main result: mode-seeking guarantees anti-alignment. We now prove that any mode-secking sampler induces the desired obtuse angle. The proof uses a monotone covariance inequality: when two random variables are both nondecreasing functions of common variable, their covariance is non-negative.
Lemma B.6 (Mode-seeking = cos yp < 0 (first order)). Assume g(x) x f(logoo, (x)) po, po. (x) with nondecreasing. For 0,- = 6t + e and small ||
cos u < + O(1[=]p))
Proof. The proof proceeds in three steps: relate cos yp the expectation of scalar function (ax) under the sampler, (ii) show that this expectation non-negative by a monotone covariance argument, and (iii) conclude that cos i O.
Step 1: Expressing the angle via a scalar projection. Let B(x) == ugg (z)) Then
E Eps. [w B
(E, F- (-'Byluo-]) 1) = =ETB,[uge (X)] = [x(](()]
Eye, [
Step 2: The weight function is nondecreasing in B. A first-order expansion around 0* gives log pa, (x) = log ps. logpe (x) + B(x) + O(1e=l?),
hence w(x) flog pe, (x) (z)) is first order) nondecreasing function of the scalar B(x). Replacing Po, by in both numerator and denominator incurs only ((I|=]])) relative error, so
Epg-
[w B
Bg B
+ (()=)))
Epg-
[w]
Cov (w, B) because Ep, [B] Epg. [ug-] = 0. Since and B are nondecreasing (as functions of B), the monotone-covariance inequality yields Cov pu- (w, B) 2 O, with strict > 0 unless w is a.e. constant or B is degenerate. Therefore [B] 2 0 to first order, i.e. (s, F F-EEE [uo- I 2 0 (up to ((
Step 3: Mooooooee covariance inequality. Now Ep-- B]
Step 4: Conclusion. Finally, 6 = [ug-] implics
(e, b-) (E, F F.FFFFEEELu (2g- 1)
cos || bblF
< 0 (strict < 0 generically),
|| F bllF
up to O([E]|e).
Interpretation and consequences. Lemma B.6 reveals why mode-seeking samplers are problem- atic for standard self-training but beneficial for Neon. By concentrating mass high- proobbbbbbyylityy regions, these samplers create a bias that points away from the model error e (obtuse angle). This anti-alignment makes the synthetic gradient a poor direction for improving the model-but by reversing this direction, Neon converts the degradation into improvement.
Combining Lemma B.6 with Theorem B.4 yields safety window for the merge weight w follows by substituting Н. = F into Theorem B.4. Since temperature 1, top-k, and top-p sampling are ubiquitous in practice, this result establishes that Neon applies universally to autoregressive models.
for sufficiently small ||e]|r. The explicit
B.7 ACULE-ANGLE CONDITIONS THAT IMPLY < O (DIFFUSION & FLOW)
We now establish the geometric condition cos 0 for diffusion and flow matching models. Unlike autoregressive models where mode-seeking arises from explicit inference-time modifications (temperature, truncation), diffusion and flow models exhibit mode-seeking behavior even with neutral inference settings. The key insight is that any practical finite-step ODE solver introduces discretization enror that systematically favors high-density regions. This makes Neon universally applicable to continuous generative models, regardless of guidance sampling choices.
28

---

Published as a conference paper at ICLR 2026
Loss and geometry.
We use standard pathwise quadratic losses. For diffusion score models,
(a) ! w(()) Ep, ||s0 (Xe,) - -= (xt,, dl,
and for flow matching.
Boo (0) / w(t) Epg (Xo, v² (xx:t) dt.
Let (0,,(x) = Volg (z) and Jz(x) == (000,r(x)) Define the pathwise Fisher
Path !. w(() B,, (x..)(()))((xx)) de,
and the angle (mirroring the AR case)
(c, "puti Aputh) Fish
Fug [o w(() Φ0- (Xx)
Ppath
||E|Lep.n || (pata)) Pan
Opuh
Anti-alignment corresponds to cos 4Ppath
< 0.
Finite-step ODE solvers are mode-seeking. We now show that discretization error in numerical ODE solvers creates systematic mode-seeking bias. The mechanism is subtle but universal: finite step sizes cause the integrator to overweight trajectories with stronger local contraction, and these high-contraction regions coincide with high-density modes.
Consider the probability-flow ODE with velocity / Rdd x [0, 1] Rd;; for diffusion, f(x, 1) - Va log (z). An explicit one-step scheme with step size hi gives
dzk-1
2k-1 2y + (xkk tx),
Jk
+ h (
dzk
Using tr log( + A) tr( tt((A²) + 0(11-11)) with A = h V (and tr(Aa) ||A| when v,f symmetric; otherwise take its symmetric part),
log det J4 htr(Vvf))
IV O(h²).
2
The key observation is the second term: the excess contraction - A22
/ depends on the local
T
curvature of the flow. Chaining steps and comparing to the exact ODE yields terminal reweight of the reference law:
((r)) exp($ +o pa. (r0);
((ra)
- F Σ
V
( 1
Xo
=x= T
mo}.
1/h.
For diffusion,
[(x, 1) -1) v (xr so that Vas(x; (1)2 V logre(x). hence
C(xa) = E Σ (tx))* IV logpr. (X ) Xu B0
To establish mode- sede-seeking behavior, we need to show that ((x)) increases with density. This requires a mild coupling assumption between curvature and probability:
Assumption (A-MONO: pervature-densinn coupling). The map 3o ((ro) is weakly increasing in log Po, (zo); i.e., log Po, (zo) S log Po. (x then ((a))= I <O(o))
Intuition behind A-MONO. Finite-step integrators overweight trajectories with stronger contrac- tion (large V.ffl). Near modes, log Pt is more curved (the Hessian has larger eigenvalues), so the flow exhibits stronger contraction. Thus C(xa) grows with local density. Importantly, as h- 0, the bias vanishes and - Po. (neutral). This means the mode-seeking behavior is purely an artifact of discretization,
inherent property of the continuous dynamics.
Remark B.7 (Step-size scaling). From log det J/ = h htrr 4 I + O(h²), the per- A²
step excess contraction is 6₁ = 2 fl? + Ohh3). Summing over x 1/h steps yields the terminal reweight exponent Ex 5 ((x)) + ohh). Consequently, the pathwise linear bias bpath L. (t) Φg- (Xx) dt] obeys |16pethl| | = (h). and the curvature tilt || lop = O(h).
FM
Both vanish linearly h O. making the sampler neutral in the limit.
29

---

Published as a conference paper at ICLR 2026
Flow matching. The same discretization mechanism applies to flow matching models. For updates Ik-1 = Ik + 20 20( (Tk: tx),
h? tu tu((vzu)))) + O(h)).
log det JR httr(V
so 67 = 2 V + 0 2 0 and the same reweight form emerges. With the flow analogue of A-MONO (the conditional expectation of Σ / increasing in log Po, (zo)), finite-step flow solvers are likewise mode-seeking.
Classifier-free guidance (CFG) is mode-seeking. CFG amplifies the meek-ng effect by explicitly increasing the guidance toward high-density regions. CFG modifies the diffusion velocity via a guided score
Byy (x, Suncond (2, + ((Soond (x), (z, t) Suncond (I, ()),
7 0,
so the probability- flow velocity becomes (I, = -t()) s, S, (x, t). Repeating the derivation above with f fy yields the same reweight form
4- (2o)
exp
22 (zo) o(1i)) (9..((00))
where Pa..y the guided reference law and
Cy(oa) E
Σ (X
Xo 20
Because Vury -02 Suncund
V (Scond Sumcond)).
IVa = / 27 (Va,, -o2 ( (Somd Suncond) Fr + 722 1l-102v (Scond Suncond
Fr
Near condition-relevant modes, the guidance term increases the magnitude (and contraction) of the flow, so C. (2n) is larger higher-density regions po...: this the same curvature-density coupling A-MONO, now for the guided dynamics. Hence finite-step CFG is mode-seeking in the sense above, and becomes neutral as h-
Summary and consequences. We have shown that diffusion and flow models exhibit mode-seeking behavior through three mechanisms:
Discretization error: Any finite-step ODE solver creates a systematic bias toward high-curvature (high-density) regions.
Flow matching: The same discretization mechanism applies to velocity- based formulations.
Classifier-free guidance: CFG amplifies the mode-seeking effect by explicitly strengthening the flow near modes.
Under the mild A-MONO assumption (curvature increases with density), all three mechanisms yield cos PPaath first order. Combined with Theorem B.4, this establishes that Neon reduces risk for diffusion and flow models under standard sampling procedures. Importantly, the bias scales ((h) and vanishes in the continuous-time limit, confirming that mode-seeking purely a discretization artifact.
B.8 NEIGHBOR MODELS: STABILITY AND UNIFORM NEON IMPROVEMENT
The previous subsections established that Neon reduces risk when at a single base checkpoint 0,. In practice, we often have access to multiple nearby checkpoints-frrot different training stages, ensemble members, or even different architectures. natural question arises: can we use a single synthetic dataset and merge weight to improve all of them simultaneousty? We now show that anti- alignment is stable under small perturbations in parameter space, allowing single Neon configuration to safely improve entire neighborhood of models.
30

---

Published as a conference paper at ICLR 2026
Setup. Fix the synthetic sampler 90,,. generated once the reference 0, 0" + (so q is frozen).
Consider any neighbor checkpoint
On 0, + 0" (=+
| Hr small.
All quantities below (gradients, alignments) are evaluated at 8,. but the synthetic law remains q6..x- This asymmetry is key: generate synthetic data once and reuse it across neighbors.
Local expansions at neighbor. By the same first-order argument as in Appendix B.4, with En == +
O0() = En = 0(1ssn122.) r,(O,) = Hdsn + b + A A + O(I)))lla,,) (B.7) where Rs = b+ A e with b b= Byloor and A == Eog - Exe. [Jo-] (as Appendix B.5). Define x(0) = (ra(f)), (0).
Stability of anti-alignment. The following proposition shows that the alignment scalar 8 varies smoothly with the checkpoint location. This continuity guarantees that if s(t,) < then remains negative in a neighborhood around 0,.
Proposition B.8 (Alignment is locally Lipschitz in a neighborhood). Let K == PH)//2 with ml 3 K 3 MI, and let no := [[b|l 1h := llop WJ¹ There exist constants C1, C2 (depending only on M, Noo o and the local regularity from (A2)) such that, for all sufficiently small [|
((O.) s(0,) & Ci CIlllll + To + 1) || + C2(11=lla + 11 16
In particular, s(.) is continuous at O, and varies most linearly with [|8]| to first order.
Sketch. Insert equation B.7 into s() = (ra, Pr,) and whiten with a == H2/2 d = -H2/2
b. = H 1/2 = H 1/2 A 1/2 K == H/// 1HH
1/2 to write (ef. Appendix B.5)
s(A)
Ka - a Tbb - T Kйa + 0(1la12)
and likewise with a + a a d at on Expanding saa + d) s(a) and bounding each term with
KK] lop = M, || S mh |bbl2 = no yields the stated linear-plus-quat quaaaaaaa-tic control in ||d||
|0|| Ha-
Corollary B.9 (Uniform anti-alignment in ball). Assume s(0,) S for some margin 0. Choose
p > 0 such that (I + + +1) + C2(1EE| + +1) p² S 2
Then s0) < p// < 0 for every neighbor with 110 0, |1 Sp
This corollary powerful: it shows that margin of anti-aliga anti-alignment at 0, guarantees anti-alignment throughout a ball of radius around 6,. The size of this safe zone depends on the margin μ and the local geometry controlled by C1, C2.C2.
Uniform Neon improvement for set of neighbors. Let C (0: - 0,1 S Sp} be any finite collection of neighbor checkpoints. Perform one short synthetic fine-tune at each E N (same frozen q) to obtain (0) f aPrr (0) O(a?). and define the Neon merge @ (0) (1+ w 0 (0).
The next thcorem shows that we can choose a single merge weight w that improves all neighbors simultaneously. rather than tuning separately for each checkpoint.
Theorem B.10 (Single w that safely improves all neighbors). Suppose s(0) < 0 for all 0 e N (e.g.. by Con. B.9). Assume either (i) Ha(() ( 2 0 for all A N. or (ii) uniform directional curvature bound holds:
d(0) R + rd(()) d(0)
Lair [|d(0))12 for all 0 N, [0,
where d(0)= (O). Let
Smin min s()) 0,
Qmax := max (0) PT Aa(()) (r) (or Lair [I(( under (ii)).
31

---

Published as a conference paper at ICLR 2026
Then any
2 Smin
w
a Qmax
guarantees Reata (ONoon Rdata (up (((wa))) for every 0 E ...
Proof. Apply the one-step expansion from Thm. B.I at each 0 E W and take the worst-case (most conservative) quadratic coefficient and the most negative linear term.
Practical implications. Theorem 10 has important practical consequences:
Remark B. (Practical takeaway). If a single base checkpoint 8, exhibits altinnlimmmnt with margin (negative s(0,)). then all sufficiently close neighbors inherit s()) < 0 and thus benefit from the same Neon recipe. In practice, one can either (a) choose single conservative w that safely improves an entire validation-selected pool of nearby models, or (b) tune w per checkpoint using its local s(0) and curvature proxy.
Remark B.12 (Cross-architectur transfer). The same frozen sampler go..s can safely improve nearby checkpoint from a different architecture, provided the two models are close in the data-risk geometry.
Concretely, let models (A) and (B) share the same per-example loss Eg and data, with H.(a) RaRaatt( and H(() = Rdataa their (population) Hessians at the same minimizer 8".
Generate ge,.. once at reference o!A²) for model (A), and consider neighbor / for model (B).
the Hessians are spectrally close and their norms are equivalent on the relevant subspace, i.e. there exist 0 < c c C < 00 and a small <> 0 such that
c <||0|| H | s ([ and ||
(B) ( I1.p. (H!*)) U.S.
and the sampler-induced terms are close,
[B 10 (anga) |A AWll| (11(0)- S S,
then the alignment scalar & transfers continuously:
(B) ((A)()(A))
S
O())
O ((A) (1
cross-arch mismatch
neighbor shift
Hence, if (OA) S - < 0 with margin and the cross-architecture mismatch and neighbor distance are small enough, then (B8) (0(5)) remains negative. In turn, Thm. B.10 provides a single merge weight w that (to second order) reduces Rdata simultancously for the (A) and (B) neighbors. Practically, using a common preconditioner P defined in a data-geometry (e.g., an empirical Ha estimate) further stabilizes cross arohitecchte transfer.
Summary. This subsection establishes the stability of Ncon's improvements. Anti- aliimlligmmnn at a single checkpoint implies anti-alignment in neighborhood, and single merge weight safely improve all nearby models. This robustness extends even across architectures when models are close in the data-risk geometry. These results explain why Neon's 's hyperparameters transfer well in practice: the synthetic dataset and merge weight found for one checkpoint often work for many others without retuning.
B.9 WHEN SELF-TRAINING HELPS
We have established that Neon reduces risk when s < O. and that all standard inference proce- dures-temperrature 7 top-k, top-p, finite-step ODE solvers-are mode-secking and thus guarantee s < 0. This naturally raises the question: are there settings where > 0? If so, what happens to Neon and standard self-training in these regimes? We now complete the theory by characterizing the opposite case: iiversity-seeking samplers. We show that these rare configurations reverse the sign of s, making standard self trainini beneficial while Neon's 's negative extrapolation would harm performance. This section thus delineates the boundary of Neon's applicability.
32

---

Published as a conference paper at ICLR 2026
First-order effeet of self-training. A short synthetic fine-tune takes the step 0, 0, -aPrs O(a²). The corresponding first-order change in real-data risk
Rdata (O,) - RRdata(O,) = (rd, Pro) + + - as + 0
Thus self-training helps (decreases Rdata) when 0. This is the opposite of the Neon regime, where 8 < makes negative extrapolation beneficial.
lower bound on s. To understand when s > O, we need a lower bound that complements the upper bound from Theorem B.4. The following theorem provides this by reversing the inequalities in the proof.
Theorem B.13 (Directional lower bound for s). For 0, = 0 + with || small,
8 2 (m - MM m ||ell - no ||e|| |- cos 4 + Ollell
Proof. All (+) are || m. From the local expansions.
= T HaPHas HaPb E HAPPE + O(1l=1i,a.
Whiten with a H2/22, b H² 1/2,, Å H1a 1/2 AH1 and K = H2222 H122PHH 2H2 to obtain
= a T Ka T Kk- - T Tā + O(1aa)))
Ka Lower bound each term (reversing the inequalities from the upper bound proof): (i) Kaa 2 m ||ali = m I|e (ii) Write T _u = || || cos with the Euclidean angle between 1/2 and K Then
-a Kō 2 - 11 aall || || [- cos 0]+ 2 - M |all2 |1612 [- cos w|
VM and identify yp (the Hz-angle between E and H b) with 0 up to whitening. This gives -a T Kō 2 M no || [- cos s/++. (iii) a Kйa
where used ||/222||||
- -|| llop ||op 2 - .M m ||=ll?r. Combine (i) (i)-(ii) and absorb ((Iaa
Corollary B.14 NNturallgradien geometry). If PP= HTT then K (so m= and
z (1 m |ell| No || [- cos s0 O(1e)))a.)
Diversity-seeking samplers reverse the angle. We now identify the class of samplers that make O. These are precisely the samplers that reduce probability mass on modes rather than concentrating it.
We say q is diversity-seeking if q(x) x f(log Pe, (z)) pa, (z) with f nonincreasing and not a.e. constant. This the opposite of mode-seeking: lower oog-probability samples receive higher weight, spreading mass away from peaks.
Lemma B.15 DDieersiy-seekinn = cos 2 0 (first order)). In the NLL specialization (ōe = H. b -E, (uo }). if is nonincreasing then, for 0, 8" + E and small ||=||r,
cos & > =)
Proof. The proof mirrors that of Lemma B.6 but with reversed monotonicity.
Let B(x) = ε ug- "gg(x)). (x). As in Appendix B.6, logpe, (x) = lo- pa- (x) + B(x) + O[[ so
w(x) floog po, (x) is (to first order) nonincreasing function of ((x). Replacing Pa, by pe-
in [B] Eyg. [wB]
Eyg- [w O (2.).
incurs only O([[=l|)) relative error, hence E. [
Epg. [w]
Eyg- [
Monotone covariance with opposite monotonicities gives Cov (w. B) S 0: since Ep. [B] 0, we
have Epg- Bw < O, so E, [B] < to first order. Therefore (E, F E. [u==]) [B[ < 0, and
with b - EE [ ] we obtain i
(e. p0p--1b-- 2 up O([e[l1)).
T
F
33

---

Published as a conference paper at ICLR 2026
When self-training helps: combining the bounds. Putting together the lower bound (Theo- rem B.13) and the angle reversal (Lemma B.15), we obtain the main characterization of the self- training regime.
Proposition B.16 (Self-training helps near good models under diversity seeking). Suppose f is nonincreasing (diversity seeking) so that Lemma B.15 gives cos yp 0 to first order. Then, for sufficiently small ||el], and 77 < m/M,
2 (m - Mm) || + O(I > 0,
and the self-training step 0, p, 0g = 0, - a ar, decreases Rattaa to first order. In the natural- natural-gradient case (P- H suffices that !/1 < 1.
Interpretation: the role of angle geometry. The lower bound in Theorem B.13 reveals a symmetric structure to the upper bound:
(lower) Z (m Mm)llelli. Mno | cos vp++ vs.
(upper) I M(1 mi Il meno |e| H2. cos yp]+
Both have the form "quadratic minus linear", but the coefficients differ. The key insight is:
When cos y < 0 (mode-secking): The linear term -mmol/e|| HH,( cos is negative, the upper bound is dominated by the linear term and < O.
When cos &p 2 (diversity-seeiing): The linear term vanishes, so the lower bound is dominated by the quadratic term (m Mnn || > 0, hence > O.
This geometric dichotomy completely characterizes when Neon helps (s < 0) versus when standard self-training helps (s > 0). The angle yp between the model error and sampler bias is the decisive quantity.
Examples of diversity-seeking samplers. they do exist:
While diversity-seeking samplers are rare in practice,
High temperature in AR 1): q x Po. (f(=)=e(\/r))) 10/--1)= is nonincreasing) diversity- seeking, cos 2 0 to first order.
Procedures that downweight peaks and upweight tails (c.g., sampling after complementary filtering of top-p mass) nonincreasing transforms of log po,.; the same conclusion applies.
Anti-mode truncations:
In these cases, the Neon formula NNo. (1+ w w)0, w0, still applics, but the optimal merge weight satisfies -1 < w < O (interpolation) rather than w > 0 (extrapolation). Equivalently, one should use standard self-training rather than Neon's reversal.
Summary. This section completes the theoretical picture by identifying the complementary regime where self-training helps. Diversity-seeking samplers reverse the geometric angle (cos yp > 0), which in turn reverses the sign of to positive. In this regime, moving toward the synthetic gradient (standard self-training) reduces risk, while Neon's negative extrapolation would increase it. Crucially, this is not a "failure" of the Neon framework-the mathematics still holds, but it prescribes a different sign for the merge weight. The complete theory thus provides unified understanding: the sign of cos şo determines the sign of optimal w. and standard inference procedures universally yield cos y < O. making Neon the right choice in practice.
B.10 NOTES ON FINITE SYNTHETIC SET AND EFFECT OF SHORT FINE-TUNING
The theory developed so far assumes access to the population synthetic gradient- that is, an infinite pool of synthetic samples. practice, we generate fixed finite synthetic set and perform a brief fine-tuning run before merging. This introduces two sources of approximation error: (i) finite-sample variance in the synthetic gradient estimate, and (ii) curvature bias from the finite-step optimization trajectory. This subsection analyzes both effects, showing that short fine-tuming with moderate synthetic dataset size produces a reliable, low- variance estimate of the degradation direction. We also explain the empirically observed U-shaped performance curve function of [S].
34

---

Published as a conference paper at ICLR 2026
Setup. Fix synthetic dataset S {2x, H drawn once from 98..x and then kept fixed. Let g(z, () E IP the per-example gradient of the synthetic loss (with internal randomness Ç, e.g., diffusion time/noise), and
r (0) = - g(r;;))
[æ; 0) == (x, (:0)],
(0 :- Ex [{;
Short fine-tuning (FT) from 0 uses step size > 0, T steps, and positive-definite preconditioner P:
0 8, Pfk:
E g( k
(B.8)
where } are fresh draws each time the fixed examples are reused. Let O₂ :- Or and define the scaled displacement
02 0r oT E ER.
dr
Two sources of finite-sample error. practice:
There are two independent sources of approximation error in
Dataset error (finite |S]): at O,,
(0))) (0,);
Edata
with Edata == Cov Co
......... (1;; 8. This is familiar statistical error from having a finite sample rather than the full population.
and irreducible unless grows. This is the
Monte Carlo (MC) error in timehooie: write
F2 = (0x) + Ex. EiG, 0x] 0, Cov(E& 0x) = Eme (0x)-
This captures stochasticity from internal randomness (e.g., time/noise sampling diffusion models). Unlike dataset error, MC eтror can be reduced by averaging over multiple forward passes through the same data.
Local smoothness. T E [0.
Let H.) = (or! Assume there exists Ldit 2 0 such that for all and
(S))) (0, + + (8) (0.) (0, ||= S Iidir 11011/
(B.9)
This is directional smoothness condition that controls how quickly the gradient changes along any direction. will allow us to bound the curvature bias from finite-step optimization.
Short fine-tuning produces low-bias estimate. The following lemma shows that if we keep o'T small (short tne-tuning), the displacement dy concentrates around the negative gradient direction with controlled bias.
Lemma B.17 FShort-FT displacement). Under equation B.8 and equation B.9, if aT S c/ for a small absolute constant then
dr
(S) r
(0.)
1
Σ EK
OaT (00)))))))))
7
The key observation that the bias term scales as OaTT), so keeping the total budget of small ensures the displacement accurately estimates the gradient direction.
Proposition B.18 (Direction concentration). Suppose >maas (0) S o² in neighborhood of 0,. Then for any unit vector
||I| o²
E
(u, dr + Pr Pr)(s) (8.))
+ C2 (aT)2,
T
where C' depends only on Lai., |PH (0,) and || (5) (1):()(()))) (0/)ll,. Hence, ifT + oc and aT + 0,
dr -. (S) (0,)- (S)))
35

---

Published as a conference paper at ICLR 2026
This proposition reveals the fundamental trade-off: the MC variance decreases 1/T (so larger T better), but the curvature bias increases as (aT)² (so larger af is worse). The sweet spot is to increase T while keeping small enough that aT stays small.
Practical guidance for learning rate and budget. Lemma B.17 and Proposition B.18 provide concrete guidance for choosing the fine-tuning hyperparameters.
The curvature bias dr scales like O(TT), while the MC variance shrinks like 1/T. Thus:
Decreasing a reduces bias (keeps the trajectory in the local linear region) but does not change the 1/TT variance term.
Increasing T averages MC noise but increases bias unless is reduced so that aT stays small.
A practical regime is
and large enough that PPllop VT
oT
$ Ldir
P
(l.)l22
This balances bias and variance optimally: the budget aT is small enough to stay the linear regime, while T' is large enough to average out MC noise.
Quadratic proxy for Neon and finite |S]. Let ra(O,) = (oReatt( and Ha = Radata ( Define
S5 = (ra(O+), P ())))
=s (r=))))(0))) pT Paaa
For the Neon merge @Neon (1 + w)0, wf, and short FT, the real-risk change admits the local expansion
AR(w) wa 85 + (wa)² 2s,
with minimizer and minimum
85
$
025 Ws
ARs ~
2 25
Using dr as a plug-in in estimate for -Pr (S) (0,),, set ST = (ra(0,), -dr) and 2 = (dr, Hadr). Then
TT as + OH(T-1/2 aT), 3T + Op( aT),
SO T ( concentrates on wg as T and aT 0.
Explaining the U-shaped performance curve. One of the striking empirical observations in the main paper (Figure 3) is that Neon's performance function of synthetic dataset size |8| exhibits U-shape: too few or too many samples both hurt, with an optimal intermediate size. The following remark explains this phenomenon.
Remark B. 19 (Why performance vs. [S] is U-shaped). Write (! = + Es with Or(n-1/2). Then
55 (Ta, Pr,) + (ra, PES), 2s =T=TJP HaPPr (cross/es terms).
For very small [S], variance dominates: 8s and as are noisy and the attainable improvement ARS 2 - is weak. For very large (S], variance vanishes (Es - -) 0) but the synthetic direction Pr, tends to align with high-curvature eigenvectors of Ha induced by mode-seeking samplers, increasing 2s faster than |ss] grows; consequently ARg shrinks slightly. A moderate [S] balances these effects: variance small enough to stabilize ss while the direction has not collapsed onto the sharpest curvature, keeping 2s moderate. This yields the empirically observed U-shaped curve in Neon performance as function of [S].
Summary. This subsection connects the idealized population theory to practical implementation. With fixed, finite synthetic set generated once, short fine-tuning (small a, modest T so that aT is small) produces a variance-reduced and reliable estimate the synthetic gradient direction (T( (0r), stabilizing the empirical coefficients (#s, =s) and the merge weight w. Very small [S] variance- variance-limited; very large [S] inflates As via curvature. Therefore, a broad, moderate |S] is typically best, confirmed by the U-shaped performance curve observed empirically. These results provide theoretical justification for the hyperparameter choices used in practice and explain why Neon is robust to reasonable variations in synthetic dataset size and fine-tuning budget.
36

---

Published as a conference paper at ICLR 2026
B.11 Toy EXPERIMENT
Now we present a toy experiment to empirically validate and provide deeper intuition for the theoretical results presented in the paper. The goal is create controlled environment where can directly observe the effects of sampler behavior on self-training and measure the key theoretical quantity: the directional alignment between gradients.
Setup. The task is to learn 2D Gaussian distribution, WHret Eeet), where the mean is Peet [0, O] and the covariance is Eter = [2, 1; 2] We use a small Denoising Diffusion Probabilistic Model (DDPM) with an MLP backbone, trained over a short diffusion process of 7 20 steps with a cosine noise schedule. base model, 0,, is trained for long duration (10, 000 epochs) on a small dataset of Nuss 10th real samples with learning rate of 10 to ensure it has converged.
To control the sampler's behavior during synthetic data generation, we introduce scalar hy- perparameter, Ç. which directly scales the model's score. The standard score is defined as (0 t) = co t)/ v/1 āt, where co is the model's noise prediction. During sampling, we use modified score, (xt, = Ç so t), to generate samples. This allows precisely control the sampler's characteristics:
1: The sampler becomes mode-seeking.
< 1: The sampler becomes diversity-seeking.
Ç 1: The sampler is neutral.
Experiment 1: FID vs. Merge Weight. In our first experiment, we validate the main prediction of our paper. We generate synthetic datasets using a mode-seeking sampler (ç 1.1) and a diversity- seeking sampler (ç 0.9). We then fine-tune 0, on each of these datasets to obtain a self-trained model 0g. We form a merged model via the one-paramett extrapolation formula:
0u = (1 + w)6, wf,- = 0 w(O, 0,)
0) corresponds Neon's negative extrapolation, moving away from the self-trained model. A negative weight (w < 0) corresponds to positive extrapolation (interpolation). Letting w -a for a O, the formula becomes 0.. = (1 a) at,, which is standard interpolation and equivalent to step of self-training.
positive weight
The results, shown Figure B.2, perfectly match our theory. For the mode-seeking sampler, the optimal FID is achieved at w= O, demonstrating that negative extrapolation (Neon) helps. Conversely, for the diversity-seeking sampler, the optimal FID is achieved w* O, showing that positive extrapolation (self-training) is beneficial.
HID 50 Epochs
vs. Epochs
0.0 (Diveritt-seekkin))
0.9 (Diveriny-sesking)
1.1 (Moxle-ookkngg)
1.1 (Modo-sekinn)
Loggo (FID)
to 1o (FID)
0.5
0.5
-0.3
U.S
Mergo Weight (w)
Merge Weight (w)
Figure B.2: FID vs. Merge Weight (w) validation. For the mode-seeking sampler (C = 1.1), the optimal FID is at w > 0 (Neon helps). For the diversity-secking sampler (ç 0.9), the optimum is at < 0 (self-training helps).
Experiment 2: Gradient Alignment vs. Sampler Type. In our second experiment, we directly measure the directional alignment between the real and synthetic gradients by computing their cosine
37

---

Published as a conference paper at ICLR 2026
('aaPaa""' We estimate the population real-data gradient ra and the similarity, cos(1) I| Padam Iraa Patan
Adam preconditioner PAdam from large set of Npop 10 real samples. We then sweep the score scale across the range [0.8, 1.25] and compute the cosine similarity for each value.
The results in Figure B.3 provide clear visualization of the alignment direction. The cosine similarity is positive for diversity-secking samplers (G < 1), corresponding to an acute angle between the gradients. This confirms they aligned, and self-training should help. The similarity becomes negative for mode-seeking samplers (C > 1), corresponding to an obtuse angle. This confirms they anti-aligned, and negative extrapolation (Neon) is the correct approach. Furthermore, we note that at the neutral point Ç 1, the cosine similarity is still negative. This provides a powerful validation of theoretical finding (Appendix B.7) that any practical, finite-step ODE solver-which our DDPM sampler is an instance of in--rouuodues small discretization error that is inherently mode-secking, thus producing a negative alignment even without explicit score scaling.
Gradient Cosine Similarity vs. Sampler Type
Cosine Similarity cosd (P)
0.0
0.8
0.9
1.1
1.2
Sampler ({)
Figure B.3: Direct measurement of the gradient alignment direction. The cosine similarity cos(1) (1)
is positive for diversity-seeking samplers (C 1) and negative for mode-seeking samplers (ç >> 1), crossing zero the neutral point (
38

---

Published as a conference paper at ICLR 2026
C EXPERIMENTS DETAILS
key advantage of Neon is its implementation simplicity, Given an existing training and generation script for base model, Neon requires only minimal add script that takes model checkpoints and weight w to construct the final model parameters. To ensure reproducibility and build directly on prior work, all our experiments start from official public codebases and use publicly available pre trained checkpoints as our base models. The repositories we used for each model family are listed below:
Diffusion Models (EDM): NVlabs/edm
Flow Matching: atong(1/conditioaa / -oaammhhinn
Autoregressive Models (VAR, xAR): Foundation Vision/VAR and OliverRensu/xAR
Few Step Models (IMM): lumalabs/imm
For the fine tuning stage, we adhere closely the default training configurations proposed by the original authors for each model. Our primary modification involves adapting the learning rate policy for the fine tuning context. This typically means using small target learning rate, which in some cases is reached via a linear warmup schedule. All other settings, such as the optimizer and batch size, remain unchanged. During this process, we save model checkpoints periodically (typically every 250k or 500k images seen) to evaluate performance over the course of training.
Our evaluation procedure is as follows. For each saved checkpoint, we perform a hyperparameter search to find the optimal merge weight w (and CFG scale where applicable). This search is conducted by generating 10k samples per setting to calculate a preliminary FID score. Once the optimal hyperparameters are identified, we generate final set of 50k samples to compute the final FID score reported in this paper.
Below, detail the specific configurations for each experiment.
EDM-VP on CIFAR-10.
S Generation:
Generated with -steps=18 -rho=7 -S -S_churn=0.
Default script of cond=1 -arch-ddpppp with modified -1r=1e-4 4. For the unconditional experiment, the script used -cond=0.
Fine tuning:
Neon Evaluation: Grid search over merge weight E [0,3.0]
EDM-VP on FFHQ-64.
S Generation:
Generated with -steps=40 -rho=7 -S_churn=0.
Fine tuning: Default script -cond=0 -arch=ddpmpp -batch=256 -cres=1,2, 2, -dropout=O_ 05 -augmen==0.1 with modified -1
Neon Evaluation: Grid search over merge weight w E 3.0).
Flow Matching on CIFAR-10.
S Generation: Generated using the dopri5 ODE solver with
Fine tuning: Default script of -ema decay=0 9999 with a modified learning rate of -1r-2e-4.
Neon Evaluation: Grid search over merge weight w E [0,3.0].
xAR-B B on ImageNet-256.
S Generation:
Generated with -cfg=2. -flow_stees==0 -num_iter=256.
Default script of -model=xar_ base -vae_embed_de _im dra__stride=1 with modified -b1r=1e-6, using a linear warmup schedule over the Mi images seen.
Fine tuning:
Neon Evaluation: Joint grid search over merge weight E [0,3.0] and CFG scale 7 [2.7, 5.0).
39

---

Published as a conference paper at ICLR 2026
xAR-L ImageNet-256.
S Generation:
Generated with -cfg=2.3 -flow_steps=50 -rum_iter=25.
Fine tuning: Default script of -model=xar_large
@nse_embed_diree
waa_sstrdd=116 with modified -blr=1e-6 6, using a linear warmup schedule over
the Mi images seen.
Neon Evaluation: Joint grid search over merge weight E [0,3.0] and CFG scale y E [2.3, 5.0].
VAR-d16 on ImageNet-226.
S Generation:
Generated
with -cfg=1.2 -top_k-900 -top_p-0...
model__epthesd
Fine tuning: Default script of -depth=16 =bs=786 -fp1 6=1 -alng=le-4, modified to use a linear warmup to a target learning rate of 1e-5 over 7.5 Mi images seen.
Neon Evaluation: Joint grid search over merge weight E [0, and CFG scale [1.25, 4.0].
VAR-d30 on ImageNet-512.
S Generation:
Generated
with -cfg=2.0 stop_k=900 -top_p=0.95
-model despilddepph1.....
Fine tuning: Default script of -depth=36 -bs=24 -fp16=1 -a1ng=5e-6 -saln=1 -pn=512, modified to use linear warmup to target learning rate of 1e-5 over 3 Mi images seen.
Neon Evaluation:
Joint grid search over merge weight E [0, and CFG scale 7 [2.0, 4.5].
IMM on ImageNet-256.
S Generation: Generated using the imagenet256_ts___ pkl model with -T-8 ofg_scale=1.5.
Fine tuning: Default training script with modified learning rate of - -1r-1e-6.
Neon Evaluation: For each T E {1,2, 4, 8), a joint grid search over w E [0, 5.0] and y e 3.0).
Metric Calculation Details. For the EDM and flow matching models, we used the official FID calculation script from the NVlabs/edm repository. The pre computed reference statistics were downloaded from the URL provided by the authors. For all autoregressive (xAR, VAR) and few step (IMM) models, we used the wwrchhffeeliie library. The reference statisties for ImageNet were sourced from the cennai/guided-iiiee Inception V3 features and computed the metrics using the prdd library with k
repository. For Precision and Recall, we extracted
Practical Note on Normalization Layers. The Neon merge, ONonn = (1 + w w w is applied directly to model parameters. The architectures in our experiments use LayerNorm. GroupNorm, RMSNorm; since these do not have running statistics, no special handling (c.g., recomputing statistics with a forward pass) required.
Practical Note Mask Buffers. The Neon merge applies only the learned parameters (0) of model. Architectures like xAR may use fixed buffers for attention masks containing infinity values. These buffers are not parameters and should be excluded from the merge. We follow the standard practice of copying all buffers directly from the base model 6,.
Practical Note on Numerical Precision. Some models use half precision (fp16 6). Performing the merge directly in fp1 using (1 + w)0,- w0, can cause numerical overflow. To ensure stability, we recommend one of two approaches:
1. Perform the merge in fp1 6 using the stable formula: 0, ww,, 0,)-
40

---

Published as a conference paper at ICLR 2026
2. Cast weights to higher precision (e.g., Σp32) before merging, then cast back to fp 6. We use the first approach in our implementation for its stability and efficiency.
Practical Note on Efficient Hyperparameter Search. While we performed a full grid search for thoroughness, more efficient search is possible in practice. The relationship between the merge weight w and FID strongly unimodal and locally quadratic. For finding an optimal w, one can use standard ID optimization algorithms like Brent's 's Method (Brent, 1973). For jointly optimizing and y, this extends to fitting a 2D quadratic surface, which we found requires only six well-distributed points find near-optimal configuration.
41

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS FOR DIFFUSION AND FLOW MATCHING MODELS
We extend the precision-recall analysis from Section 4.1 to additional diffusion and flow matching experiments. Figure D.1 presents the complete FID, precision, and recall curves as a function of merge weight w for EDM-VP on FFHQ-64 and Flow Matching on CIFAR-10.
For EDM-VP on FFHQ-64 (top row), we observe similar dynamics to those discussed in the main text. The FID curves exhibit the characteristic U-shape with optimal values around 1.0-1.5, achieving FID as low as 1.12 from a baseline of 2.39. The precision monotonically decreases with increasing w, dropping from approximately 0.78 0.40 as w increases from 0 3. The recall shows the expected inverted-U pattern, peaking near the FID-optimal weight and demonstrating that Neon's improvement stems from recovering under-represented modes. As the fine-tuning budget increases from 1.5 Mi to 3 Mi, the effects become more pronounced: the FID improvement deepens, the precision drop steepens, and the recall peak sharpens.
For Flow Matching CIFAR-10 (bottom row), the pattern is consistent but with model-specific characteristics. The baseline FID of 3.5 improves to 2.32 at optimal w R 1.0. The precision- trade-off is less extreme than for EDM-VE with precision declining from approximately 0.73 to 0.55 and recall peaking around 0.72. This suggests that flow matching models may have a different mode coverage profile compared to diffusion models, but still benefit from Neon's 's redistribution mechanism. The optimal merge weight remains relatively stable across different fine-tuning budgets, indicating robust degradation directions for this architecture.
EDM VPI PPFIQQQQ4 64
EDM VP IFFIIQ-64 64
EDM VPI PFIIFIQQ
0.8
0.8
Precisiom
0.75
FID
Recall
0.7
23B 3-1
0.65
0.6
Flow Matching CIFAR 10
Flow Matching CIFAR 10
Flow Matching CIFAR R-10
0.7
0.7
FID
Precision
0.65
Recall
0.85
2.211 11534
223
0.6
0.55
Figure D.1: Neon's precision-recall trade-off across diffusion and flow matching architectures. FID, precision, and recall as functions of merge weight w for EDM-VP on FFHQ-64 with [S] 18k (top row) and Flow Matching on CIFAR- 10 with S 25k (bottom row), shown across different fine-tuning budgets B. Both architectures exhibit the characteristic pattern: FID reaches a minimum at intermediate values, precision monotonically decreases, and recall follows inverted-U curve peaking near the FID optimum.
42

---

Published as a conference paper at ICLR 2026
E XAR-B ON IMAGENET-256 SYNTHESIZED IMAGES
Figure E.1: Images generated from Neon model with 4.25 (Mi), w 1.4, 3.8.SS| 750k,
FID 1.31
43

---

Published as a conference paper at ICLR 2026
F XAR-L ON IMAGENET-2555 256 SYNTHESIZED IMAGES
Figure F.1: Images generated from Neon model with 3.75 (Mi), 1.6, 7 2.7, |S|
FID 1.02
44

---

Published as a conference paper at ICLR 2026
G VAR-D16 ON IMAGENET-255 256 SYNTHESIZED IMAGES
Figure G.1: Images generated from Neon model with B 1.25 (Mi), 1, 2.9, |S] 750k,
FID 2.01
45

---

Published as a conference paper at ICLR 2026
H
IMM ON IMAGENET-255 256 SYNTHESIZED IMAGES
Figure H.1: Images generated from Neon model with B 1.95 3.6, 7 1.8, |S| 30k,
FID 1.45
46

---

Published as a conference paper at ICLR 2026
Figure H.2: Effect of negative extrapolation weight w and CFG scale on IMM generation quality for ImageNet class 980 (valley). Each 2x2 grid shows four random samples for given (w, 7) configuration for snapshot of model at B = 1.95 (Mi) for S = 30k.
Figure H.3: Effect negative extrapolation weight w and CFG scale on IMM generation quality
for ImageNet class 14 (indigo bunting). Each 2x2 grid shows four random samples for given (w, configuration for snapshot of model at B = 1.95 (Mi) for [S] 30k.
47

---

Published as a conference paper at ICLR 2026
Figure H.4: Effect of negative extrapolation weight w and CFG scale on IMM generation quality for ImageNet class 281 (tabby cat). Each 2x2 grid shows four random samples for a given (w, configuration for snapshot of model at B = 1.95 (Mi) for S = 30k.
Figure H.S: Effect of negative extrapolation weight w and CFG scale on IMM generation quality for ImageNet class 511 (container ship). Each 2x2 grid shows four random samples for given (w, configuration for snapshot of model at B = 1.95 (Mi) for [S] 30k.
48

---

Published as a conference paper at ICLR 2026
y=20
Figure H.6: Effect of negative extrapolation weight w and CFG scale on IMM generation quality for ImageNet class 928 (ice cream). Each 2x2 grid shows four random samples for a given (w, Y configuration for snapshot of model at B = 1.95 (Mi) for S = 30k.
H=op
Figure H.7: Effect of negative extrapolation weight w and CFG scale on IMM generation quality for ImageNet class 404 (airliner). Each 2x2 grid shows four random samples for given (w, configuration for snapshot of model at B = 1.95 (Mi) for [S] = 30k.
49

---

Published as a conference paper at ICLR 2026
1 VAR-D36-S ON IMAGENET-512 12 SYNTHESIZED IMAGES
Figure 1.1: Images generated from Neon model with B 1.20 (Mi), 0.6, 7 3.2, |S| 90k,
FID 1.69
50

---

Published as a conference paper at ICLR 2026
0.5
0-0
025
0.5
0.75
Figure 1.2: Effect of negative extrapolation weight and CFG scale VAR-d36-s generation quality for ImageNet class 609 (jeep). Each 2x2 grid shows four random samples for a given (w, 7) configuration for snapshot of model B 1.20 (Mi) for |S| 90k.
05
- 5.0
We 525
0.75
Figure L.:: Effect of negative extrapolation weight and CFG scale VAR-d36-s generation
quality for ImageNet class 113 (snail). Each 2x2 grid shows four random samples for a given (w, configuration for snapshot of model B 1.20 (Mi) for |S| 90k.
51

---

Published as a conference paper at ICLR 2026
0.5
0.5
0.75
Figure 1.4: Effect of negative extrapolation weight and CFG scale y VAR-d36-s generation quality for ImageNet class 248 (husky). Each 2x2 grid shows four random samples for given (w, 7) configuration for snapshot of model B 1.20 (Mi) for |S| 90k.
055
W- 525
W-OS
0.75
Figure L.5: Effect of negative extrapolation weight and CFG scale VAR-d36-s generation
quality for ImageNet class 510 (container ship). Each 2x2 grid shows four random samples for given (w, 7) configuration for snapshot of model B 1.20 (Mi) for [S] 90k.
52

---

Published as a conference paper at ICLR 2026
0.5
--0.0.
K: 025
0.75
Figure L6: Effect of negative extrapolation weight and CFG scale VAR-d36-s generation quality for ImageNet class 933 (cheeseburger). Each 2x2 grid shows four random samples for a given (w, 7) configuration for snapshot of model at 1.20 (Mi) for [S] 90k.
05
0.0
We 23
W
0.75
Figure I.7 Effect of negative extrapolation weight and CFG scale VAR-d36-s generation
quality for ImageNet class 933 (cheeseburger). Each 2x2 grid shows four random samples for a given (w, 7) configuration for snapshot of model at 1.20 (Mi) for S 90k.
53

---

Published as a conference paper at ICLR 2026
J EDM-VP ON FFHQ-64 SYNTHESIZED IMAGES
1:15
Figure J.1: Visual demonstration of Neon on EDM-VP (FFHQ-64) with merge formula O 0, w(O, 0, ). B = 1.5 (Mi), |S| 18k. Each column shows a different w value: negative w moves toward the degraded self-trained model (w -1: FID = 3.73), w recovers the base model (FID 2.39), and positive w performs negative extrapolation with optimal results at 2 (FID= = 1.12)
54

---

Published as a conference paper at ICLR 2026
K VISUALIZING NEON'S EFFECT ON INDIVIDUAL SAMPLES
To provide intuition for how Neon operates at the level of individual training samples, we conduct controlled experiment that isolates the effect of single synthetic image on the model's output. By isolating the effect of single synthetic sample, this visualization reveals how Neon's 's parameter-space extrapolation translates into coherent function-space changes, demonstrating that the method modifies semantically meaningful image features rather than introducing arbitrary noise.
Experimental setup. We begin with a base model 0, (VAR-d36) and generate single synthetic sample Jayn using fixed random seed. This sample serves two roles: (i) the sole training example for computing the synthetic gradient ræ. and (ii) as the reference image we will regenerate with different model weights to observe changes. Concretely, we:
Generate zsyn from 0, using seed 999 and CFG scale 7 1.5
Compute the synthetic gradient r, (0,) = VollI 0)le, averaged over 10 forward passes with different internal randomness (dropout, etc.)
Create Neon models: (w) 0, + w a rr(0,) for various merge weights w 0.,, 1.0, 1.5, 2.0}, with step size 10
4. Regenerate the same image using each Ncon (w) with the identical seed 999
5. Compute the pixel-wise change: As(w) [I[(s; 0,) - [(; ONeon (w)) where the norm is taken across RGB channels
Figure K.O shows the results across diverse ImageNet classes. For each class, the top row displays the regenerated image fs; (w)) at each merge weight (at = O, this is identical to Zsya), while bottom row shows pixel-wise difference map || (w=)ll normalized to [0, 1] (heatmap: black change, bright = large change). As increases, we observe semantically meaningful deviations from the original sample, confirming that Ncon's extrapolation operates in a structured manner rather than introducing random perturbations, adding simple realistic details to the generated image.
0.0
1.0
1.5
08
0.4 [A/
0.0
0.0
0.8
0.4 [A
0.0
55

---

Published as a conference paper at ICLR 2026
0.0
0.5
1.0
1.5
10
0.
[0]
0.4
0.
0.0
1.0
1.5
0.8
[4/
0.4
0.0
0.8
0.4
0.0
0.
0.4
0.
56

---

Published as a conference paper at ICLR 2026
W 0.0
Wi 0.5
W 1.0
1.5
W 2.0
10
0.6
0.2
0.0
W 0.5
W 1.0
w= 1.5
2.0
10
0.6
0.
w 0.0
W 0.5
W 110
W= 1.5
W 2.0
10
08
05
02
0.0
w 0.0
W 0.5
1=100
1=
W 2=0
1.0
08
06
0.4 [4/
02
0.
Figure K.O: Neon's 's effect on individual training samples across diverse ImageNet classes.
57


Reviews:
[{"review_id": "MTr5AwlSjR", "reviewer": "Reviewer_dZE2", "summary": "In this paper, the author proposed a new self-training algorithm for image generative models NEON. Specifically, NEON uses synthetic images generated by the generative model to finetune the model, and use the performance degredation as a learning signal. The key insight is that most inference samplers are mode-seeking, biasing samples towards high-density regions of the model distribution, resulting in model collapasing, and worsens FID. Consequently, NEON uses negative extrapolation between the reference model and briefly self-trained model, avoids mode collapsing. The author tested their NEON algorithm across different generative models and datasets and shows consistent FID improvement with little computational overhead.", "questions": "- Can the authors provide qualitative visual examples to illustrate how Neon changes the generative distribution? Are the observed gains primarily due to improved diversity (recall), or do they also enhance perceptual fidelity compared to naïve self-training—or possibly hurt fidelity due to the diversity–precision trade-off?\n- Can the author also include some baselines for the precsion/recall/FID analysis?\n- Do the authors see Neon as applicable to other domains such as NLP generation?\n- For figure 9, what is the synhetic dataset used samped from for the 30k model?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper introduces Neon, a simple but effective method that interprets self-training degradation as a useful signal for improvement. The approach is theoretically grounded and empirically validated across diffusion, flow, and autoregressive models.\n\n- The core idea of reversing the direction of self-training degradation is both simple and effective. The implementation requires only a single weight extrapolation step and no architectural or loss modifications, making the method easy to implement and apply.\n- Neon adds little computational overhead and scales well across model families. Its simplicity makes it readily to deploy in large-scale training pipelines.\n- The experiments are thorough, spanning multiple datasets (CIFAR-10, FFHQ, ImageNet-256/512) and model architectures. Across all cases, Neon consistently improves FID and recall. The recall and precision analysis provided nice support to the theory of Neon. It treats precision for recall, encouraging diveristy by negative extropolation. \n- Ablations show that Neon’s benefits generalize across architectures and even compensate for reduced real data availability.\n- The controlled toy experiment from the appendix effectively illustrate the theoretical predictions and help readers grasp the distinction between mode-seeking and diversity-seeking regimes.\n\nI did not verify all derivations in detail, but at an intuitive level the theoretical claims are consistent and well-motivated. I defer to other reviewers for a closer assessment of the mathematical rigor and proofs.", "weaknesses": "- The paper does not include quantitative comparisons against other self-training baselinesin the precision/recall/FID analyses. Even showing results for the simplest baseline of direct self-finetuning would help clarify how much of Neon’s gain comes from the negative extrapolation itself versus the self-training process. Moreover, comparisons in terms of data and compute efficiency would contextualize Neon’s benefits relative to prior self-improvement algorithms.\n- While the quantitative results are compelling, the paper would benefit from a few qualitative visual examples to illustrate how Neon changes the generative behavior. It remains a bit unclear whether the performance gains are purely distributional (i.e., improved diversity and recall) or also reflect higher fidelity and perceptual realism compared to naive self-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HymAAX64GS", "reviewer": "Reviewer_YZmD", "summary": "Usually training/retraining generative models on synthetic data can degrade performance metrics. On the opposite, this paper proposed a technique to leverage (bad) synthetic data from generative models, and improve generation. This work is in the direct line of work following Karras 2024 and Alemohammad 2024.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written, and the idea is clearly explained. Experiments are overall rather well presented. Figure 2 explains well the approach.", "weaknesses": "- I do not understand the novelty with respect to previous works.\nCould authors explain what is the difference between their work and [1] and mostly [2]? Especially, I really would like to better understand better the difference with [2]: the idea of using negative guidance from synthetic data is already in [2], in particular, Equation (1) of the proposed manuscript resembles line 4 in algorithm 1 of [2]. Are you doing SIMS, but in the parameter space? Could authors comment on that?\n- How significant do you consider the empirical results? Can you show the same plots on test set, with the Dinov2 embedding?\n- in particular I would be interested to see if the minimum value on the training set correlates with the minimum value on the test set\n\n\n[1] Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine.\nGuiding a diffusion model with a bad version of itself.\n\n[2] Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John Collomosse, and Richard Baraniuk. Self-improving diffusion models with synthetic data\n\n\nNon-scientific comment: I would remove the following quote, \"In the words of Martin Luther King, Jr., “Sometimes to move forward, we have to go backward.”, used to motivate negative parameter guidance. It does not feel appropriate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2sNI4GMr3s", "reviewer": "Reviewer_Srik", "summary": "The paper proposes NEON (Negative Extrapolation from self-traiNing): briefly self-train a generative model on its own samples to obtain degraded weights , then extrapolate backward from the degradation direction via a simple parameter merge . The authors give a theoretical account showing that common mode-seeking samplers create anti-alignment between synthetic and population gradients, so reversing the self-training direction reduces true data risk. Empirically, NEON improves diffusion, flow-matching, autoregressive, and few-step models on CIFAR-10, FFHQ, and ImageNet, often with <1% extra compute; notably it pushes xAR-L on ImageNet-256 from FID 1.28 → 1.02 with only 0.36% additional training compute.", "questions": "see weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Simple, general, post-hoc procedure that requires no extra real data, no auxiliary models, and no inference changes; just a short self-training run and a weight merge. \n\nClear theory: formalizes sampler-induced anti-alignment and gives conditions where negative extrapolation lowers population risk; also analyzes failure cases (diversity-seeking samplers). \n\nBroad empirical coverage across diffusion, flow matching, AR, and few-step models with consistent FID gains; includes precision/recall analysis explaining NEON’s recall-boosting mechanism. \n\nStrong results with tiny cost (often ≤2% of base training compute; sometimes as low as 0.36%), and improvements with as few as 1k synthetic samples. \n\nSOTA highlight: ImageNet-256 xAR-L FID 1.02, plus useful studies on (w, γ) co-tuning and cross-architecture transfer of the degradation signal.", "weaknesses": "Positioning vs. simple weight merges: needs stronger comparisons to generic weight interpolation/extrapolation baselines (e.g., linear checkpoint merges/SWA-style extrapolation) to isolate NEON’s specific benefit beyond “negative LR step”. \n\nBenchmark scope: focuses on standard class-conditional/unconditional image generation; lacks large-scale text-to-image or broader modalities, and relies mainly on FID + P/R without human eval. \n\nHyperparameter sensitivity: performance depends on w (and γ for AR/CFG); while grids are shown, guidance on automatic selection or stability across training checkpoints could be expanded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hFOEfSpugk", "reviewer": "Reviewer_BLA3", "summary": "The authors propose Neon, a self-training method that fine-tunes a base model using its own self-synthesized data with reversed gradient updates. Remarkably, Neon is effective even with as few as 1,000 synthetic samples. Its performance is validated across a wide range of architectures and datasets.", "questions": "* The key idea behind Neon—\"synthetic degradation and real-data improvement point in opposite directions\"—is intriguing. However, the current derivation of the supporting theorems appears loosely structured. Could you provide a clearer logical overview of how these derivations are organized? Additionally, how should this statement be interpreted intuitively in the context of model training? \n\n* In Figure 3, the FID increases as the self-training budget B grows. Does this suggest that extended training with Neon may lead to performance degradation? If so, does it challenge the assumption that s < 0 (negative gradient alignment) consistently holds?\n\n* In the images presented in Appendices E through I, are these generated samples from the Neon self-trained models, or are they the synthesized images used during the self-training process? Could you clarify this distinction and provide an analysis of the synthesized images used for self-training? Additionally, a visual interpretation explaining why these images are effective in contributing to the self-training process would be highly valuable.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Neon is simple to implement and effectively addresses common self-training challenges such as model autophagy disorder (MAD) and model collapse.\n\n* The method is evaluated across a wide range of architectures—including diffusion, flow-matching, autoregressive, and momentum-matching models—and diverse datasets such as ImageNet, CIFAR, and FFHQ.", "weaknesses": "* The effectiveness of Neon hinges on the assumption that s < 0 (negative gradient alignment). However, this condition may not consistently hold in practice, raising concerns about the robustness of the approach.\n\n* The hyperparameter configuration—particularly the training budget and negative extrapolation strength w —could introduce additional tuning complexity. Ablation studies in Figures 3 and 4 suggest that Neon is sensitive to these hyperparameters, which may impact its ease of deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)