Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)
Authors: Nikita Maksimovich Kornilov
David Li
Tikhon Mavrin
Aleksei Leonov
Nikita Gushchin
Evgeny Burnaev
Iaroslav Sergeevich Koshelev
Alexander Korotin
Venue: ICLR 2026 Oral (2026)
Keywords: Diffusion models
Flow Matching
Acceleration of diffusion/flow models
Distillation of diffusion/flow models
Primary area: generative models

Abstract: While achieving exceptional generative quality, modern diffusion, flow, and other matching models suffer from slow inference, as they require many steps of iterative generation. Recent distillation methods address this by training efficient one-step generators under the guidance of a pre-trained teacher model. However, these methods are often constrained to only one specific framework, e.g., only to diffusion or only to flow models. Furthermore, these methods are naturally data-free, and to benefit from the usage of real data, it is required to use an additional complex adversarial training with an extra discriminator model. In this paper, we present **RealUID**, a universal distillation framework for all matching models that seamlessly incorporates real data into the distillation procedure without GANs. Our **RealUID** approach offers a simple theoretical foundation that covers previous distillation methods for Flow Matching and Diffusion models, and is also extended to their modifications, such as Bridge Matching and Stochastic Interpolants. The code can be found in https://github.com/David-cripto/RealUID.

Published as a conference paper at ICLR 2026
UNIVERSAL INVERSE DISTILLATION FOR MATCHING MODELS WITH REAL-DATA SUPERVISION (No GANs)
Nikita Kornilov"
David Li"
Applied AI Institute, Moscow, Russia MIRAI, Moscow, Russia BRAIn Lab, Moscow, Russia jhomanik1 4 40 com
AI Foundation lab, Moscow, Russia MBZUAL, Abu Dhabi, UAE David wwt.miiiiibiiii.ie ac.
Tikhon Mavrin'
Aleksei Leonov
Applied AI Institute, Moscow, Russia tixonmavrin@@mai.. com
AI Foundation lab, Moscow, Russia MIRAI, Moscow, Russia
Nikita Gushehin
Evgeny Burnaev
Applied Al Institute, Moscow, Russia AXXX, Moscow, Russia
Applied Al Institute, Moscow, Russia AXXX, Moscow, Russia
Iaroslav Koshelev
Alexander Korotin
AI Foundation lab, Moscow, Russia
Applied AI Institute, Moscow, Russia AXXX, Moscow, Russia ama l aamaeexooooting com
ABSTRACT
While achieving exceptional generative quality, modern diffusion, flow, and other matching models suffer from slow inference, they require many steps of iterative generation. Recent distillation methods address this by training efficient one- step generators under the guidance of a pre-trained teacher model. However, these methods are often constrained to only one specific framework, c.g., only to diffusion or only to flow models. Furthermore, these methods are naturally data- free, and to benefit from the usage of real data, it is required to use an additional complex adversarial training with an extra discriminator model. this paper, we present RealUID, universal distillation framework for all matching models that seamlessly incorporates real data into the distillation procedure without GANs. Our RealUID approach offers simple theoretical foundation that covers previous distillation methods for Flow Matching and Diffusion models, and is also extended their modifications, such Bridge Matching and Stochastic Interpolants. The code can be found httpps. /ttp:t////t/a/ /tt.t/gtttbb////// hub. com ceteeerrrptoo/eaaiairraeaee RealUID.
INTRODUCTION
In generative modeling, the goal learn sample from complex data distributions (c.g., images), and two powerful paradigms for it are the diffusion models (DM) and the flow matching (FM) models. While they share common principles and are even equivalent under certain conditions (Holderrieth et al., 2024; Gao et al., 2025). they are typically studied separately. Diffusion models (Sohl-Dickstein et al., 2015: Ho et al., 2020; Song et al., 2021) transform data into noise through forward process and then learn reverse-time stochastic differential equation (SDE) to recover the data distribution. Training minimizes score-mater matching objectives, yielding unbiased estimates of intermediate scores. Sampling requires simulating the reverse dynamics, which is computationally heavy but delivers high-quality and diverse results. Flow Matching (Lipman et al., 2023; Liu, 2022)
"Equal contribution

---

Published as a conference paper at ICLR 2026
instead interpolates between source and target distributions by learning the vector field of ordinary differential equation (ODE). The field is estimated through unbiased conditional objectives, but the resulting ODE often has curved trajectories, making sampling costly due to expensive integration. Beyond these, Bridge Matching (Peluchetti, 2023; Liu et al., 2022) and Stochastic Interpolants (Albergo al., 2023) generalize the framework and naturally support data couplings, which are crucial for data-to-data translation. Since all of the above optimize conditional matching objectives recover an ODE/SDE for generation, we refer to them collectively as matching models.
Despite their success, matching models share a major drawback: sampling is slow, as generation requires integrating many steps of an SDE or ODE. To address this, range distillation techniques have been proposed to compress multi-step dynamics into efficient one-step or few-step generators. Although matching models follow a similar mathematical framework, many distillation works consider only one particular framework, e.g., only diffusion models (Zhou et al., 2024a;b), Flow Matching (Huang et al., 2024), or Bridge Matching (Gushchin et al., 2025). Furthermore, these distillation methods are data-free free by construction and cannot benefit from the utilization of real data without using additional GAN-based losses. Thus, the following problems remain:
1. Similar distillation techniques developed separately for similar matching models frameworks.
2. Absence of natural way to incorporate real data in distillation procedures (without GANs),
Contributions. In this paper, we address these issues and present the following main contributions:
1. We present the Universal Inverse Distillation with real data (RealUID) framework for matching models, including diffusion and flow matching models ($3) as well as Bridge Matching and Stochastic Interpolants (Appendix C). It unifies previously introduced Flow Generator Matching (FGM), Score Identity Distillation (SiD) and Inverse Bridge Matching Distillation (IBMD) meth- ods ($3.2) for flow, score and bridge matching models respectively, provides simple yet rigorous theoretical explanations based on a linearization technique, and reveals the connections between these methods and inverse optimization ($3.3).
2.
Our RealUID introduces novel and natural way to incorporate real data directly into the distilla- tion loss, eliminating the need for extra adversarial losses which require additional discriminator networks used in GANs from the previous works ($3.4).
2 BACKGROUNDS ON TRAINING AND DISTILLING MATCHING MODELS
We describe the Diffusion Models and Flow Matching frameworks ($2.1) and distillation methods for them (52.3). Then, we discuss how real data can be added to distilling methods via GANs ($2.4)
Preliminaries. We work on the D-dimensional Euclidean space R. This space is equipped with the standard scalar product (zr, v) El= Taya. the f2-norm ||=||| = V(r, æ and fz-distance ||z yll. Yz, y RDD We consider probability distributions from the set P(BP) of absolutely continuous distributions with finite variance and support on the whole R.D
2.1 DIFFUSION AND FLOW MODEL.
Diffusion models (Sohl- Diccktiet et al., 2015; Ho et al., 2020; Song et al., 2021) consider a forward noising process pt that gradually transforms clean data Po into a noise pr on the time interval [o, T]:
dx. = fc It dt + St dw,, To ~ Po:
where f; and Sr are dime-dependent scalars and wa is standard Wiener process. This process defines conditional distributions P. (1 (ro) =N Wa I)), where
ar = exp (L, Js ds
( exp (r ds
1/2
Sudu)
Each conditional distribution admits conditional score function, describing it: (xx(20)) == Vz, log (xre(zo)) - (ar 0..0)/0².
The reverse dynamics from the noise distribution pr to the data distribution Po is provided by the following reverse-ting time SDE with reverse-time Wiener process Wy:
dx (St It d8 (xe)))) + g.dwe,
2

---

Published as a conference paper at ICLR 2026
where (2(2)) Croppoo unconditional score function is learned via minimizing the denoising score matching (DSM) loss:
pot- (m.) (2 [au)] is the score function of (z = po) This
LssM s'', po) [0, [o, popp [>ells(((re) (
(1)
where "/r are some positive weights. The reverse dynamics admits a probability flow ODE (PF-ODE): dr, = (x)))d, ue(se) = (Se 3t - 98 sx(2))22),
which provides faster inference than the SDE formulation.
Flow Matching framework (Lipman et al., 2023; Liu et al., 2023) constructs the flow directly by learning the drift (xx)). Specifically, for each data point zo ~ Po. one defines conditional flow pe( (2o) with the corresponding conditional vector field u, (3r generating via ODE:
dx, ue
Then, to construct the flow between the data Py and noise pr. one needs to compute the unconditional
vector field (x,)) = Engr-po (- (12,)) [2, [zn)] which generates the flow (x;) [ (
can be done by minimizing the following Conditional Flow Matching (CFM) loss:
(e, po)
(.o--:..T...0mp...r-pp [llux(xe) (z (20) [b]]
(2)
In practice, the most popular choice the Gaussian conditional flows pt (x N Ma of For this conditional flow samples can be obtained as = 0,30 toe, NN I) and the conditional drift can be calculated as uy (2re = 6ç30 + dyc.
We recall data-to-data models working with data couplings, such as Bridge Matching and Stochas- tic Interpolants, in Appendices C.1 and C.2, respectively.
2.2 UNIVERSAL LOSS FOR MATCHING MODELS
From a mathematical point view, it was shown in (Holderrieth et al., 2024; Gao et al., 2025) that flow and diffusion models basically share the same loss structure. We recall this structure, but use our own notation. We call diffusion and flow models and their extensions as matching models.
A matching model constructs probability path pt the time interval [0, T), transforming the desired data po E P) to the noise pr E P(RRD). This path built as a mixture of simple conditional paths Pe (-(20)) conditioned on samples z Po. i.c., Pt (x+) .RO (o (1o) Yx, E D The path pt determines the function P [0, T RDD RD which recovers it (e.g.. score function or drift). The conditional paths also determine their own simple conditional functions f (-(ro) so that they express f (xx) |x (Iz| where Po(+|z+) is a data distribution po conditioned on sample at time t. Since f cannot be computed directly, it is approximated by a trainable function f [0. T| x DRD -> TRDD via minimizing the squared Ez-distance between the functions at each time t from [0, T] and point H, Pp:
ff = Ilf. - (2. (2r|=0)]112 (m/ & Ero-po (llf ( (2.12z))1
We also change the sampling order ~ Pt, xo ~ Po(-(wz) to more natural xn ~ po: 2t ~ pr(-1zo). Definition 1. We define Universal Matching (UM) loss LuM(f.PN that takes trainable functton f and distribution po E P(RP)) as arguments and upon minimization over f returns the function fro:
LUM(J, (po)) B Beo (p(l)))) (e(x)) ((( ( pro min {LuM(J S. Po). (3)
The notation t ~ [0. T) hides time sampling and loss weighting inherent to the given matching model.
2.3 DISTILLATION OF MATCHING-BASEE MODELS
To solve the long inference problem of matching models, a line of distillation approaches sharing similar principles was introduced: Score Identity Distillation (Zhou et al., 2024b;a, SiD), Flow Generator Matching (Huang et al., 2024, FGM), and Inverse Bridge Matching Distillation (Gushchin et al., 2025, IBMD), for diffusion, flow, and bridge matching models, respectively.
The SiD approach trains a student generator Go Z + R.P (parameterized by 0) that produces a distribution of from latent distribution 22 on 2. This approach minimizes the squared fy-distance
3

---

Published as a conference paper at ICLR 2026
between the known teacher score function s* arg min LUSM(s', (a0) real data pa and the unknown student score function s":
min a
(x]))] s.t. 8 arg min S L's (P))),
(4)
where p is the forward noising process for the generated data př- The authors propose the tractable loss with parameter asiD to approximate the real gradients (4)
(O) 10 B2 Go(=G))))) 1-21--222 2wr OrSiD (ssiD|ss (x]) -
- -016
(2))11
Bur (s7 2010 (x.) sx (x) s = arg min a LDSM s PO). (5) where wy are normalizing weights and gradients w.r.t. 9 are not calculated for the variables under stop-gradient sg[-] operator. The SiD pipeline is two alternating steps: first, refine the fake score sea(0) by minimizing DSM loss (I) new pl from the previous step. Then, update the generator Go using the gradient of (5) with frozen s* The asiD parameter is chosen from the range [0.5, 1.2), although theoretically only the value asD = 0.5 restores true gradient as we show in our paper.
The authors of FGM propose similar approach, but for the flow matching models. Specifically, they also use a generator Go to produce a distribution ph- but instead of DSM loss (1), consider CFM loss (2). The method minimizes the squared f2-distance between the student and teacher drifts:
min g ja, (0.t 2x, [
un (xx))) s.t. u arg min CCFM (v, P(),
(6)
where the interpolation path p is constructed between the noise pr and generator p distributions. To avoid the same problem of differentiating through arg min operator in SiD, the authors derive tractable loss whose gradients match those of (6):
(F) E0.10. G= |- || (x+) 2960 (2)))1
(22 un (0.00 (x ) uz (x]) nf (z |0)))], s.t. u arg min CCFM( PO)-
For data-to-data bridge matching models, the IBMD method applies the same idea of minimizing the difference between student and teacher drifts using similar loss. Notably, all these approaches (SiD, FGM, IBMD) are data-free, i.e., they do not use any real data from Po to train a generator.
2.4 GANS FOR REAL DATA INCORPORATION
Although FGM and SiD methods exhibit strong performance in one-step generation, the generator in these methods is trained under the guidance of the teacher model alone. This means the generator cannot get more information about the real data that the teacher has learned. For example, is not expected to correct the teacher's s errors. To address this, recent works (Yin et al., 2024a; Zhou et al., 2024a) propose adding real data via GANs (Goodfellow et al., 2014). such approaches, the encoder of fake model is typically augmented with an additional discriminator head D that distinguishes between the generated and real data noising processes via the following adversarial loss:
Ladv = 1e_ [Ex; [In D. (x:)] + Ezg-pp [In[1 - D.
The overall objective such hybrid frameworks (Zhou al., 2024a) consists of generator loss:
Las LEGMISID + A (0),
And fake modelldiscriminatoo loss:
CD LCFM/DSM Cad
Here, AGo and AD are weighting coefficients for the adversarial components. Despite empirical gains, the GAN augmentation entails nontrivial costs: it necessitates architectural modifications, such as an auxiliary discriminator head, and inherits the well-known optimization problems of adversarial training. such as stationary objectives, mode collapse, and sensitivity to training dynamics.
UNIVERSAL DISTILLATION OF MATCHING MODELS WITH REAL DATA
In this section, we present our novel RealUID approach for matching models enhanced by real data. First, we show that the previous data-free distillation methods can be unified under the single UID framework (63.1). Then, we describe how this framework is connected prior works ($3.2) and inverse optimization ($3.3). Using this intuition, we propose and discuss the real data modified UID framework (RealUII) with natural way incorporate real data without GANs ($3.4).
4

---

Published as a conference paper at ICLR 2026
3.1 UNIVERSAL INVERSE DISTILLATION
To learn a complex real data distribution Pû. one usually trains a teacher function f arg minn (U)( that is then used in multi-step sampling procedure (Def. 1). To avoid time-consuming sampling, one can train a simple student generator Gş : Z + IR" D with parameters 0 to reproduce the real data Pú from the distribution p on the latent space Z. The teacher function serves as a guide that shows how close the student distribution pe and the real data pe are. FGM and SiD methods ($2.3) train such generator via minimizing the squared fy-distance between the known teacher function f* and an unknown student function f == arg min, LuM(f. PP):
10, I8 P (1l.2 (x() (x [IP] 10,77 2 ~/ (1 (x Egg- 1/ (<g (=)))
= [o B pp f1 (1 1 2 EE, Bog -
(x)))
+ Bot mp ||| 11888-881111011 LF (= =0)1
not traciable
where pe is the probability path constructed between generated data p and noise pr. The problem is that the final term (9) cannot be calculated directly. This is because involves the math expectation inside the squared norm, unlike the other terms which are linear in the expectations. It means that a simple estimate of || (a] [=]|=])] using samples z6 and xf will be biased. Moreover, to differentiate through the math expectation inside the norm, an explicit dependence of p on 0 is required, while, in practice, usually only dependence of samples zff on is known.
Making loss tractable via linearization. To resolve this, we use the identity ||a|| max { -|111||2 + 2(b, a)}. Va IR.. For fixed time and point af we reformulate the squared norm (9) as this identity and parametrize vector b via an auxiliary function 5 [0, T] x RR - RD:
Eo [ll.ft (x:) f (=f)) {x(=1)
max BL- [=116((2))11 ((6)(2)), ft (x+) f (z?))]
-(-pp
x/-pp
max Be (x))))) de(mg)
(7 - (8x(())) Egg 1 U/ (x |20))>] (10)
Lincer and tractable
Now, all loss terms are linear and can be sampled. The parameterization 5 f-- with fake
function f [0, x R + RD allows us to get an elegant form:
(10)=man Bo e f((=g) of
Se -( (11)
= =max E 1, [1 (- 1o x 2-p
(2 (12)
fe(af) (
(
Summary. We build a universal distillation framework as single min-max optimization (13), implicitly minimizing squared Ey-distance between teacher and student functions. When real and generated probability paths match, these functions match as well, and the distance attains its minimum.
Theorem 1 (Real data generator minimizes UID loss). Let teacher f := arg min, LuM(f ff.(f. be the minimizer of UM loss (Def. I) on real data p E P(TRP). Then, real data generator Gg- S.I. pl Pū is a solution the min-max optimization of Universal Inverse Distillation (UID) loss Eu P() over fake function f and generator distribution pi:
min max & (f.u(f.p)) == Lfm .P%) - LuM(f (f.)))-
(13)
Lemma (UID loss minimizes squared fdddistacc.. Maximization of UID loss (13) over fake func- tion I represents the squared fy-distance between the student function fd == arg min, Lum(s, Pg) and the teacher f" = arg min, Lum (P.)))
max fLum (f. ( = (i.10o.TB222-0
(x
(14)
In UID framework, the trained fake model simply learns the current student function f by minimizing UM loss LfM(f. p('). Note that for points 2 out of the generator's 's domain p (xf) se 0, the distance (1 (14) vanishes, and the generator cannot receive feedback from the uncovered real data. Moreover, if the teacher function is inaccurate, the generator will learn it with all inaccuracies.
5

---

Published as a conference paper at ICLR 2026
Golal
HEI -0
Integolation
Goneraatrr
resdel
66
Figure I: Pipeline of our RealUID distillation framework ($3) with the direct incorporation of real data pi adjusted by parameters a, B E (0, 1]. The figure depicts flow matching models predicting denoised samples. It distills costly frozen teacher model f (blue) into onc-step generator Go (red) upon min-max optimization of Li (f po) loss over fake model f (green) and generator distribution pó with parameters 0. It updates the fake model several times per one generator update for stability. Algorithm's 's pseudocode is located in Appendix B.
3.2 RELATION TO PRIOR DISTILLATION WORKS
FGM and SiD approaches formulate distillation constraint minimization of generator loss subject the optimal fake model. For generator updates, the explicit UID loss (11) matches FGM loss (7) and SiD loss (5) with @ssiD 0.5. For fake model, it also minimizes the UM loss on the generated data. The work (Gushchin et al., 2025) was the first to formulate the distillation of bridge matching models in their IBMD framework as min-max optimization of the single loss (12).
Although previous works derive the same losses, we give new, simple explanation using lineariza- tion technique. This technique is more powerful and general for hundling intractable terms than complex proofs for concrete models from FGM, SiD, IBMD. It allows us to build other distillations, e.g., loss for minimizing the di-distance instead of the squared one (see Appendix A.4).
3.3 CONNECTION WITH INVERSE OPTIMIZATION
We derive UID loss (13) by minimizing the squared dy-distance between teacher and student functions. However, this loss admits another interpretation: its structure is typical for inverse optimization (Chan al., 2025). this framework, one considers a parametric family of optimization problems miny C(F,0) 0) with objective loss L(f, 0) depending on argument f and parameters 0. The goal is to find the parameters 8* that yield a known, desired solution f arg miny [(f, 0*). One standard way to recover the required parameters is solve the same min-mas problem (13):
ming max {C( 0) C(f.)) ~ min of (C("" ()) min {[((
(15)
The inverse problem (15) always has minimum which is attained when 0"
Although the inverse optimization can handle arbitrary losses L. it does not describe the properties of the optimized functions or how to find solutions. In our case, we show that all losses are tractable and minimize the distances between teacher and student functions (Lemmas and 2).
3.4 REALUID: NATURAL APPROACII FOR REAL DATA INCORPORATION
Previous distillation methods add real data during training only via GANs with extra discriminator and adversarial loss. We propose a simpler, more natural way that requires extra models or losses.
Based on intuition from inverse optimization ($3.3), we see that the min-max inverse problem (15) compatible with other losses. This allows us to redesign the UM loss (3) to incorporate real data into it. A key constraint that the loss must still yield the same teacher upon minimization on the real data. Thus, derive novel Unified Matching loss with real data weighted sum of two UM-like losses on generated and real data parameterized by a, 3 E (0, 1] which control the weights.

---

Published as a conference paper at ICLR 2026
We define Universal Matching loss with real data (RealUM) Cg. CR.uM( (f.()) that is parametrized by a, (0.1) and takes trainable function and generated data ph as arguments:
Definition 2.
CR. Cf = a [0 B ~ x (-10)
[1
_ a f (2)1200)
generated data pa term
+ a1- (a po ppt (lar))
f1 (xjlax) )1 (16)
real data 2ig term
For 1, we consider only 3 == 1, i.e., the pure generated data term.
RealUM loss (16) for all , and UM loss (3) yield the same teacher when input is real data Po- since if we consider only the f-dependent terms in the losses, we have:
Cg.9 (.. x Be,, m [0 + all (I)(() + 2(a -
X a m; [1]{(())) 17 (1)1=)))1
arg min y RM (s p6) arg min , (uM( (S. 5
Hence, the min-max inverse scheme (15) with RealUM loss and the old teacher ʃ* still has a real data generator as a solution, but now real data is incorporated via the real data terms of CR. P8):
min o (CR. .P) min
CR. CR- .P)) min /( (6))))
20
-Camn "f .PG)
Theorem 2 (Real data generator minimizes RealUID loss). Let teacher f== arg min C (P)) be the minimizer of UM loss (Def. 1) real data P PP.P). Then, real data generator Gg- s.I. pe Ph is a solution to the min-max optimization Universal Inverse Distillation loss with real data (RealUID) Cg. (f p6) over fake function f andd generator distribution ph (see Def. 2):
min @ max { (f (f.P%) == L .P) CR. (..
(17)
We provide analysis of RealUID in Appendix 1. below we highlight the most important findings.
Role of coefficients o. 3. Our RealUID uses real data only to minimize C ( .Q) function f. Thus, the trained fake function memorizes both the real data and the generator's 's current state. In turn, the generator is influenced by the real data indirectly, only via this fake function. As shown in Lemma 2, RealUID implicitly minimizes the rescaled distance (18) between the teacher and generator functions. This distance is still minimal when pd = pp. alternatively proving Theorem 2.
loss over fake
Lemma (Distance minimized by RealUID loss). Maximization of RealUID loss (16) over fake function f represents the weighted squared fa-distance between the student function == arg min, Lom(f. P8) and the teacher f == arg min, LuM (f.P))
(. bot (z))))) ((())) (nf (x7) ((7) 5T(7)) P ((()((( - a1) (x]) (x)(=)))/
ca.o max CRUUD
The proof of Lemma 2 is located in Appendix A.1.2. With the help of real data, our RealUID loss now provides the generator with the feedback the real data domain it needs to cover, i.e., the distance (18) does not vanish for points Te s.t. p" (xx) se 0, (xx) > 0 (see Appendix A.1.3). Moreover, if teacher function is inaccurate, RealUID now provably fix teacher's errors (see Appendix 1.1.4).
Choice of coefficients a; B. Lemma 2 shows that, instead of values a and 3, actually the values and 8/a determine the balance between real and generated data in the minimized distance (18). Furthermore, coefficient a only sets the general scaling of the distance, while 8/a plays the most important role, as it determines the relation between ff and f² inside the distance.
The value 8/az = 1 yiclds the distance identical to the data-free distance (14) up to scaling. Thus, even when a and real data is formally added, it may have no effect on the generator. Excessively
7

---

Published as a conference paper at ICLR 2026
low a and 3 diminish the effect of the generated data terms in the trained fake function, leading to vanishing gradients. The same issue occurs with 8/a < 1 in (18), while 8/4 > 1 diminish the effect of real data in the right term of (18). See complete distance amalysis in Appendix A.1.3. Moreover, if teacher function is inaccurate, only the choice 9/a * 1 can fix teacher's errors (see Appendix A.1.4).
Hence, good coefficients a, 8 E (0, 1] can be chosen by first finding good 9/o- + 1, it has the largest impact, and then adjusting a < 1. Both 8/a and should be close to 1.
Comparison with GAN-based methods. Unlike SiD and FGM with GANs (8), we do not extra adversarial losses and discriminator to incorporate real data, We only modify UM loss, preserving its core structure and fake model architecture. Whi While general adversarial loss is unrelated to the main distillation loss and has uninterpretable scaling hyperparameters, our RealUID loss and weighting coefficients a, 3 E (0, 1J come naturally from the data-free free-ffreeeee UID loss. The original UID loss (13), equivalent to FGM (7) and SiD (5) with asin 0.5, is obtained when a = 3- 1.
Alternative loss form. Our RealUID is implicitly related to the linearization scheme used to ob- tain data-free UID ($3.1). The loss (17) can be derived by splitting each term in the linearized UID loss (10) between real and generated data according to proportions a and 3 (see Appendix A.1.I). This form helps to prove RealUID's s properties and extend it beyond the inversion scheme ($5).
Extension for Bridge Matching and Stochastic Interpolants frameworks. In Appendix C.3, we demonstrate that our framework can be easily extended to data-to to-a matching models by parame- terizing the generated data coupling T (xo, (T) instead of the data distribution pl-
EXPERIMENTS
All our PyTorch implementations and the latest checkpoints are publicly available in
This section provides an ablation study and evaluation of our RealUID, assessing both its performance and computational efficiency. We begin in ($4.1) by detailing the experimental setup based on flow matching models. In ($4.2), we show that our incorporation of real data via coefficients a, B improves performance, speeds up convergence, and enables effective fine-tuning In ($4.3), we assess the benchmark performance and computational demands of RealUID relative to SOTA methods. Additional experimental details and results are provided in Appendix D.
4.1 EXPERIMENTAL SETUP
Datasets and Evaluation Protocol. The experiments were conducted on the CIFAR-10 dataset with 32 x 32 resolution (Krizhevsky et al., 2009) and on the CelebA dataset with 64 x 64 resolution (Liu et al., 2015), see Appendix D.3. In line with the prior works (Karras et al., 2019; 2022), we report test FID scores (Heusel et al., 2017), computed using 50k generated samples.
Implementation Details. We implement our RealUID framework for flow matching models from Appendix B. In contrast to prior studies (Zhou et al., 2024b;a; Huang et al., 2024) which employ the computationally demanding EDM architecture (Karras et al., 2022) our work adopts a more lightweight alternative (Tong et al., 2024). We also train our flow matching teacher models using CFM loss (2). Further implementation details and efficiency analysis are provided in Appendix D.
4.2 BENCHMARKING METHODS UNDER A UNIFIED EXPERIMENTAL CONFIGURATION
We evaluate RealUID under a unified experimental protocol (fixed architecture and implementation) We begin by conducting an ablation over o., 3 to assess the influence of real-data incorporation. We then compare RealUTD to a GAN-based alternative, showing that RealUID achieves comparable or superior accuracy. Furthermore, we analyze convergence, indicating that RealUID variants with real data train substantially faster than baselines without real-data. Finally, we explore a fine- mee-tunin stage initialized from strong RealUID checkpoints, showing further performance gains.
Ablation study of coefficients a, 3, We restrict the search for optimal parameters a and to values near 1, specifically 3 [0.85, 1.0] with increments of 0.02. Setting these parameters too low leads
8

---

Published as a conference paper at ICLR 2026
to noisy generated samples. Following the analysis in ($3.4), we perform grid search over the values a and f/a instead of the original a and 3. The results are reported in Table As a baseline, we highlight the UID model without data incorporation, i.c., our RealUID with 1.0, B 1.0.
As shown in the table, the ratio f/a has the largest impact on the final metrics, while a only adjusts them. Using real data with f/-- 1 or with large values outside the range (0.98, 1.02] consistently degrades performance. In contrast, values P/a- = 0.98 or 8/a- 1.02 outperform the baseline for majority of a. Note that these practical results match the theoretical description in ($3.4).
Generation
0.96
0.98
1.00
1.02
1.04
0.90
2.66
2.44
2.66
2.25
2.55
Generation
X
(J)
0.92
2.73
2.36
2.65
2.23
2.66
Unconditional
0.94
2.79
2.35
2.65
2.28
2.58
0.1
0.3
2.42
0.96
2.85
2.37
2.58
2.29
Unconditional
0.3
I
2.29
0.98
2.97
2.33
2.62
2.38
1
3
2.39
1.0
2.58
5
2.54
0.90
2.34
2.16
2.38
2.19
2.26
0.1
0.3
2.22
0.92
2.28
2.12
2.35
2.21
2.23
Conditional
0.3
1
2.12
Conditional
0.94
2.29
2.13
2.35
2.19
2.25
1
3
2.15
0.96
2.36
2.09
2.32
2.13
2.27
5
2.40
0.98
2.34
2.02
2.26
2.05
2.21
Table 1: Ablation studies of our (a, () parameters in the left table and adversarial weighting parameters
Gg X in the right table for CIFAR-10. The baseline RealUID 1.0, B 1.0) does not use real data. Configurations that sligtly and substantially outperform the baseline highlighted. All values report FID J. where lower is hetter. The best configuration in each case is bolded. The mark "_" denotes infeasible parameters.
Comparison with GAN-based method. We integrate the GAN-based approach (8) proposed by (Zhou et al., 2024a) as an alternative method for incorporating real data, enabling a direct comparison with our RealUID formulation. We combine the GAN loss with the UID baseline. As shown in Table 1, the best- rest-performing configurations are achieved with GAN losses (A = 0.3, X = 1). While this setup performs comparably to RealUID (a 0.92, 0.94) in the unconditional setting, it remains clearly inferior to RealUID (a = 0.98, / = 0.96) in the conditional case.
Table 2:
This table presents the results of ablation study of our RealUID framework, evaluated using the FID metric under both unconditional and conditional generation setups. The Teacher Flow model with 100 NFE reported reference. The performance of the UID (FGM) baseline without real-data incorporation indicated in italic. For emphasis, we underline the two counterparts that incorporate real data: the GAN-based and our RealUlD methods. The best-performing configurations, obtained via an additional fine-tuning stage, are highlighted in bold. Qualitative results are presented in Appendix D.5.1.
Model
FID (1)
Model
FID (L)
Tenoher Flow (NFE=100)
Tracher Flow (NFE=100)
5.56
ULD (PGM)
2,58
ULD (PGM) UID 4 GAN GA -03,
2.21
UID a (2S. 0.3, -1 1 25, - - 76)
2.10
2T 26, 26 75)
1.88
ReaUUD 0.92, J 0.94 0.92, 0.50) (Ours)
1.98
RedUUD 0388 D 0.56
0.96, 0.52) (Ours)
1,57
Unconditional
UID
7
Conditional
0.92, 0.9.4
any 0.92, fory
stage
5.56 (Teacher Flow)
monnntuning..mmm stage
5
a 0.38, 0.996
0.96
4
8-7 A -
How
3
2.10 (UID GAN
1.08 (UID CANA
50
100
150
200
250
300
50
100
150
200
250
300
350
400
Iterations (x10
Iterations (x100))
Figure 2: Evolution of FID during CIFAR- 10 distillation for (i) the UID (FGM) baseline, (ii) the best best-performing RealUID configurations, and (ii)) subsequent fine- fine-tuning, evaluated in both unconditional and conditional settings. The performances Teacher Flow and UID+GAN are indicated by horizontal lines their respective colors.

---

Published as a conference paper at ICLR 2026
Convergence Speed. Our RealUID(o, 33 with parameters which highlighted in Table 1 achieves faster convergence than the UID baseline. For clarity, we present qualitative comparisons in Figure 2. The best RealUID configurations reach the saturated performance level of the baseline after ~00k iterations, whereas the baseline requires 300k iterations to achieve comparable metrics.
Fine-tuning stage. We observe that RealUID and GAN frameworks offer substantial flexibility for fine-tuning. In this procedure, the generator is initialized from the best-performing checkpoint obtained during training from scratch of the corresponding framework, while the fake model is initialized from the teacher. Fine-tuning then proceeds with new values OFr and 9eT for our RealUID and Xpp and Xpr for GANs. We present the best-found fine-tuning configurations for both methods in Table 2. Ablation study analyzing the effect of loss coefficients is provided in Appendix D.2.
Scaling to larger datasets. In Appendix D.3, we provide the results of the same ablation study on the CelebA dataset with 64 x 64 resolution. Notably, our RealUID performance and the optimal values 8/n remain the same across datasets.
4.3 BASELINE COMPARISON
As shown in Tables and 4, our RealUID after fine-tuning consistently outperforms all prior flow- based models CIFAR-10 10, significantly surpassing the strongest flow distillation baseline, FGM. Despite its compact and lightweight architecture ($4.1) with nearly 2x faster inference, it achieves performance comparable to leading diffusion distillation methods SiD (Ası==1.0%) 1.0., while falling short of adversarially enhanced models such as SiD²A. We hypothesize that this performance gap is attributed to architectural and teacher capacity differences rather than the lack of adversarial loss.
Our latest checkpoints and metrics (Appendix D.4) are available in our repository.
Table 3: Comparison of unconditional generation CIFAR-10. The best method under the FID metric in each section is highlighted with bold.
Table Comparison of conditional generation on CIFAR- 10. The best method under the FID metric in each section is highlighted with bold.
Family
Madel
NFR
DDPM (Ho et 2020))
1000 35
3.37
VP-FDM (Koraa et al., 2022g
1.97
Fanily
Model
XFE
FID (2)
SyylsiAA2S-AAA-TE Tine (Karms et al., 2020)
Diffusion ProjectedAAN (Wing et all, 2022)
SpleeAAN--AAA--ime DI (Lo et al. 2023)
2.54
2.92
2.7L
VP EDM (Kams al. 2022)
GRT-Hooe (rkngg mtll. 3001)
35
6.25
1.79
iCT-deep (Song & Dhuriwal, 2025)
2.51
BiGAN Ture (Brock 21
5.47
Difiesion& & GAN
DMD (Yio al., 30244)
Dift Instract (Loo et al., 2023)
3.77
4,53
StyleiAN444 ADA (Kara 20200
PAX -Tree (Kamas et 2001)
2.42
3.49
CTM (Kim al. 2023)
1,98
ADAGAAAAAAA-- (Las ei 2025)
2.37
sCD (I. & Song. 2024)
3,66
SoleGANN-N1 XL (Suaer al., 2012)
1.85
sCT (La Soog.
2,85
ShileSNNXXL, (Takala 2025)
1.36
SIDA, ohan 12hooo 20242)
SID. 1.0 (2hou 2024b)
1.2 (22bou 20240)
1,92
1.52
2,00
Tilliaou GAN
DMD (rca. (L) et 20040)
BMM (ono ing (Yinia 3074h)
20240)
2.56
3.82
5.58
SITPA,
3024g)
1.50
GDD I (2heesg ct. 2004)
1.44
SID' A. oso 1.0 cll 2024a)
1.50
CTM ct 3023y
1.33
CEM (Yang al., 2024)
5.34
SID osp (2hon 20040)
1.95
IMM (Zhon 2025)
3.30
SIDA,
G2hoa
201200
1.45
Row-lusd
MasaFlow (Ciang al., 2025)
FACM (Peng et 2025)
2.92
2.69
SIDFA, 1.0 (2bos al.., 2024a)
SID"A 1.2 (2hoz sl., 20242)
1.39
1.40
i-RoFlow (+Dieill) en 2023)
RRReloo (+Disiil) (Liu et al., 2023)
Reflow (+Disill) (Liu et al., 2023)
4.83
5.21
6.18
Flou-boazd
FGM (ituagg et al., 2004)
RealU|D (Oun)
2.58
1.87
PGM (Huang et al., 2014)
3.08
RealUID + FT (Ours)
1,98
5 DISCUSSION AND EXTENSIONS
Extensions. Our RealUID framework ($3.4 and Appendix C) can distill Flow/Bridge Matching, diffusion models, and Stochastic Interpolants enhanced by a novel natural way to incorporate real data. In Appendix A, provide three extensions of our RealUID: more flexible General RealUID (Appendix A.2), General SiD framework for all matching models with real data and OssDD # 1/2 (Appendix A.3) and Normalized RealUTD for minimizing non-squared fg-distance (Appendix A.4).
Relation to DMD. Instead of minimizing the squared fnddistance between the score functions, Distribution Matching Distillation (Luo et al., 2023; Wang et al., 2023; Yin et al., 2024b;a, DMD) approach minimizes the KL divergence between the real and generated data. Its gradients are computed using the generator and teacher score functions, leading the similar alternating updates. We would like to highlight that DMD does not fit UID framework. Nevertheless, we investigated an opportunity incorporate real data into DMD without GANs in Appendix A.5.
10

---

Published as a conference paper at ICLR 2026
6 BROADER IMPACT
This paper presents work whose goal is to advance the field of Artificial Intelligence, Machine Learning and Generative Modeling. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Acknowledgements. The work was supported by the grant for research centers in the field of Al provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 000000C311925PPFFO00 and the agreement 1=139-10-202--033.
LLM USAGE
Large Language Models (LLMs) were used only to assist with rephrasing sentences and improving the clarity of the text. All scientific content, results, and interpretations in this paper were developed solely by the authors.
REFERENCES
Michael Albergo. Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprini arXiv:2303.08797, 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Timothy CY Chan, Rafid Mahmood, and Ian Yihang Zhu. Inverse optimization: Theory and applications. Operations Research, 73(2):1046-1074 2025.
Valentin De Bortoli, Guan-Horng Liu. Tianrong Chen, Evangelos A Theodorou, and Weilie Nic. Augmented bridge matching. arXiv preprint arXiv:2311.06975 2023.
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin Patrick Murphy, and Tim Salimans. Diffusion models and gaussian flow matching: Two sides of the same coin. In The Fourth Blogpost Track at ICLR 2025, 2025.
Zhengyang Geng, Ashwini Pokle, and Zico Kolter. One-step diffusion distillation via deep equilibrium models. Advances in Neural Information Processing Systems, 36:41914-41931, 2023.
Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505. 13447, 2025.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Nikita Gushchin, Alexander Kolesov, Alexander Korotin, Dmitry P Vetrov, and Evgeny Burnaev. En- tropic neural optimal transport via diffusion processes. Advances in Neural Information Processing Systems, 36, 2024.
Nikita Gushchin, David Li, Daniil Selikhanovych, Evgeny Burnaev, Dmitry Baranchuk, and Alexan- der Korotin. Inverse bridge matching distillation. 2025.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 2020.
Peter Holderrieth, Marton Havasi. Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky TQ Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. arXiv preprint 1rXiv:2410.20588 2024.

---

Published as a conference paper at ICLR 2026
Zemin Huang, Zhengyang Geng, Weijian Luo, and Guo-jun Qi. Flow generator matching. arXiv preprint Sttt::2410.193110 2024.
J Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18(4): 203-210, 1986.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 15:265655-26777 2022.
Dongjun Kim, Chieh- Hsiin Lai, Wei- Rei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Ue- saka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint urXiv:2310.0227) 2023.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling, The Eleventh International Conference on Learning Repre- sentations, 2023. URL https: pttettss///ppeeeeeeee
n Eorum? iinin QVRRDTMDC......
Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint MvXiv:2209.14577, 2022.
Xingchao Liu, Lemeng Wu, Mao Ye, et al. Let us build bridges: Understanding and extending diffusion generative models. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023. URL https: / wwwp//wweeeeew.ee/o/pp/e. net fotttfoorust (tttt//ooommiii-///111.1
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint 0rXiv:2410.1108 2024.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36:76525-76546, 2023.
Stefano Peluchetti. Non-denoising forward-time diffusions. arXiv preprint arXiv:2312.14580 2023.
Yansong Peng, Kai Zhu, Yu Liu, Pingyu Wu, Hebei Li, Xiaoyan Sun, and Feng Wu. Flow-anchored consistency models. arXiv preprini arXiv:2507.03738, 2025.
A Sauer, K Schwarz, and A StyleGAN-XL Geiger. scaling stylegan to large diverse datasets. In Proceedings of the SIGGRAPHI Conference. ACM, pp. 1-10, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256-2265. pmlr, 2015.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint (tt::2310.14889 2023.
12

---

Published as a conference paper at ICLR 2026
Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021. URL https: http:///wnnneeeeeee.eeeeee
1ddPxTTG122RHH.
Yuhta Takida, Masaaki Imaizumi, Takashi Shibuya, Chich-Hsin Lai, Toshimitsu Uesaka, Naoki Murata, and Yuki Mitsufuji. San: Inducing metrizability of gan with discriminative normalized linear layer. arXiv preprint 1.Xii:2301.12811,, 2023.
Alexander Tong, Kilian FATRAS, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https:/ http:////ppeeeeeeeeeeeeeooooeoo.
3AW. Expert Certification.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.02262, 2022.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: figh-fidelity and diverse text-to-3d generation with variational score distillation. Advances in neural information processing systems, 36:8406-8441, 2023.
Ling Yang, Zixiang Zhang, Zhilong Zhang, Xingchao Liu, Minkai Xu, Wentao Zhang, Chenlin Meng, Stefano Ermon, and Bin Cui. Consistency flow matching: Defining straight flows with velocity consistency. arXiv preprint arXiv:2407.02339, 2024.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455 47487, 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Tacsung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6613-6623, 2024b.
Kaiwen Zheng, Guande He, Jianfei Chen, Fan Bao, and Jun Zhu, Diffusion bridge implicit models. arXiv preprint arXiv:2405.1 15855, 2024.
Linqi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching. arXiv preprint 07X1::2503.07565 2025.
Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. arXiv preprint arXiv:2410.1449 141.99999, 2024a.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024b.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Backgrounds on training and distilling matching models
2.1
Diffusion and flow models
2.2 Universal loss for matching models.
2.3 Distillation of matching- based models
2.4
GANs for real data incorporation
3
Universal distillation of matching models with real data
3.1
Universal Inverse Distillation
3.2
Relation prior distillation works
6
3.3
Connection with Inverse Optimization
3.4
RealUID: natural approach for real data incorporation
4
Experiments
4.1
Experimental setup
4.2
Benchmarking methods under a unified experimental configuration
8
4.3
Baseline comparison
10
5 Discussion and extensions
10
Broader impact
11
7 LLM Usage
Theoretical proofs and extensions
15
A.1 RealUID theoretical properties
15
A.1.1
Alternative RealUID split form
15
A.1.2
Proof of RealUID Distance Lemma 2
16
A.1.3
Explanation of the choice of coefficients a and 8
17
A.1.4
Correction of teacher's errors
18
A.2
General RealUID loss
20
A.3 General SiD with real data
22
A.4 Normalized UID and RealUID losses for minimizing di-dtttaece
23
A.5 DMD approach with real data
24
RealUID Algorithm for flow matching models
26
C Unified Inverse Distillation with real data for Bridge Matching and Stochastic Inter- polants
27
C.1 Bridge Matching
27
14

---

Published as a conference paper at ICLR 2026
C.2 Stochastic Interpolants
28
C,3 Objective for general data coupling
28
D Experimental details and additional results
29
D.1
CIFAR-10 distillation from scratch
D.2
CIFAR-10 distillation fine-tuning
30
................................
D.3 CelebA distillation
................................
30
D.4 Further hyperparameters gridsearch
................................
32
D.5
Example of samples for various methods
32
D.5.1
CIFAR-10 generated images
33
D.5.2
CelcbA generated images
39
A THEORETICAL PROOFS AND EXTENSIONS
In this appendix, discuss RealUID framework (Appendix A.1) in theoretical details and pro- vide three extensions of it: General RealUID framework with 3 degrees of freedom (Appendix A.2), General SiD framework with real data (Appendix A.3) and Normalized RealUID framework for mini- mizing Ey-distance between teacher and student functions instead of the squared one (Appendix A.4). All proofs are based on the linearization technique and splitting terms in linearized decomposition between real and generated data. We also propose approach to incorporate real data into DMD framework, which is unsuitable for our RealUID (Appendix A.5).
A.1 REALUID THEORETICAL PROPERTIES
In this section, discuss RealUID loss in detail. We begin by presenting its alternative form and how it connects linearization technique and real data incorporation (Appendix 1.1.1). We then demonstrate that the loss minimizes squared fy-distance between the rescaled teacher and student functions (Appendix 1.1. Finally, we provide the motivation of the best choice of coefficients # B from the perspectives of the better distance (Appendix A.1.3) and the correction of the teacher's 's errors (Appendix 1.1.4).
A.1.1 ALTERNATIVE REALUID SPLIT FORM
Let us recall the linearization trick that we apply to make the minimized squared norm between the student function fo and the teacher f tractable. For each time t and generated point x/. we restate this squared norm as the identity ||| = maX { [-|6b]2 + 2b, a)}, Va R and use an auxiliary function 5, (xx) to parametrize vector b. In the end, substitute the student function f (af) with its conditional and differentiable estimate f(" (x? (=)):
[, [IISe (xf) (=:))1 </-p?
E.,/0, (=155((=))) + be(af)
12 (+()))
-E ~ (=. (z)) ((6. (( xp-pp (-
(19)
In addition, we use parameterization 8 f" with fake model f to obtain our UID loss which matches the previous distillation losses.
Originally, we derived our RealUID loss (17) from the idea of splitting each term the linearized form of data-free UID (19) between the generated and real data in proportions defined by coefficients and (1 - ()), and Lemma 3, and this form completely matches the inverse optimization form defined in Theorem 2.
ox) and and (1 - (3). We present the split form of RealUID loss in
Lemma (RealUID split form). The RealUID loss (17) can be restated as
Ca.3 P() E 1,,1,,11,20 16. 1 [=a]]),(= 2|2 2a(6. (x?)) (z)), (+ af ((pp((1aa))
15

---

Published as a conference paper at ICLR 2026
1o (1 ()]l= 2(1 - a)((((1)) - 2(11 (271)))) zi simpi
with the parameterization 5 f.
The idea of splitting coefficients between two data types helps to prove properties of RealUID, and extend our real data incorporation technique to general form (Appendix A.2), SiD framework with osaD + (Appendix A.3) and new distances (Appendix A.4).
Proof. Putting explicit values for RealUM loss (16) in RealUID loss (17), we get:
CR-UID (f.
CRUUM (S" .P) - ca.8 (S.
- Ne,,,
(x])x - (x
+ (1 - a) Eg, [x;)
1 1 S7 ( (=)
- -a (a..E,,a R
[1f((TE)
B (x (1)1))))1
- -1 a) Ra,
[IVa(at)
1 1 3
1 1 57 (x;
a
Then, we group the factors with the same data type and multipliers:
R--UID
ra.9
(f, pb)
= CR.UM p8) (f.
= -Baaggin
a (1 3 a (x -
a (1f((z]) -
2 f (0))==0)1
(1 a)) (1)f((;;) 1 - 1 5f a)
a) 11((2)) 1 1-3
1 1-a 1.
+ E
Ey,ag.nt
(2(())))--2..(((2))) a 11 +3 - (f
+ ( 11)
+ Ei,
Braag.nn .n
[28 -,g
(f/(()) (((6)
Ee -2(1-1) (2)1-8)-(57(x)) 2211-190110440--0))"6(6660010001
Finally, denoting parameterization f, we obtain the required form:
CR.0 (f: (f. p0) E, - 2o( 12
Er,,g of (ucp::/---aaaaaaodddd--d----ada-
A.1.2 PROOF OF REALUID DISTANCE LEMMA
Proof of Lemma 2. this proof, we use the split form our ReaLUID loss from Lemma 3. First, we take math expectation over data points v Since the expectation can be taken in a reverse order, ie., =u
Ext-mi =a- (1-n((117)) we see that
Exg Exgept ~ot (12g)) ((06 (7 (x](=)))] = Exj~pt ((5,(1]), Eugeng ap (x110001
= Exg-n; (()(())) (z
(20)
For the generated data term Egg-pt -f ( (=))) the reasoning is similar. Thus, we write down RealUID loss in an explicit form with d, f ft:
(112)) Jf (of |=())] B (x, f (=f))),
CR.UnD (ò, p = Ee-10. 1a + Bo Bep [-(1
(o.
28(
==)l + 20112)(1006))).)..)0))) ()) 2(1 -10---)))0))))))0)0-0--4-0011100 (21)
16

---

Published as a conference paper at ICLR 2026
Then, rescale the generated data terms in RealUID loss (21) using the equality pf())) p((x)) (x()) (24e for E IR (we assume (x4) > the real data, i.e,
0, Vxy, f) leaving only math expectation w.r.t.
-1(1 a) + 1 [1))(1))11
f (6. P0) = 10.1 2) ~pi
pt (x
22 (xt)
pl(x)
[o.T 21 mpi
(6e( fe (x1)) 2 o.)
(xt)
pł pf
Finally, we maximize the loss w.r.t. dy (2f)) for each æ and as a quadratic function. The maximum is achieved when
113 a)
p 15 (xt 8 3plta))
(F S. (x
(x())
pr p( ).
1(1 - a)) +
11 (xTg)
pi (x,)
or in terms of the fake model f = f0 -8
f((x)) (1- 3) + (z+)
g. pr (mx)
arg max CRUDD (F. (F,8)) CRiD (t,
(22)
(1 a) + a (x,)
(=x))
The maximum itself equals
lIf((zt) ((B a) + a [1
pi(mi)
- f (zt)
goice)
I??
p((
max C CR.UD p = Bo-(o.T)B2i-- Exi 1
(1 a) + p (x
It is easy to see that when p p and S 1 this distance achieves its minimal value 0. Moreover, optimal fake model this case matches the teacher f", i.e.,
(xx) (1 3) + (x:)
arg max C (. (1,
(14).
(1 -1-68 a) + (z,) (z.)
A.1.3 EXPLANATION OF THE CHOICE OF COEFFICIENTS & AND 33
Here we show that the best way to incorporate real data during generator training is to set 8/a * 1.
Following Lemma 2, we know exactly what distance our RealUID loss implicitly minimizes. Below we examine it for various a, 3 E 1];
max CR.0m (f. Po) = [
le(2 3, a,
f
(2)) 1) + p. (2+)) (z))
(l)))) (1)))
41,,,,
(1 - a)p (ax) + (pp(
where (2,, 3, denotes the distance for the particular point zt.
The total distance mostly sums up from the two groups of points: incorrectly generated points from the generator's main domain, i..., (xxe) >> 0, (2,) se 0, and real data points which are not covered by the generator, i.e., pf (xz) R 0,p" >> O. For the points out of both domains pf (..)) Rs O. p (#.) Rs O, the distance tends as well as for matching points p (ax) R P (xt).
Choice of coefficients a, B. Next, we consider various coefficients a, B E (0, 1] and how they affect two main groups of points.
All configurations affect the incorrectly generated points I : (x1) R O, (x+) > 0: 1r 3, (x) R (xx) (x,) - (x.( - (x)))1 (xx))l pe (xx) > 0. (23)
(x)(
Note that increasing f/a > 1 will diminish the weight the distance in comparison with a = B = 1, while decreasing otherwise will lift the weight up.
17

---

Published as a conference paper at ICLR 2026
Figure 3: RealUID loss for LDGasssiass under various coefficients (a, 9).
Configuration 8 < is unstable for uncovered real data points 2, pl 0, >> 0:
4121, a) (x,) - 1) (x,) - (x.) (x,)11 20.
(3())
Configuration a 1 (UID loss) does not affect uncovered real data points Ty (x()) so 0,p" (2;) > 0:
3, (xx) (xs)
(2,))))01))) - pl.))) (xx))l (m)) - s. = 0.
(xx)
Configuration 3 =a does not affect uncovered real data points 2₁ : (x+)
0, 0, (zx) > ((,,,,,0) |app (2x) (p)( [off -3f (p) (x+))²
(1 0z (x+)
(1 (1-a)
pł (x
Notably, in this configuration, the distance drops even faster than when = B 1, what makes it even less preferable.
Omly configuration f/a- F affects the uncovered real data points dı (.())) 0. (xt) > O:
4 3.a) Ip) a.) f2 (xx) (3.)-3pp(z)) (xx)) a (xx)
0.
Visual illustration. We analytically calculate the loss surface (z 3) between the FM models transforming one-dimensional real data Gaussian N(n", 1) and generated Gaussian N (( 1) to noise (0, 1) on the time interval [0. 1]. In this case, the generated and real data interpolations are (xx) (x f), 12 + (1 - t)2) and p (xx) = [><x=lpp"
t), 12 + (1 1)2). The
unconditional vector field between N N(,, ( 1) and N(0, 1) can be calculated as
Jt. 20 t.
St To
xr zo t) 10. -Nooll, 1)dco
ue(xr)) Ezo
W
t
a(2t² 2t)
bt2
(xt - 1u11 t))²
V2T(1 21. + 2/ exp
2(1
21 + 212y2
(24)
In Figure 3, depict the loss surfaces for the fixed time 1/3, real data μ 2, generated data 11" and various pairs (a., (3). We can see that configurations 8/a do not detect the real data sample, even when B < 1 and real data is formally used. while 3/ax # 1 actually spots both domains, increasing the weight of generator domain when 8/a and decreasing it otherwise.
A.1.4 CORRECTION TEACHER'S 'S ERRORS
In this chapter, we assume that instead accurate teacher f*= arg min, ( we have access only to the arbitrary corrupted teacher f*. We will show that adding real data via our approach with + 3 provably mitigates the teacher's 's errors the final generator.
Minimized distance. With the corrupted teacher f" and 8 = f - f. our corrupted RealUID loss takes the split form from Lemma 3
Ca. (6, p.) x2 (11x))))))))))
Bespo,11eegppp 1 -20_0/2))11222000))) -23() (22
18

---

Published as a conference paper at ICLR 2026
(o, I -o "I p/mpi(e =/mp((()=))
(2tp:/))))(0_(_11(0())=))))))))) -21 -210))))))))))))
Note that sampled terms fi (x)) and (2 are not affected by the corruption and give the
accurate functions I/ (at)) B (2 (x)) and (2(x)) = E (02)
| 2 (0. (xf)) 23.6,(()) + [.. T7 Bex ~pi [-(1 a) ( + 2(1 a) (x)() fr (x1)) - 2(1 ((01-)))(()))))(((()))))
RRUID (ò. p 10.10 R2 2-
Then, we rescale the generated data terms using the equality p (are)
p((ee) p (2i) for dt E RD
p(((e)
(we assume pī (x+) > 0, Vare; 1) leaving only math expectation w.r.t. the real data, i.e,
p((=;)
(ō. Bo- -111
+a)+ -((1-)++
(z0
pf(xt)
2(0x(z)). (16)=)))((3)) + (xi)
fl (1:)
-E =/~pp
p (+
Finally, we maximize the loss w.r.t max; ( (0)
(<())) for each x and I as quadratic function
(1f(())) ((1 a1 + a
- (1 ()) 32122 m()
5f (xt)))
pt [a=)
10 Bwirpt
(25)
(1 a) + a (ar pt (x:)
Hence, max-min optimization of the corrupted RealUID loss implicitly minimizes expected distance (25). However, due to arbitrary function f. we now cannot guarantee that minimum is achived when the relation inside the norm equals O. Previously, we could use the solution p p which obviously achieved a minimum of O. Now, due to the implicit and complex relationship between fo and p", we can neither find an explicit form for the optimal p" nor guarantee the minimum of 0.
Choice of coefficients a, B. Here we give an intuition on why coefficients 8/4 * 1 can fix the teacher's errors. while $/6 cannot. For simplicity, we assume that the minimized distance (25) actually attains minimum of O when
((1 a)) (mx) + (z))
(x()) (1 B)p (ax) (x.) - (x = 0. (26)
(xx)
case of a = 3 = 1, we have fi = f2, i.e., the generator learns the corrupted function.
case of a = B < 1, we have
(1 - (x.) (x)) (1- a1-a)pt( + (x)) (x:) + (1-a)pt(z)) apl (1
(xx)
this convex combination, the corrupted function jr is always between the true teacher function f" and the optimal generator function f", i.e., the generator learns even worse function.
In case of B/a + 1, there exist intervals of a, 3 which can give better generator function than the corrupted teacher. For example, coefficients a # close (z.) and (1 ())t((,) f( fz,z)) in (26) to get ff (z,) s 9. (zz.) Hence, we can steer S towards the true teacher picking 9/a 1 or f/a > 1 depending on the corrupted and clean teacher's values. However, we cannot find these intervals analytically due to complex distributions and functions.
allow to neglect the terms
(1-a)p;
Note that we derive the same recommendation 8/4 # from the perspective of correcting the teacher's errors and from the perspective of the minimized distance surface from Appendix A.1.3.
Visual illustration. For visual demonstration, dimensional real data Gaussian N(4,,, , 1) and generated Gaussian A (( 1) to noise N/ 1) on the time interval [0, 1]. In this case, the generated and real data interpolations are p (arx) (1
consider the FM models transforming one-
noise.N(O,,
19

---

Published as a conference paper at ICLR 2026
t), t), (1 and pi (zz) (1 (1-t), t), +1 (1-1111 ))
The unconditional vector field
between Netween.V(H.. (N,,1) and M 1)
can be calculated as
Tt 20
vo 1) 10.1 -N 1)dzo
ut (x+) Ex #xompa(-1or)
.N
t
a2t 21)
bt²
(Et pil 11 t))²
exp
(27)
V2T(11 21
212)
2(1 21 + 21
In Figure 4, we depict the optimal generator mean 1e and vector field u satisfying (26) for various deviations ü' u² and fixed time 1/3, real data μ 2 and point 2t -1
We can see that with a 3 = 1, the generator learns the corrupted vector field, and with a = 3 < 1, the learned field and means are often even worse. In contrast, with 8/a 1, the generator can learn vector fields and means which are closer to the real data. Although the generator cannot satisfy relation (26) under large deviations, still produces better results with the real data.
compresu"
compled
compres
0.503
5,000 offs
0.500
3.56
0.53
0.603 0.455
--0.1
pis a 7777 at
m-0.5
0'-u"
Figure 4: Learned generators for RealUID loss between ID-Gaussians with corrupted teachers.
A.2 GENERAL REALUID LOSS
Extending our real data incorporation. We recall that UID loss (Theorem 1) can be restated via linearization technique with 8 as:
Lum p6) E...ooT 2/-p(( (112g)
2(0.0))))) -266)) -6)
Following alternative definition of RealUID loss from Lemma 3, incorporate real data into data-free loss by splitting each term in the linearized form between generated and real data as:
CR.UD (6. ) = E o. -p <p-pp (-)=g)
(1 (2060)))))) - (z)
E. ((mp)(11)))
(1 a) - ())(()(((())))))))))))))))))))))
2(1 ()(6 - (1
In RealUID loss (17), its three terms are split with proportions a and a. a and 1 - a and 3 and 1 - B, respectively. We go even further and split the first quadratic coefficient - (-)1 using new parameter y to create one more degree of freedom. Moreover, we can use other parameterization of 8, since its form does not change the proof of distance lemma.
20

---

Published as a conference paper at ICLR 2026
Definition 3. We introduce General RealUID loss Cx p&) on generated data p E P(R?) with coefficients ax y,
Lx-U1 PQ) == E p aog (0 (0 (3(6. x 2xpp (11g))
E 1. |- 7) 10)6 201 2(1 ())
22 <i-pi (1125)
Optionally, one can change default parameterization 5 = f (e.g. with 6 = f( f f)), and substitute sampled real data term ft (2]10²) with the unconditional teacher f7 (a:) and vice versa.
Theoretical properties. In case of 6 = f and y * a, the General RealUID loss cannot be expressed as inverse min-max problem (15) for simple losses, since some scalar products do not eliminate each other. Nevertheless, min-max optimization of LRUUDD still minimizes the squared f2-distance between the weighted teacher and generator functions, attaining minimum when r = Pū- Lemma 4 (Distance minimized by General RealUID loss). Maximization of General RealUID loss LR CR. over 8 represents the weighted squared Eg di-dtanncce between the student function f" := arg min, LfM(f. p&) and the teacher f == arg min, LUM (f.P)):
- (ft) p. (zt )5 (x:)) (x) (z (2))
max fR (6, pb) = (0,7). zimpi
(xt)) max (1 7)p7 + mel (=7))//²
(28)
The distances being minimized for RealUID (Lemma 2) and General RealUID (Lemma 4) are almost identical except the scale factor. Thus, we keep the same recommendations for choosing coefficients or B9 as we discuss in Section 3.4. The factor 9/a still has the largest impact within the distance, while and set the scaling. Values 9/o and should be chosen close to 1, but not exactly 1.
Proof.
First, we take math expectation over data points z6- Since the expectation can be taken in (2-pp(( = Bug-nt-=a~na(ie we see that - -
reverse order, i.e.,
Engarg
( mg) (6 (xf). f. (x;1=6)M = Exjrpt (zx();), Eug-ot (x)1))))))))))))))
(=]1=()])
Extrot (6 (x;), (xt)))
(29)
ff (af))) the reasoning is similar. Thus, we write down General RealUID loss (Def. 3) in an explicit form with 6, = ft - ft
For the term Exg~pg..m .f -f/ ~pp(( (6 (6)
fft (xf [=f)>|] = Eng ((x)) J
CR.UID (6. p = (,- Eeg - 666 (=.)
f2 2a (6, (i (2+)) - f2
E (o) Bep [- (1 7)1 (z 11 + 2(1 a)) (xx (x1)) - 2(1 (x)) J (=+))) Then, rescale the generated data terms in the General RealUID loss using the equality (xx) pl(x)) Pi(x,] (iz))2E(e (1x)) for xe TR (we assume pf (vx)) > 0, Var, 1) leaving only math expectation w.r.t. the real data, i.e,
- -1(1 ((1-)) + 18 (xz) pī (x)
CR.0 (5. pb) = Eo-10.T) xi upi
1110 (z)()1
p (zi)
2,3 pl (6, (6, J (a;))
+ E3 E 10. ~pp
2[33 - a) +
pl (x7)
p (x
Next we maximize the loss w.r.t. de(x;) for each o and / as quadratic function. If (1-7) p5
pl (=t) < 0, then the maximum tends to +00. Otherwise, the maximum is achieved when
[BB a) + a pf (o) (oi)
1110
p((xf) (x:4)
(x()))
pf pf
(30)
[(1 (117))+ +2
[((2))) p(()))
The maximum itself equals to
15((;)) ((3 a) + a p((x7)
5- (x:))
max CR.9.7 (6. P8) = EB But
pp(x?)
(1 - 7) +
p (x))
21

---

Published as a conference paper at ICLR 2026
Alternative parameterization. In the proximity of the solution, when generated data approaches real one, i.c., pl =s Př, the optimal 69 (30) approaches
[B3 a) + a f) (xt) - 3 fl(xt) & (U
5e (z))
-2
(1 +7.1
Thus, the parameterization 6, = B(ft - fo) may naturally help reach the solution without making the fake model learn extra information about the teacher near the optimum.
In experiments in Tables and 7, this parameterization with the corresponding coefficients 7 = a and B yields slightly better metrics from +0.02 to +0.04.
Extra ranges for coefficients a, 3. New perspective on our RealUID loss allows us to expand the range of feasible configurations for the parameters o, 8, and 7. Specifically, it is now possible to set a = 1 for any 3, whereas in the original loss (16) this configuration unavailable due to division by zero in the real data term. Additionally, one can now use values a, 3,, >
However, we observe that in the experiments reported in Tables and 7, these extra configurations are highly unstable and lead to degraded results. This happens due out-of-domain generated samples and negative quadratic summands leading to infinite losses and metric (28). Hence, we stick to the original ranges a, 3, 7 E (0, 1].
A.3 GENERAL SID WITH REAL DATA
Our real data incorporation. We recall that data-free UID loss (Theorem 1) can be restated via linearization technique with 6 = f as:
( (6. (.)))))
(x. 2(6) f(
(31)
Following alternative definition of our RealUID loss from Lemma 3, one can incorporate real data into data-free loss by splitting each term in the linearized form between generated and real data c (5. 1))
E 1. 1.T 200 1 (1
(z)))
e (-lan)
1o (mpp((12g)
(- (16, 2(1 (=) - -(1-1))666))
(32)
General data-free SiD, The authors the SiD framework (Zhou et al., 2024a;b) for diffusion models empirically notice that scaling the first coefficient -166(= (=?) l by the factor 20sa in the UID loss (31) for generator updates yields better performance. Hence, we generalize the SiD loss to other matching models. Namely, the General SiD loss for the generator the following loss with 5 == f - f and parameter asaa E [0.5, 1.2]:
(f) = (o [-20sp +2, Pepppp (-1=8)
(33)
while the UM loss (Def. 1) for the fake model remains intact. The same positive effect is observed in experiments with flow matching models in FGM (Huang et al., 2024), where the authors do not calculate the gradient through some loss terms and obtain the General SiD loss (33) with OSiD = 1, achieving better performance.
General SiD with real data. Following the structure of the General SiD loss (33), we propose to scale the first coefficient in our RealUID loss (32) during generator updates. The whole General SiD pipeline with real data (RealSiD), defined by coefficients a, B E (0, asi E [0.5, 1.2) and teacher f*, two alternating steps:
1. Make one or several fake model f update steps, minimizing UM loss with real data La.3 C (f. 18):
a. -eEEeoo, Eng ~pp(ttl [][a(=]) - 3 - (
L IRuMM(.Pp (..P))
generated data 2e term
22

---

Published as a conference paper at ICLR 2026
+ (1 ar) 1 ~ pp (11
f7 [=))))))
real data Pó term
Make generator update step, minimizing the loss LR =
E E ~p [-20ssiD a]] (x))) (x 23 (34)
"/ppp (-1a))
where 6, ft- f-ff..
In the SiD framework for diffusion models, the data-free generator SiD loss (33) is additionally normalized, and the SiD loss with real data (34) should be normalized the same way. For more details on normalization, time sampling, weighting, etc., refer to the original articles (Zhou et al., 2024a;b).
Experimental validation. We modify the data-free SiD loss in the official SiD implementation with real data and conduct short ablation study on unconditional CIFAR-IO. The SiD codebase for diffusion models can be found in
https: https://githusee http:/tww...ommmmmmnyyuaaohooo/a
We compare the data-free SiD loss (33) and our RealSiD loss (34) with the best coefficients az, from Table I for both the theoretically justified OsiDD = 0.5 and the best practical heuristic OssDD = 1.2. We do not change anything clse and use the default hyperparameters and training pipeline as described in (Zhou et al., 2024b). The results are presented in Figure 5.
22
0.5
g- p=o.
22.5
1.2
92,
p-0.94
0=9..22 9=0.94
am 96, pm-0 98
1.000 |=1.00
18
17.5
16
15.0
12,5
12
10.0
10
5.0
Iterated M-Images
Ieertee M-images
Figure 5: Evolution of FID during unconditional CIFAR-10 distillation for the data-free SiD loss 8 1.0) and our RealSiD loss asa (left) and asa 1.2 (right).
For the accurate asıDD 0.5, the RealSiD results for diffusion models are similar to those for flow models. Configurations with 8/a = 1.02 boost convergence compared to the data-free baseline (a = == 1), whereas the case of a = 3 o 1, the convergence speed remains close to the baseline.
However, for the heuristic OsiDD 1.2, our best configurations with 2/6 * 1.0 either degrade performance compared to the baseline become unstable. This suggests that heuristical SiD may require a different approach to incorporate real data, more careful tuning of the coefficients a, and other hyperparameters, due to diffcring architectures and training pipelines.
We would like to highlight that all analyses and recommendations were justified only for asi = 0.5. For other ass values, this may not hold true.
A.4 NORMALIZED UID AND REALUID LOSSES FOR MINIMIZING & -ISSTANCE
Using the linearization technique from ($3.1), we estimate the non-squared -ddistance between the teacher f : arg min, Lum(f, po) and student f : arg min, Lum (f. 16) functions. this case, the connection with the inverse optimization disappears.
For a fixed point af pf and time ~ [o, T], we derive:
(x
(xf) (2
max (x()
f((=?)
18 (a+))
|6 (x)4
23

---

Published as a conference paper at ICLR 2026
de(2)) [|0e(2))11
5,(x9)
max Eng 1000ppp(-aa {x(2))
(o,(a 13 ((())(a))
(35)
With the parameterization &c = 17 - Se. the Normalized UID loss Luw(S pi) for solving ming jo. [1 (af) - f2 (=f))) is
(T(a)))(((z))) (2 1 p }. (36) (if((())) ( fa(x9))
min max (un(S. = EE 10.1 m.00-pp x p ( (-1=0)
0
Adding real data. Following alternative definition of RealUID loss from Lemma 3, we can incorporate real data in Normalized UID loss (36) as well. We need to split two terms in the linearized form (35) into generated and real data parts with weights a, (1 a) and B, (1 (1- B).
Definition 4. We introduce Normalized RealUID loss LR. (.. (S.P)) on generated data v E P(RD) with coefficients a, (0,1]:
[i(0)) - Se(a9)
ca.9
(f. = E Wo
(1157((0 (x f((t)) a.. (z - 3 3- (2/ (29
(1t) (o0t))ff(a1) fe(17)
(177 (((13)1 (1 - a) (z())) (1 - 3) 57 (x
27 07-0p7 (-10%)
Similar to the proof of RealUID distance Lemma we can show that min- max optimization of Normalized RealUID loss minimizes the non-squared n-oormm between the similar weighted student Ie and teacher f functions:
8 8 max UID (f, po) = EE Bui [[((33 aa) a) + f
(((())
1 (ax0i)) p(()
p7 (xt)
This distance attains minimum when r = po, justifying the procedure.
A.5 DMD APPROACH WITH REAL DATA
Distribution Matching Distillation (Luo et al., 2023; Wang et al., 2023; Yin et al., 2024b;a) (DMD) approach distills Gaussian diffusion models with forward process Ia = 2n + oec, ~ N I).
This approach minimizes KL divergence E.- llet)l EoT, log
pt (x
between the generated data pf and the real data pł- The authors show that the true gradient of Bo-(o,,r\DL [pP)) w.r.t. 0 can be computed via the score functions:
dD (vl lPpt)
Egupp Barnp,..fGG()..oo Gf==()))a( (Vag In (x()) Vag In (x))) dGo(2))
Ee-10.11
d0
d0
Then, this true gradient is estimated with the teacher score function s* == arg min, LDSM p6) and student score s arg min, (DSM
(s, pb) obtained via minimizing DSM loss (1):
dD KL (ve lP:)) Eo (0
(o9(x))) st (x,)) de
dGo)
E E2, [0,7]
de
The final algorithm alternates updates for the fake model and the generator similar to SiD approach.
We would like to highlight that DMD) does not fit our UID framework. The UID loss is uniquely determined by its input UM loss. In the case of Diffusson models and DMD, the UM loss is the LDSM(s, loss. With this loss, the resulting UID loss becomes exactly the SiD loss, not DMD.
Adding real data. We investigated a theoretical possibility to incorporate real data into the DMD framework. We found that we use the DSM loss with real data (16) to train the modified student score function St 8,0 arg min, CR. (s. p8) with coefficients
13:
CR.DSM = a E,- Eng wg of (-w)) (neel (Tells
- S (22xx (=)))
gencrated data pi tenn
24

---

Published as a conference paper at ICLR 2026
+ (1-a) Bug pop (1m)) Ins (x])) (x (=5)
real data P6 term
Then apply the generator parameters update based on the KI. divergence between mixed distributions. Lemma 5 (DMD with real data). Consider real data distribution pà PPRDD) and generated by generator Ge distribution p E P(RP). Then, KL divergence between mixed and real data for (0, 1] has the following gradients with modified student score St and teacher score of = arg min, (s
8,0 == arg min, CRinsM P.)
[x
dD (a p + (1 a) Př lp:)
E T. # Go),
a(0):((0)) (x())
dGg de
d8
Although this approach is theoretically justified, it requires coefficients 3 which work poorly for our RealUID, see Table 1. In the proof below, we also show that use of coefficients a F 3 in the fake model loss leads to the total collapse of generator. The proof itself follows (Wang et al., 2023).
Proof. We aim to minimize KL divergence between generated distribution z6 and the real data pb
min p8
((()) == Eewoo.ry [Dri(a pe + (1 - a) pp p[1)p]
First, we use (Wang et al., 2023, Lemma 1) which says that, for any two distributions p.g E P(RP) and point x E RRD. we have
8D (Dkl(qll)) [ar] log ((x) - log p(x) + 1.
dq
Second, for the parameterization 2023, Lemma 2)
x Go(z), 2 and a fixed point ai, we have (Wang et al.,
(xx) [0] = / pl (xal=()) (z)dz. (z))d
allows us to obtain
(p()
5D 5DKL (a (())())))))))))))))))))))))))( (1 (1-a) a) p ( (l)l(t)) [0]]
[0] = E.
5p6
dph
5D (1))
dge
ōp, (x+)
E,
[xrc]
[x]]
[0]
dx.
dq:
ppp
[log(a= (xx) + (1 1 a:) (xx)) log(p; + +1] a. / (l(all=))p (x))d dx
RR,
la p1 (2 + (1 a) pi (x()) -a log(o; (x!)) aa]
((()))
+ (1 - a)) + +a],
Ee,,, log
(37)
p. (x+)
where 2i
Go(z), 2 ai + oic.c No0, I). Finally, take derivative w.r.t. 0 from (37):
(p())
pf (x()
dxf
V
[0] = B
a Vee log
+ (1 a)
P. (xq)
80
(x4)
dGe
E
a V log
(1 a)
pt (x()
00
(1 (22l)//p(()) (x
aGo(z)
E
a
(38)
(x
00
Now, we show how obtain unbiased estimate of this gradient. We minimize the following loss function over the fake model s:
D.D (8, 26)
[2 Exp~pf ag- we) (Talls((0?) <>.21=0))
25

---

Published as a conference paper at ICLR 2026
+ (1-a) B... (rllls(x7)) s5 (x(11=)))1
This loss is equivalent to the following sequence
min
a B......lllllsx/' s|x[()]1 + (1 aa) . (2)
min
a Jo-.o.TT.llls(c.. V logr +1 (1--)) 0.) E [1)s(())) Va; logrt(
rpp
x/mpl
aa (((]))(())) V Vlogp (x))1
pl (1- a) -[(s((())) Vlogg
min E. xprpp
pł px{)))
The optimal solution a. this quadratic minimization for each point 2, and time moment is
(2re) (x+)
log (o + (1 a) Vx, (x;)
a
0.0 (xt)
[(4) ((.) + (1 a))
Thus, we have the following estimate with modified student score a and teacher score (2x) == V o, log pł (zz))
(w.)
V log log pf (xx) + (1- Vas (x.)
8t (.0 (xx) (xx)
P. (2x)
Var. log (x+)
(w:) (x,) + (1 - a))
(xx)
(V log p (xt) Vz, log))(.:))
(x))
(m,)
+ -a)
P. (2t)
log (x))
Va, (p) (xt)/pp((()) (m))
p((a))
+ - a)
a (2s)
+ (1 a)
Hence, this estimate completely matches with required gradient (38):
(38) E,,,,= a (s") s (={)). (Ga(z)
00
The use of other coefficients during student score optimization does not work. For the other student scores sf l.a.. == arg min, CR (s. P&), the estimate 8, 9.0.8 (xx) Vz, (g:)5(z)) does not lead to
the necessary difference Zz, log p (Iz) Vz, log pt (z,) = 0, and the optimal generator collapses due to large bias.
B REALUID ALGORITHM FOR FLOW MATCHING MODELS
We provide practical implementation of our RealUID approach for flow matching models in Algorithm 1. In the loss functions, we retain only the terms dependent on the target parameters. For the fake model, reformulate the maximization objective minimization. We use alternating optimization, updating the fake model K times per one student update for stability.
26

---

Published as a conference paper at ICLR 2026
Algorithm 1 Real data modified Unified Inversion Distillation (RealUID) for Flow Matching
Input: teacher drift u", student generator Gg. fake drift Uy, real data pÙ. coefficients a, E (0, 1), generator update steps K, number of iterations N, batch size B, fake drift minimizer Optsr- generator minimizer Optgen latent distribution p noise distribution p1. do
1: for 0,... N
2: Sample generated batch {~6. Go(z:)) |= p² and noise batch {01,i) ~ Pri
3: Sample time batch (4))²1 ~ Uni f Jorm [0, 1] and calculate =f, = (1 - 4, 4,)000. + 4415
4: if student step (n%(K + 1) + 0) then
5: Sample real data batch {= ~ Pp and calculate T (1 f:)= ; + 1,11, 1
6:
Update fake drift parameters & via minimizer Opta step with gradients of
1
allu (t,, (t,, 14. 2g
=010 )1 +1 -
- -x1. -0,i
1-
(21, 200 )1?
B
7:
else
8:
Update generator parameters 9 via minimizer Optgen step with gradients of
alllu((t, (t,, (t,, a.. 2 ( - -1,)11 - a ulllugg( (ts, x...)) ) B [a1, - 28.2017
B
1=1
9: end if
end for
C UNIFIED INVERSE DISTILLATION WITH REAL DATA FOR BRIDGE MATCHING AND STOCHASTIC INTERPOLANTS
C.I BRIDGE MATCHING
Bridge Matching (Liu et al., 2022; Peluchetti, 2023) is an extension of diffusion models specifically design solve data-to-data, e.g., image-to-image problems. Typically, the distribution pr is the distribution of corruptedd data" and is the distribution of clean data, furthermore, there is some coupling of clean and corrupted data π (x0, .T) with marginals (o())) and pT(TT). To construct the diffusion which recovers clean data given a corrupted data, one first needs to build prior process (which often the same forward process used in diffusions):
dx; = fr(2;)dt + g,dw;,
where fe(.) is drift function, St is time-dependent scalar noise scheduler and w, is standard Wiener process. This prior process defines conditional density pt (ate (m)) and the posterior density (e æT) called "diffusion bridge". To recover po from pr, can use reverse-time SDE with a reverse-time Wiener process Wy:
dæ (fi(z)) - oi ux(x)) dt g.dwe,
where the drift u. (x::) learned via solving of the bridge matching problem:
CuM(U, T) pr
(x)) V., log (z
(39)
However, this reverse-time diffusion general does not guarantee that the produced samples come from the same coupling = used for training. This happens only if << .r) solves entropic optimal transport between po and pr. To guarantee the preservance of the coupling π (xip, xT), there exists another version of Bridge Matching called either Augmented Bridge Matching or Conditional Bridge Matching (De Bortoli et al., 2023), which differs only by addition a condition on 2y to the trainable drift vy (x
(U T) = Ee--oo11, (.==
[Ilve(Te, ØT) Va, log
The learned conditional drift u(re, 2T) is then used for sampling via the reverse-time SDE starting from a given 2p Pr:
dr. (fe(zz) 36 ux(xi, (T)) dl. gedw..
27

---

Published as a conference paper at ICLR 2026
C.2 STOCHASTIC INTERPOLANTS
The Stochastic Interpolants framework generalizes Flow Matching and diffusion models, constructing diffusion flow between two given distributions Po and pr. To do so, one needs consider the interpolation between any pair of points (Fo, TT) which are sampled from the coupling π (Zo with marginals po and pr. The interpolation itself given by formula
= zo, TT) 7c E E~ I,,,1, E [0,
where /(, Io, IT) zo. 50, interpolant defines a conditional Gaussian path Pt (z₁ (rr) Note that in the original paper (Albergo al., 2023), the authors consider the time interval jo. 1), but those two intervals are interchangeable by using a change of variable t' 4. Thus, the ODE interpolation between po and pr given by:
Jo, IT) IT, To YT 0 and 7: for all & E (0, This .T))
dxa = (x(
Po:
where ue (I) : B[ = x] E[0, 20, feeler x| is the unique minimizer of the quadratic objective:
Ls((U, (e0, (II (0.1 zo, IT) xre))l?]
(40)
The authors also provide way of matching the score and the SDE drift of the reverse process by solving similar MSE matching problems.
C.3 OBJECTIVE FOR GENERAL DATA COUPLING
The essential difference of Bridge Matching and Stochastic Interpolants from diffusion models and Flow Matching with Gaussian path is that they additionally introduce coupling क ( Tr) used to sample 2n and can work with conditional drifts.
This difference can be easily incorporated the generator Gg to output not the samples from the initial distribution pg. but from the coupling T One can do by setting n (2o, xT) = ( (x0 (TT), where conditional data distribution (2( (Zolar) (ar) is parametrized by the student generator Gg Z x R + RR" conditioned on sample IT ~ Pr. This approach is specifically used in Inverse Bridge Matching Distillation (IBMD) (Gushchin et al., 2024). Hence, our Universal Inverse Distillation objective can be written just by substituting student distribution p by student coupling n², substituting real data po by real data coupling x² and adding extra conditions.
RealUID distillation framework just by parametrizing
Definition S. We define Universal Matching loss with real data for general coupling on generated data coupling T² P(RDD RRD) with a, 3 10:
3 LR-UM-coup (S, 0)=====- a Ee-jo,
.T) & - 1T)
oy propp all (-br)) at ppmpi la ((g)))))
generated data 8
1 - 3
+ (1 a) Ee E by ppr an way (-Iar) ppp (- (-Ioo.arr)
(1fe(2z, - 1 a
fi (xi 1T
real data x²" term
And the corresponding Universal Inverse Distillation loss with real data for general coupling is:
min max Lo- (f, x == 8
R-UUM-coupp (8* x0) - C Co- (f, x0)
8
In case of coupling match = = the RealUID loss for couplings attains its minimum, i.c.,
min max C-UUD--oupp (f, 0²) = min o CRRMMM--oo (f", x )- min /CR--UM-coop (f,
0
CR-UM--oup (f, - min 8 nT RRUMssspp (5*)
Ca
C-UUM--coup (f,T") = 0.
28

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
D.1 CIFAR-I0 DISTILLATION FROM SCRATCH
Codebase, dataset and teachers. Building on the reference codebase and network architectures of (Tong et al., 2024), we implement the training algorithm described in our Algorithm 1. We evaluate the resulting approach CIFAR-IO (32x32), under both conditional and unconditional settings, benchmarking against established baselines. The codebase implementation is publicly available in
https: hetps://githube com http:///tttnooooo/niiiiiiioo wolooooollllllllooooo fwwwwllowwaa------ www.mmatchiing..
Note that in this codebase, the time flow is reversed, i.e., the time t = 0 corresponds to the pure noise, while the time Flow Matching checkpoints from the above repository. For conditional setup, we slightly modify the original code and train our own teacher. Our trained checkpoints, along with the code, are located in
is the real data. As an unconditional teacher, we use already trained Conditional
Training hyperparameters. We train our models with Adam (Kingma & Ba, 2014), using (31, (0, 0.999), learning rate 3 10-5 and 500-step linear warm-up. Similar to Sil) framework (Zhou et al., 2024a), we do not recommend setting momentum B1 * as is crucial for successful convergence in our min-max optimization.
To regulate adaptation between the generator and the fake model, the generator updated once for every K 5 updates of the fake model, following DMD2 (Yin et al., 2024a). While the SiD framework leverages an EDM architecture (Karras et al., 2022) and updates the generator after a single update of the fake model (K 1), our RealUID approach becomes unstable for values K < due to the different (Tong et al., 2024) architecture.
We do not dropout in generator and fake models. We set batch size of 256 and maintain EMA of the generator parameters with decay 0.999 (Hunter, 1986). Additionally, at each optimization step we apply & gradient-norm clipping with threshold 1.0 both the generator and the fake model.
Training time. All distillation experiments were trained for 500,000 gradient updates, correspond- ing approximately 5 days. The experiments were executed on a single Ascend910B NPU with 65 GB of VRAM memory.
Generator parameterization and models initialization. We parameterize generator G-(-) using a time-dependent U-Net go (0, -) with fixed time input / = 0 and one-step integration scheme:
Go(z) = + go (0)
We initialize the model go with teacher model, and the fake model with random weights. Empirically, we observe that this initialization strategy lead to improved performance on the considered datasets.
GAN details. We integrate GAN loss into framework in line with ? and DMD2 (Zhou et al., 2024a; Yin et al., 2024a). In the original setup of (Zhou et al., 2024a), the adversarial loss employs a coefficient ratio of / = 102 (see Table 6 in Zhou et al. (2024a)), a choice that poses practical difficulties due the extreme imbalance between the generator and discriminator losses. To mitigate this issue, we adopt the formulation of (Yin et al., 2024a), where the ratio is 3. and evaluate different coefficient scales (see the results in Table 1). Additionally, we can select the range of times within which adversarial loss is applied between noised generated and real data samples. We found that the best choice not to take only clear real data or the whole interval [0.1], but rather to take the range of not severely corrupted data, namely times from 0.8 to 1.
Evaluation protocol. We evaluate image quality using the Fréchet Inception Distance (FID; Heusel et al., 2017), computed from 50,000 generated samples following (Karras al., 2022: 2020; 2019). In line with SiD (Zhou et al., 2024b), we periodically compute FID during distillation and select the checkpoint achieving the minimum value. To ensure statistical reliability, repeat the evaluation over 3 independent runs, rather than 10 as in SiD, because the empirical variance of FID in our experiments below 0.01.
29

---

Published as a conference paper at ICLR 2026
Efficiency comparison. In terms of efficiency, RealUID leverages lightweight architecture based on (Tong et al., 2024). Therefore, as summarized in Table achieves nearly 2x faster inference, lower memory usage, and reduced model size compared recent distillation approaches (Zhou et al., 2024b;a; Huang et al., 2024).
Methods
Inference Time (ms)
# Total Puram (M)
Max GPU Mem Alloc (MB)
Max GPU Mem Reserved (MB)
RealUID (Ours)
18.636
36.784
165
172
SID (2hoo al. 2004kug)
(Baung 2024)
30.745
55.734
242
276
Table 5: Inference complexity on an Ascend 910B3 (65 GB) NPU. All methods require only 1 NFE. For cach method, we report (i) the mean inference time image (bs=1, fp32), averaged over 10,000 iterations; (ii) the total number of parameters (Millions); and (iii) peak NPU memory usage (maximum allocated and reserved, in MB). Best values are bolded.
D.2 CIFAR-10 DISTILLATION FINE-TUNING
This section presents an ablation study of the fine-tuning stage over the boss-balancing coefficients for GANs and our RealUID CIFAR-10. In this stage, the generator is initialized from the best- performing checkpoint obtained during training from scratch of the corresponding framework, while the fake model is initialized from the teacher model. In the unconditional setup, the best configuration RealUID with (a 0.92, 3 0.94) and FID 2.22, and GAN with (> 0.3, X 1) and FID 2.29. In the unconditional setup, it is RealUID with (a = 0.98, 3 0.96) and FID 2.02, and GAN with (A adv 0.3, > 1) and FID 2.12. Fine-tuning then proceeds with new values orr and 9PT for RealUID and x and X for GANs. The results are summarized in Table 6.
Per our RealUID and fine-tuning scales (A APr Table Ablation of fine-tuning parameters (AFT, apr for GANs for unconditional (left) and conditional (right) generation. All values report FID J. where lower better. The mark "_" indicates that configuration is infeasible, and the mark converge. Best results for each method are bolded.
shows that the method did not
ner 5z
0.92
0.94
0.96
1.0
1.02
1.04
Bx
0.98
1.0
1.06
1.08
0.92
1.99
1,98
2,02
88
2.04
2.04
2,02
0.92
1.92
1.91
1.99
1.96
1.94
1.92
0.94
2.02
2,02
2.04
2,07
2.06
0.94
1.92
1.90
1.88
1.96
1.91
0.96
2006
2.04
2.09
2,08
0.96
1.93
1.94
1,87
1.96
0.98
2.07
2,05
2,07
0.98
1.91
1.95
1,95
Apr
0.1
0.3
1.0
5.0
25.0
100.0
Xpp
0.1
0.3
1.0
5.0
25.0
100.0
AD
0.3
1.0
3.0
15.0
75.0
300.0
FT
0.3
1.0
3.0
15.0
75.0
300.0
FID.
2.25
2.10
2.12
FID4
1.94
1.88
2.04
We observe that fine-tuning highly sensitive the choice of factor frr which still brings the main opr
impact. The best factors from scratch (Table 1), i.e., fine-tuning relies more on information from real data rather than on guidance from a teacher. Meanwhile, configurations closer to 1.0 are unstable, underscoring the crucial role of real data. In the case of GANs, small adversarial losses similarly fail to converge, and only high scales which particularly emphasize real data achieve improvement.
fer 0.94 SL 1.06 are much farther from 1.0 compared to training oUr
our
Training details. We run fine-tuning with smaller learning rate 1 x 10-5 and without warm-up. All other details remain the same as described in Appendix D.I for training from scratch.
Training time. All fine-tuning experiments took a little more than I day, starting from the best distillation checkpoints. The experiments were executed on single Ascend9 10B NPU with 65 GB of VRAM memory.
conducted for 100,000 gradient updates, which
D.3 CELEBA DISTILLATION
In this section, we present the results the same ablation study from (64.2) on the CelebA dataset with higher 64 x 64 resolution (Liu al., 2015). The results summarized in Table 7. Similar to
30

---

Published as a conference paper at ICLR 2026
Table I for CIFARIO, the same pairs of coefficients with P/a 1.02 or A/4 0.98 yield significant
improvement in quality over the baseline (a = 1.0, 3 = 1.0), reaching level comparable to GANs.
0.96
0.98
1.00
1.02
1.04
Go adv
A D adv
FID
(4)
0.88
1.03
1.08
1.36
1.14
1.45
0.1
0.3
1.14
0.90
1.06
1.03
1.38
1.06
1.48
0.3
1.18
0.92
1.12
1.04
1.28
1.10
1.69
0.94
1.13
1.03
1.18
1.10
1.64
1.10
0.96
1.24
1.11
1.25
1.07
1.69
5
15
1.04
0.98
1,65
1.26
1.22
1.29
25
75
3.31
1.0
1.20
Table 7: Ablation studies our (a, ()) parameters in the left table and adversarial weighting parameters (Aude Awav) in the right table for CelebA, 800,000 iterations. The baseline RealUID (a - 1.0, 3 1.0) does
not use real data. Configurations that substantially outperform the baseline are highlighted. All values report FID J. where lower better. The best configuration bolded. The mark "_" denotes infeasible configurations.
Training hyperparameters and details. We take the same architecture (Tong et al., 2024) for the CIFAR-I0 dataset, but adapt it to a larger resolution. We train with Adam (Kingma & Ba, 2014) for 800,000 iterations, using ($1. 32) (0, 0.999). learning rate 5 10-6 and a 500-step linear warm-up. up. Similar to SiD framework (Zhou et al., 2024a), we do not recommend setting momentum 81 * 0 it crucial for successful convergence in our min-max optimization.
To regulate adaptation between the generator and the fake model, the generator is updated once for every K = updates of the fake model, following DMD2 (Yin et al., 2024a). While the Sil) framework leverages an EDM architecture (Karras et al., 2022) and updates the generator after single update of the fake model (K 1), our RealUID approach becomes unstable for values K due to the different (Tong et al., 2024) architecture.
We do not use dropout in generator and fake models. We set batch size 64 and maintain an EMA of the generator parameters with decay 0.999 (Hunter, 1986). Additionally, at each optimization step we apply & gradient-norm clipping with threshold both the generator and the fake model.
All other details remain the same as described in Appendix D.1 for CIFAR-10.
Teacher training. For CelebA, we train our own teacher model based on the official implementation of the conditional flow matching procedure from (Tong al., 2024). We the same pipeline, architectures, and hyperparameters, but with larger networks and different dataset. The adapted code for teacher training and final checkpoints for distillation can be found in our repository:
Fine-tuning. For fine-tuning, we hold the data-free UID baseline (our RealUID with a = 1.0, = 1.0) and all highlighted GAN and RealUID setups from Table 7 for twice as long, i.e., for 1,600,000 iterations. The best-found configurations and results reported in Table and Figure 6. According to it, our RealUID still outperforms data-free UID baseline, reaching the same performance as GANs.
Training time. All experiments were executed on single Ascend910B NPU with 65 GB of VRAM memory. Regular 800,000 gradient updates took approximately 5 days, while longer fine-tuning with 1,600,000 iterations took 10 days.
31

---

Published as a conference paper at ICLR 2026
Table This table presents the results ablation study our RealUID framework, evaluated using the FID metric on CelebA dataset, 1,600,000 iterations. The Teacher Flow model with 100 NFE is reported as reference. The performance of the UID (FGM) baseline without real-data incorporation is indicated in italic. For emphasis, we underline the two counterparts that incorporate real data: the GAN-hased and our RealUID methods. The hest-performing configuration is highlighted in bold. Qualitative results are presented in Appendix D.5.2.
Model
FID (4)
Teacher Flow (NFE=100)
2.46
UID (FGM)
0.96
UID + GAN (A 1.0, A 3.0)
0.87
RealUID (a 0.88, 8 = 0.90) (Ours)
0.89
UID
10
a 0.88, 0.90
6
2.46 (Teathhe Flow)
0.87. (UID GAN).
200
400
600
800
1000
1200
Iterations (x 10a)
Figure 6: Evolution of FID during CelebA distillation for the data-free UID baseline and the best-performing RealUID configuration. The performances Teacher Flow and UID+GAN are indicated by horizontal lines their respective colors.
D.4 FURTHER HYPERPARAMETERS GRIDSEARCH
The primary goal across experiments this paper was to study RealUID framework, focusing on the effects of the coefficients and B, and provide a fair comparison with GANs. For this reason, we kept all other hyperparameters fixed at their standard values. Now that we have identified the optimal settings for RealUID, findings, while the latest hyperparameters sets and training pipelines are described in our repository
explore other hyperparameters. Below, provide list of useful
https:/ /ttp://ggttttbttt com wwom/Davvd-crnt htt.e//ww..cc.ooooooooettttaeaaa
EMA decays. One track not only single EMA decay but range of values, e.g., [0.999, 0.9996, 0.9999], during a single training run. In long-distance training, larger EMA decays can lead to more stable convergence dynamics and better metrics, whether training from scratch or fine-tuning.
D.5 EXAMPLE OF SAMPLES FOR VARIOUS METHODS
This section presents representative sample outputs from various studies conducted within the RealUID framework.
32

---

Published as a conference paper at ICLR 2026
D.5.1 CIFAR-10 10 GENERATED IMAGES
Figure 7: Uncurated samples for unconditional generation by the one-step data-free baseline UID trained on 10..R-10. Quantitative results are reported in Table 2.
33

---

Published as a conference paper at ICLR 2026
Figure 8: Uncurated samples for unconditional generation by the one-step UID + GAN (A 0.3, 2D 1|>E 25, X 75) trained CIFAR-10. Quantitative results reported in Table 2.
34

---

Published as a conference paper at ICLR 2026
Figure Uncurated samples for unconditional generation by our one-step RealUID (a 0.92, 8 0.94 OFF = 0.92, 3FT = 0.86) trained on CIFAR-10. Quantitative results are reported in Table 2.
35

---

Published as a conference paper at ICLR 2026
Figure 10: Uncurated samples conditional generation by the one-step data-free baseline UID trained on CIFAR-10. Quantitative results are reported in Table 2.
36

---

Published as a conference paper at ICLR 2026
Figure 11: Uncurated samples for conditional generation by the one-step UID + GAN (A 0.3, 2D ||> 25, D 75) trained CIFAR-10. Quantitative results reported in Table 2.
37

---

Published as a conference paper at ICLR 2026
Figure 12: Uncurated samples for conditional generation by our one-step RealUID (a 0.98, 8 0.96 OFF 0.96, 3FT = 0.92) trained on CIFAR-10. Quantitative results are reported in Table 2.
38

---

Published as a conference paper at ICLR 2026
D.5.2 CELEBA GENERATED IMAGES
Figure 13: Uncurated samples by the one-step data-free baseline UID trained CelebA. Quantitative results are reported in Table 7.
39

---

Published as a conference paper at ICLR 2026
Figure 14: Uncurated samples by the one-step UID + GAN (A 1.0, AD- -3.0) trained on CelebA. Quantitative results are reported in Table 7.
40

---

Published as a conference paper at ICLR 2026
Figure 15: Uncurated samples by our one-step RealUID (a = 0.88, B 0.9) trained on CelebA.
Quantitative results are reported in Table 7.
41


Reviews:
[{"review_id": "2B2QiavvKp", "reviewer": "Reviewer_oamB", "summary": "This paper addresses a core problem in generative model distillation: existing data-free methods (e.g., SiD, FGM) cannot leverage real data to improve the student model, and the only current way to do so relies on complex and unstable GANs.The paper's contributions are twofold:\n1. **Theoretical Unification (UID)**: It introduces a general min-max distillation framework (UID) and demonstrates that recent methods like SiD and FGM are special cases of this unified structure. \n2. **Methodological Innovation (RealUID)**: This is the paper's main contribution. The authors propose RealUID, a novel, non-adversarial (GAN-free) mechanism to integrate real-data supervision. This is achieved by cleverly redesigning the inner loss function of the min-max game ($\\mathcal{L}_{R-UM}^{\\alpha,\\beta}$) to include terms from both the generated distribution ($p_0^\\theta$) and the real-data distribution ($p_0^*$).\n\nExperiments (on CIFAR-10) show that RealUID not only outperforms the data-free baseline in FID but also achieves comparable or superior results to GAN-based methods, all while significantly accelerating convergence and avoiding the complexities of adversarial training.", "questions": "1. **On Scalability (W1)**: Can the authors provide even preliminary results on a single higher-resolution dataset (e.g., FFHQ-64 or CelebA-256)? How does RealUID converge, and are the $\\alpha, \\beta$ hyperparameters stable across different resolutions?\n2. **On $K=5$ (W3)**: The experimental details state the fake model $f$ is updated 5 times for every 1 generator update ($K=5$). How critical is this ratio? Does the RealUID framework become unstable at $K=1$? This seems to hide a 5x computational cost for the fake model's training loop relative to the generator's.\n3. **On $\\mathcal{L}_{R-UM}^{\\alpha,\\beta}$ Coefficient Design (S2)**: The specific coefficients $\\frac{\\beta}{\\alpha}$ and $\\frac{1-\\beta}{1-\\alpha}$ in Eq. 17 are clearly deliberate. The paper claims this is to ensure $f=f^\\*$ is the optimal solution when $p_0^\\theta = p_0^*$.\nIs this specific mathematical form the only one that satisfies this property? Would the framework fail if simpler, unscaled $L_2$ losses were used (e.g., setting $\\beta/\\alpha = 1$ and $(1-\\beta)/(1-\\alpha) = 1$)? This seems highly relevant given the hyperparameter sensitivity seen in (W3).\n4. **On \"Correcting\" the Teacher (S2)**: A potential benefit of RealUID is that the fake model $f$ sees both the teacher $f^\\*$ and the real data $p_0^\\*$. If the teacher $f^\\*$ is flawed (e.g., suffers from mode collapse) but $p_0^\\*$ is complete, $f$ faces a conflict during its training: should it match $f^\\*$ or the function implied by $p_0^\\*$?  Can RealUID truly \"correct\" a flawed teacher, or would this conflict lead to training instability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Significant Methodological Innovation (GAN-free Supervision)**: The paper's strongest contribution is RealUID, a novel and elegant non-adversarial method to incorporate real-data supervision into the distillation process. This directly addresses a major pain point of previous work, avoiding the need for extra discriminator architectures and the instability of adversarial training.\n2. **Clever Loss Function Design ($\\mathcal{L}_{R-UM}^{\\alpha,\\beta}$)**: The mechanism of innovation is particularly commendable. Instead of simply adding a new top-level loss, the authors redesigned the inner-loop loss of the min-max game (Eq. 17). The \"fake model\" $f$ is forced to optimize on both $p_0^\\theta$ (generated data) and $p_0^*$ (real data) during its $\\max_f$ step. This allows $f$ to serve as a \"bridge,\" passing the supervisory signal from the real data to the student generator $G_\\theta$ in a natural, non-adversarial way.\n3. **Strong Theoretical Unification (UID)**: Prior to RealUID, the paper's UID framework is a valuable theoretical contribution in its own right. It unifies seemingly separate works (SiD, FGM) under a single min-max perspective using a clear and general mathematical derivation (the linearization technique). It clarifies the underlying game structure: the \"fake model\" $f$ (the $\\max_f$ player) tries to approximate the \"theoretical student function\" $f^\\theta$, while the generator $G_\\theta$ (the $\\min_\\theta$ player) tries to minimize the distance between $f^\\theta$ and the teacher $f^*$.\n4. **Excellent Empirical Efficiency**: The experimental results strongly support the practical utility of this new method. RealUID not only achieves strong FID scores but also demonstrates significantly faster convergence (approx. 3x) compared to the data-free baseline (Fig. 2), proving that this GAN-free supervision is both effective and efficient。", "weaknesses": "1. **Severely Limited Experimental Scope (Main Flaw)**: All experiments are conducted exclusively on the CIFAR-10 (32x32) dataset. While resource constraints are mentioned, this leaves the scalability of RealUID to higher resolutions (e.g., 256x256 or 512x512) completely unknown. The need for distillation is most acute for these larger, slower models.\n2. **Not State-of-the-Art (SOTA) Performance**： While the results are strong, the best-reported FID (e.g., 1.91 for conditional generation) still lags behind SOTA adversarial distillation methods (e.g., $SiD^2A$ @ 1.39). The authors hypothesize this is due to their lightweight architecture and teacher model, but this is unproven. It is unclear if the performance gap is due to the architecture or an inherent ceiling of the RealUID method itself compared to adversarial approaches.\n3. **New Hyperparameter Sensitivity**: RealUID avoids GAN instability but introduces new hyperparameters $\\alpha$ and $\\beta$. The fine-tuning ablation (Table 6) shows high sensitivity to these, with many configurations failing to converge (marked with \"-\"). This may offset the claimed simplicity, as tuning $\\alpha$ and $\\beta$ could be as difficult as tuning a GAN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qtsSRbaqjh", "reviewer": "Reviewer_ha8b", "summary": "The paper tackles the challenge of distilling diffusion/flow-based models into fast one-step generators. The authors build on works such as SiD and FGM, which optimize the student by minimizing the $\\ell_2$ distance between the teacher and student score/velocity functions, referred to as $f$-functions in the paper. The authors propose a simple but clever modification to the training objective of the student $f$-function that includes both real and generated data, allowing for an alternative way to incorporate real data into the distillation pipeline of these methods. They theoretically justify this change by proving that it has a proper global minimizer and experimentally demonstrate its usefulness on CIFAR-10, achieving impressive results.", "questions": "- Why does the setting too low values for the coefficients $\\alpha,\\beta$ lead to collapse? Is this just experimentally seen or is there a theoretical justification for it?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The premise of the paper is very interesting and important. Distillation methods that rely on Reverse-KL-like objectives (SiD, FGM, DMD, …) are widely used in practice but are known to suffer from a diversity collapse problem. This issue arises mainly because the loss is computed only on student samples (reverse-KL). Therefore, being able to effectively incorporate real data into training without resorting to GANs (which have famously unstable training dynamics) is potentially very valuable.\n- The reasoning that leads to the RealUM and RealUID objectives is clever.\n- The supplementary material is extensive and includes additional variants of the approach, as well as theorems and proofs.", "weaknesses": "- As mentioned above, loss of diversity is a common drawback of distillation approaches with Reverse-KL-like objectives. Incorporating real data into the training pipeline can, in principle, fix this issue to some extent. Therefore, including diversity metrics in the experiments would be very helpful and could strengthen the paper’s argument.\n- The paper broadly argues that recent diffusion distillation methods either rely solely on synthetic data or use real data only in GAN-like objectives. While this is true for most Reverse-KL-like methods, it does not hold for consistency- or flowmap-based methods[1,2,3,4,5]. Including a discussion of such methods and adding them as additional baselines would help the paper better position itself within the literature.\n- Although the authors explicitly note that the limited scope of the experiments is due to computational constraints, the fact remains that the proposed approach is primarily evaluated on a small dataset using a lightweight network architecture. As a result, it is unclear whether the conclusions drawn will generalize to larger datasets and models. For example, as shown in Table 4, the proposed approach performs considerably worse than prior SiD and SiDA baselines.\n- The method seems highly sensitive to the values of $\\alpha,\\beta$, as shown in Table 1.\n\n\n[1] Song, Yang, et al. \"Consistency models.\" (2023). \\\n[2] Lu, Cheng, and Yang Song. \"Simplifying, stabilizing and scaling continuous-time consistency models.\" arXiv preprint arXiv:2410.11081 (2024). \\\n[3] Frans, Kevin, et al. \"One step diffusion via shortcut models.\" arXiv preprint arXiv:2410.12557 (2024). \\\n[4] Geng, Zhengyang, et al. \"Mean flows for one-step generative modeling.\" arXiv preprint arXiv:2505.13447 (2025). \\\n[5] Sabour, Amirmojtaba, Sanja Fidler, and Karsten Kreis. \"Align Your Flow: Scaling Continuous-Time Flow Map Distillation.\" arXiv preprint arXiv:2506.14603 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgs4zMusQN", "reviewer": "Reviewer_WywQ", "summary": "This paper introduces the RealUID framework, with two primary contributions: (1) it theoretically unifies prior distillation methods for different matching models (e.g., SiD, FGM) and extends them to Bridge Matching and Stochastic Interpolants via a novel UID loss based on a linearization technique; (2) it proposes a novel method for incorporating real data into the distillation process without relying on GANs. Experiments on CIFAR-10 provide empirical support for these theoretical claims.", "questions": "1.Beyond mathematical re-formulation, what are the new insights and practical improvements uniquely enabled by your unified UID loss perspective that were not apparent or easily achievable within the prior frameworks.\n2.Is there any adaptive criterion for tuning α and β? What is the rationale behind using different parameter values during the training versus fine-tuning stages? Do these hyperparameters generalize across datasets, or do they require costly re-tuning for each new dataset?\n3.Why does RealUID use a lightweight model architecture different from the baseline methods? Could increasing the parameter count further improve performance?\n4.Could the authors provide additional results of RealUID on other datasets and models? This is my primary concern.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The authors constructed a concise unified form through a linearization technique, and the theoretical derivation is fundamentally sound; \n2.The extension of the framework to bridge matching and stochastic interpolants is commendable; \n3.Replacing complex GANs with simple loss interpolation to incorporate real data is a straightforward yet effective approach.", "weaknesses": "1.The proposed unified framework appears more as a formal re-formulation than a substantive theoretical breakthrough. This raises concerns that the core contribution may essentially be a \"re-parameterization\" of prior methods like SID and FGM, thereby undermining its claimed novelty and independence.\n2.The proposed RealUID method suffers from a critical dependence on two newly introduced hyperparameters, α and β. Firstly, the selection of these parameters lacks clear theoretical guidance, and their design appears more as a heuristic afterthought rather than a principled derivation. Secondly, as evidenced in Table 1 and Appendix Table 6, the method exhibits extreme sensitivity and pronounced randomness to these parameters.\n3.Lemma 2 relies on the assumption that the data probability density is positive everywhere (pt∗(xt) > 0, ∀xt, t). This assumption typically fails in high-dimensional spaces.\n4.The most prominent weakness of this work is its narrow experimental setup:\n1)The evaluation is narrowly based on small-scale datasets like CIFAR-10. The absence of results on standard large-scale, high-resolution benchmarks such as AFHQ and ImageNet severely limits the empirical grounding of the paper's claims.\n2)The experimental evaluation is confined to  flow-based models, omitting tests on diffusion models. This limitation raises serious concerns about the method's competitive advantage and the framework's generalizability.\n3)The experiments lack comparisons against state-of-the-art baselines from 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nbmyNUvasv", "reviewer": "Reviewer_pZHj", "summary": "This is a paper about generative modeling, in which the main contribution is the incorporation of real data into the loss function or the training process. The goal is to overcome the slow sampling process in diffusion models and flow matching models, which are collectively called match models in the paper. The proposed method incorporates real data into the distillation loss, thus, it removes the need to use GAN for assessing the adversarial loss. Experiments were performed on CIFAR-10 and showed that the proposed method did have faster inference.", "questions": "Please see weaknesses.\nIt is unclear why on line 294, the paper requires that \"A key constraint is that the loss must still yield the same teacher upon minimization on the real data.\" With the inclusion of real data, shouldn't the teacher model be somehow different from the teacher model without using the real data? \nThe goal of equation 13 is to maximize it, which is the last term of equation 10. Does maximizing equation 13 guarantee that equation 10, as a whole, will be minimized as equation 10 has two other terms that depend on f_t^star and f_t^theta?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Stengths include the incorporation of real data into the distillation loss, thus avoiding the need for extra adversarial loss, which in turn, requires a discriminator network as in GAN. This is a logic and novel idea of implementing the distillation loss, as compared with existing approaches. Another strength is the reduced inference time, as shown in Table 3, second row from the bottom.", "weaknesses": "It seems Equation 17 is the key formula of the work as it defines the new distillation loss, which weighs the loss or contribution from real data by hyperparameters alpha and beta. It is unclear why alpha and beta are, first, confined to a small range such as between 0.94 and 1.0, and, second, why alpha and beta are so close to each other in most cases. What does that mean for understanding the distillation loss? Does it mean that the contribution from the real data is typically given a small weight since it is multiplied by (1-alpha)? \nIn the ablation study, Table 2, second row, when alpha = 1, beta=1 in this case, shouldn't the RealUID's performance equal that of the reference (the first row, Teacher Flow) since according to equation 17, when alpha=1, the second term is 0?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)