Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)

## Abstract

Abstract While achieving exceptional generative quality, modern diffusion, flow, and other matching models suffer from slow inference, as they require many steps of iterative generation. Recent distillation methods address this by training efficient one-step generators under the guidance of a pre-trained teacher model. However, these methods are often constrained to only one specific framework, e.g., only to diffusion or only to flow models. Furthermore, these methods are naturally data-free, and to benefit from the usage of real data, it is required to use an additional complex adversarial training with an extra discriminator model. In this paper, we present RealUID , a universal distillation framework for all matching models that seamlessly incorporates real data into the distillation procedure without GANs. Our RealUID approach offers a simple theoretical foundation that covers previous distillation methods for Flow Matching and Diffusion models, and is also extended to their modifications, such as Bridge Matching and Stochastic Interpolants.

## 1 Introduction

In generative modeling, the goal is to learn to sample from complex data distributions (e.g., images), and two powerful paradigms for it are the Diffusion Models (DM) and the Flow Matching (FM) models. While they share common principles and are even equivalent under certain conditions (Holderrieth et al., 2024; Gao et al., 2025), they are typically studied separately. Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) transform data into noise through a forward process and then learn a reverse-time stochastic differential equation (SDE) to recover the data distribution. Training minimizes score-matching objectives, yielding unbiased estimates of intermediate scores. Sampling requires simulating the reverse dynamics, which is computationally heavy but delivers high-quality and diverse results. Flow Matching (Lipman et al., 2023; Liu, 2022) instead interpolates between source and target distributions by learning the vector field of an ordinary differential equation (ODE). The field is estimated through unbiased conditional objectives, but the resulting ODE often has curved trajectories, making sampling costly due to expensive integration. Beyond these, Bridge Matching (Peluchetti, 2023; Liu et al., 2022b) and Stochastic Interpolants (Albergo et al., 2023) generalize the framework and naturally support *data couplings*, which are crucial for data-to-data translation.
Since all of the above optimize *conditional matching* objectives to recover an ODE/SDE for generation, we refer to them collectively as *matching models*.

Despite their success, matching models share a major drawback: sampling is slow, as generation requires integrating many steps of an SDE or ODE. To address this, a range of distillation techniques have been proposed to compress multi-step dynamics into efficient one-step or few-step generators. Although matching models follow a similar mathematical framework, many distillation works consider only one particular framework, e.g., only Diffusion Models (Zhou et al., 2024a; b), Flow Matching (Huang et al., 2024), or Bridge Matching (Gushchin et al., 2025). Furthermore, these distillation methods are data-free by construction and cannot benefit from the utilization of real data without using additional GAN-based losses. Thus, the following problems remain:

- 1.
Similar distillation techniques developed separately for similar matching models frameworks.
- 2.
Absence of a natural way to incorporate real data in distillation procedures (without GANs).

Contributions. In this paper, we address these issues and present the following main contributions:

- 1.
We present the Universal Inverse Distillation with real data (RealUID) framework for matching models, including diffusion and flow matching models (§[3](https://arxiv.org/html/2509.22459v1#S3)) as well as Bridge Matching and Stochastic Interpolants (Appendix [C](https://arxiv.org/html/2509.22459v1#A3).). It unifies previously introduced Flow Generator Matching (FGM), Score Identity Distillation (SiD) and Inverse Bridge Matching Distillation (IBMD) methods (§[3.2](https://arxiv.org/html/2509.22459v1#S3.SS2)) for flow, score and bridge matching models respectively, provides simple yet rigorous theoretical explanations based on a linearization technique, and reveals the connections between these methods and inverse optimization (§[3.3](https://arxiv.org/html/2509.22459v1#S3.SS3)).
- 2.
Our RealUID introduces a novel and natural way to incorporate real data directly into the distillation loss, eliminating the need for extra adversarial losses which require additional discriminator networks used in GANs from the previous works (§[3.4](https://arxiv.org/html/2509.22459v1#S3.SS4)).

## 2 Backgrounds on training and distilling matching models

We describe the Diffusion Models and Flow Matching frameworks (§[2.1](https://arxiv.org/html/2509.22459v1#S2.SS1)) and distillation methods for them (§[2.3](https://arxiv.org/html/2509.22459v1#S2.SS3)).
Then, we discuss how real data can be added to distilling methods via GANs (§[2.4](https://arxiv.org/html/2509.22459v1#S2.SS4))

Preliminaries. We work on the $D$-dimensional Euclidean space $\mathbb{R}^{D}$. This space is equipped with the standard scalar product $\langle x,y\rangle=\sum_{d=1}^{D}x_{d}y_{d}$, the $\ell_{2}$-norm $\|x\|=\sqrt{\langle x,x\rangle}$ and $\ell_{2}$-distance $\|x-y\|,\forall x,y\in\mathbb{R}^{D}.$ We consider probability distributions from the set $\mathcal{P}(\mathbb{R}^{D})$ of absolutely continuous distributions with finite variance and support on the whole $\mathbb{R}^{D}$.

### 2.1 Diffusion and Flow Models

Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) consider a forward noising process that gradually transforms clean data $p_{0}$ into a noise $p_{T}$ on the time interval $[0,T]$:

$$ $dx_{t}=f_{t}\cdot x_{t}dt+g_{t}\cdot dw_{t},\quad x_{0}\sim p_{0},$ $$

where $f_{t}$ and $g_{t}$ are time-dependent scalars. This process defines a conditional distribution $p_{t}(x_{t}|x_{0})$:

$$ $\displaystyle p_{t}(x_{t}|x_{0})=\mathcal{N}(\alpha_{t}x_{0}|\sigma^{2}_{t}\mathbf{I}),\quad\text{where}$ $\displaystyle\alpha_{t}=\exp\left(\int_{0}^{t}f_{s}\,ds\right),\quad\sigma_{t}=\left(\int_{0}^{t}g_{s}^{2}\exp\left(-2\int_{0}^{s}f_{u}\,du\right)ds\right)^{1/2}.$ $$

Each conditional distribution admits a conditional score function, describing it:

$$ $s_{t}(x_{t}|x_{0}):=\nabla_{x_{t}}\log p_{t}(x_{t}|x_{0})=-(x_{t}-\alpha_{t}x_{0})/\sigma_{t}^{2}.$ $$

The reverse dynamics from the noise distribution $p_{T}$ to the data distribution $p_{0}$ is provided by the following reverse-time SDE:

$$ $dx_{t}=(f_{t}\cdot x_{t}-g^{2}_{t}\cdot s_{t}(x_{t}))dt+g_{t}d\bar{w}_{t},$ $$

where $s_{t}(x_{t})$ is the unconditional score function of $p_{t}(x_{t})\!=\!\int\!p(x_{t}|x_{0})p(x_{0})dx_{0}$ given by $s_{t}(x_{t})\!=\!\mathbb{E}_{x_{0}\sim p_{0}(\cdot|x_{t})}[s_{t}(x_{t}|x_{0})]$.
This conditional expectation is learned via denoising score matching:

$$ $\mathcal{L}_{\text{DSM}}(s^{\prime},p_{0})=\mathbb{E}_{t\sim[0,T],x_{0}\sim p_{0},x_{t}\sim p_{t}(\cdot|x_{0})}\left[w_{t}\|s^{\prime}_{t}(x_{t})-s_{t}(x_{t}|x_{0})\|_{2}^{2}\right],$ (1) $$

where $w_{t}$ are some positive weights. The reverse dynamics admits a probability flow ODE (PF-ODE):

$$ $dx_{t}=(f_{t}\cdot x_{t}-g^{2}_{t}\cdot s_{t}(x_{t})/2)dt,\quad u_{t}(x_{t}):=(f_{t}\cdot x_{t}-g^{2}_{t}\cdot s_{t}(x_{t})/2),$ $$

which provides faster inference than the SDE formulation.

Flow Matching framework (Lipman et al., 2023; Liu et al., 2023) constructs the flow directly by learning the drift $u_{t}(x_{t})$. Specifically, for each data point $x_{0}\sim p_{0}$, one defines a conditional flow $p_{t}(x_{t}|x_{0})$ with the corresponding conditional vector field $u_{t}(x_{t}|x_{0})$ generating it via ODE:

$$ $dx_{t}=u_{t}(x_{t}|x_{0})dt.$ $$

Then to construct the flow between the noise $p_{T}$ and data $p_{0}$, one needs to compute the unconditional vector field $u_{t}(x_{t})=\mathbb{E}_{x_{0}\sim p_{0}(\cdot|x_{t})}[u_{t}(x_{t}|x_{0})]$ which generates the flow $p_{t}(x_{t})\!=\!\int\!p(x_{t}|x_{0})p(x_{0})dx_{0}$. It can be done by solving the following Conditional Flow Matching problem:

$$ $\mathcal{L}_{\text{CFM}}(v,p_{0})=\mathbb{E}_{t\sim[0,T],x_{0}\sim p_{0},x_{t}\sim p_{t}(x_{t}|x_{0})}\left[w_{t}\|v_{t}(x_{t})-u_{t}(x_{t}|x_{0})\|_{2}^{2}\right].$ $$

In practice, the most popular choice is the Gaussian conditional flows $p_{t}(x_{t}|x_{0})=\mathcal{N}(\alpha_{t}x_{0},\sigma^{2}_{t}\mathbf{I})$. For this conditional flow samples can be obtained as $x_{t}=\alpha_{t}x_{0}+\sigma_{t}\epsilon$, $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ and the conditional drift can be calculated as $u_{t}(x_{t}|x_{0})=\dot{\alpha_{t}}x_{0}+\dot{\sigma}_{t}\epsilon$.

### 2.2 Universal loss for matching models

From a mathematical point of view, it was shown in (Holderrieth et al., 2024; Gao et al., 2025) that flow and diffusion models basically share the same loss structure. We recall this structure but use our own notation. We call diffusion and flow models and their extensions as matching models.

Matching models work with a probability path $\{p_{t}\}_{t\in[0,T]}$ on the time interval $[0,T]$, transforming the desired data $p_{0}\in\mathcal{P}(\mathbb{R}^{D})$ to the noise $p_{T}\in\mathcal{P}(\mathbb{R}^{D})$. This path is built as a mixture of simple conditional paths $\{p_{t}(\cdot|x_{0})\}_{t\in[0,T]}$ conditioned on samples $x_{0}\sim p_{0}$, i.e., $p_{t}(x_{t})=\int_{\mathbb{R}^{D}}p_{t}(x_{t}|x_{0})p_{0}(x_{0})dx_{0},\forall x_{t}\in\mathbb{R}^{D}.$ The path $\{p_{t}\}_{t\in[0,T]}$ determines the function $f^{p_{0}}:[0,T]\times\mathbb{R}^{D}\to\mathbb{R}^{D}$ which recovers it (e.g., score function or drift generating it). The conditional paths also determine their own simple conditional functions $f^{p_{0}}(\cdot|x_{0})$ so that they express $f^{p_{0}}_{t}(x_{t})=\mathbb{E}_{x_{0}\sim p_{0}(\cdot|x_{t})}f^{p_{0}}_{t}(x_{t}|x_{0})$, where $p_{0}(\cdot|x_{t})$ denotes data distribution $p_{0}$ conditioned on the sample $x_{t}$ at time $t$. Since $f^{p_{0}}$ cannot be computed directly, it is approximated by function $f:[0,T]\times\mathbb{R}^{D}\to\mathbb{R}^{D}$ via minimizing the squared $\ell_{2}$-distance between the functions:

$$ $\|f_{t}(x_{t})-f_{t}^{p_{0}}(x_{t})\|^{2}=\|f_{t}(x_{t})-\mathbb{E}_{x_{0}\sim p_{0}(\cdot|x_{t})}f^{p_{0}}_{t}(x_{t}|x_{0})\|^{2}\propto\mathbb{E}_{x_{0}\sim p_{0}(\cdot|x_{t})}\|f_{t}(x_{t})-f^{p_{0}}_{t}(x_{t}|x_{0})\|^{2}.$ $$

###### Definition 1 .

We define Universal Matching (UM) loss $\mathcal{L}_{\text{UM}}(f,p_{0})$ that takes fake function $f$ and distribution $p_{0}\in\mathcal{P}(\mathbb{R}^{D})$ as arguments and upon minimization over $f$ returns the function $f^{p_{0}}$

$$ $\!\!\mathcal{L}_{\text{UM}}(f,p_{0})\!\!:=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x_{0}\sim p_{0},x_{t}\sim p_{t}(\cdot|x_{0})}\|f_{t}(x_{t})-f^{p_{0}}_{t}(x_{t}|x_{0})\|^{2},f^{p_{0}}\!\!:=\!\operatorname*{arg\,min}{{}_{f}}\mathcal{L}_{\text{UM}}(f,p_{0}),$ (2) $$

where $t\sim[0,T]$ denotes uniform or weighted sampling of time $t$ from the interval $[0,1].$

###### Definition 1 .

### 2.3 Distillation of matching-based models

To solve the long inference problem of matching models, a line of distillation approaches sharing similar principles was introduced: Score Identity Distillation (SiD) (Zhou et al., 2024b), Flow Generator Matching (FGM) (Huang et al., 2024), and Inverse Bridge Matching Distillation (IBMD) (Gushchin et al., 2025), for diffusion, flow, and bridge matching models, respectively.

The Score Identity Distillation (SiD) approach (Zhou et al., 2024b; a) trains a student generator $G_{\theta}:\mathcal{Z}\to\mathbb{R}^{D}$ (parameterized by $\theta$) that produces a distribution $p_{0}^{\theta}$ from a latent distribution $p^{\mathcal{Z}}$ on $\mathcal{Z}$. This approach minimizes the squared $\ell_{2}$-distance between the known teacher score function $s^{*}:=\operatorname*{arg\,min}_{s^{\prime}}\mathcal{L}_{\text{DSM}}(s^{\prime},p_{0}^{*})$ on real data $p_{0}^{*}$ and the unknown student score function $s^{\theta}$:

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}\|s^{\theta}_{t}(x^{\theta}_{t})-s_{t}^{*}(x^{\theta}_{t})\|^{2},\quad\text{ s.t. }s^{\theta}=\operatorname*{arg\,min}{{}_{s^{\prime}}}\mathcal{L}_{\text{DSM}}(s^{\prime},p_{0}^{\theta}),$ (3) $$

where $p_{t}^{\theta}$ is the forward noising process for the generator distribution $p_{0}^{\theta}$.
The authors propose the tractable loss without $\operatorname*{arg\,min}$ and with parameter $\alpha_{\text{SiD}}$ to approximate the real gradients of ([3](https://arxiv.org/html/2509.22459v1#S2.E3)) :

$$ $\displaystyle\mathcal{L}_{\text{SiD}}(\theta)$ $\displaystyle:=$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{z\sim p^{\mathcal{Z}},x_{0}^{\theta}=G_{\theta}(z),x^{\theta}_{t}\sim p_{t}^{\theta}}[-2\omega_{t}\alpha_{\text{SiD}}\|s^{*}_{t}(x_{t}^{\theta})-s^{sg[\theta]}_{t}(x_{t}^{\theta})\|^{2}$ (4) $\displaystyle+$ $\displaystyle 2\omega_{t}\langle s^{*}_{t}(x_{t}^{\theta})-s^{sg[\theta]}_{t}(x_{t}^{\theta}),s^{*}_{t}(x_{t}^{\theta})-s_{t}^{\theta}(x^{\theta}_{t}|x^{\theta}_{0})\rangle],\quad\!s^{\theta}\!=\!\operatorname*{arg\,min}{{}_{s^{\prime}}}\mathcal{L}_{\text{DSM}}(s^{\prime},p_{0}^{\theta}),$ $$

where gradients w.r.t. $\theta$ are not calculated for the variables under stop-gradient $sg[\cdot]$ operator. The SiD pipeline is two alternating steps: first, refine the fake score $s^{sg[\theta]}$ by minimizing the DSM loss ([1](https://arxiv.org/html/2509.22459v1#S2.E1)) on new $p_{0}^{\theta}$ from the previous step. Then, update the generator $G_{\theta}$ using the gradient of ([4](https://arxiv.org/html/2509.22459v1#S2.E4)) with the frozen $s^{sg[\theta]}$. The $\alpha_{\text{SiD}}$ parameter is chosen from the range $[0.5,1.2]$, although theoretically only the value $\alpha_{\text{SiD}}=0.5$ restores true gradient as we show in our paper.

The authors of FGM considered a similar approach, but for the Flow Matching models. Specifically, they also use a generator $G_{\theta}$ to produce a distribution $p_{0}^{\theta}$, but instead of denoising score matching loss, consider conditional FM loss.
The method minimizes the squared $\ell_{2}$-distance between the fields:

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x_{t}\sim p_{t}^{\theta}}\|u^{\theta}_{t}(x_{t})-u_{t}^{*}(x_{t})\|^{2},\quad\text{s.t. }u^{\theta}:=\operatorname*{arg\,min}\text{${}_{v}$}\mathcal{L}_{\text{CFM}}(v,p_{0}^{\theta}),$ (5) $$

where the interpolation path $\{p_{t}^{\theta}\}_{t\in[0,T]}$ is constructed between the noise $p_{T}$ and generator $p_{0}^{\theta}$ distributions.
To avoid the same problem of differentiating through $\operatorname*{arg\,min}$ operator as in SiD, the authors derive a tractable loss whose gradients match those of ([5](https://arxiv.org/html/2509.22459v1#S2.E5)):

$$ $\displaystyle\mathcal{L}_{\text{FGM}}(\theta)\hskip-2.0pt$ $\displaystyle:=$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{z\sim p^{Z},x^{\theta}_{0}=G_{\theta}(z),x^{\theta}_{t}\sim p_{t}^{\theta}}[-\|u^{*}_{t}(x^{\theta}_{t})-u^{sg[\theta]}_{t}(x^{\theta}_{t})\|^{2}$ $\displaystyle+$ $\displaystyle 2\langle u^{*}_{t}(x^{\theta}_{t})-u^{sg[\theta]}_{t}(x^{\theta}_{t}),u^{*}_{t}(x^{\theta}_{t})-u^{\theta}_{t}(x^{\theta}_{t}|x^{\theta}_{0})\rangle],\text{ s.t. }u^{\theta}=\operatorname*{arg\,min}{{}_{v}}\mathcal{L}_{\text{CFM}}(v,p_{0}^{\theta}).$ $$

We consider distillation of matching models working with data couplings such as Inverse Bridge Matching Distillation for Bridge Matching models and Stochastic Interpolants in Appendix [C](https://arxiv.org/html/2509.22459v1#A3). Notably, all these approaches (SiD, FGM, IBMD) are data-free, i.e., they do not use any real data from $p_{0}^{*}$ to train a generator by construction of the used objective functions.

### 2.4 GANs for real data incorporation

FGM and SiD methods exhibit strong performance in one-step generation tasks. However, the generator in these methods is trained under the guidance of the teacher model alone. This means the generator cannot get more information about the real data that the teacher has learned. For example, it cannot correct the teacher’s errors. To address this, recent works (Yin et al., 2024a; Zhou et al., 2024a) propose adding real data via a GAN framework (Goodfellow et al., 2014). In such approaches, the encoder of fake model $f$ is typically augmented with an additional head to serve as a discriminator $D$ with the following adversarial loss:

$$ $\mathcal{L}_{\text{adv}}=\mathbb{E}_{t\sim[0,T]}\bigl[\mathbb{E}_{x^{*}_{t}\sim p_{t}^{*}}\bigl[\ln D_{t}\bigl(x^{*}_{t}\bigr)\bigr]+\mathbb{E}_{x_{t}^{\theta}\sim p^{\theta}_{t}}\bigl[\ln[1-D_{t}\bigl(x_{t}^{\theta}\bigr)]\bigr]\bigr].$ (7) $$

The overall objective in such hybrid frameworks (Zhou et al., 2024a) consists of:

Generator loss:

$$ $\mathcal{L}_{G_{\theta}}=\lambda_{\text{dist}}\mathcal{L}_{\text{FGM/SiD}}^{G_{\theta}}+\lambda_{\text{adv}}^{G_{\theta}}\mathcal{L}_{\text{adv}}^{G_{\theta}},$ (8) $$

Fake model loss:

$$ $\mathcal{L}_{D}=\lambda_{\text{dist}}\mathcal{L}_{\text{FGM/SiD}}^{f}+\lambda_{\text{adv}}^{D}\mathcal{L}_{\text{adv}}^{D}.$ (9) $$

Here, $\lambda_{\text{dist}}$, $\lambda_{\text{adv}}^{G_{\theta}}$, and $\lambda_{\text{adv}}^{D}$ are weighting coefficients for the distillation and adversarial components. Despite empirical gains, the GAN-augmented formulation entails nontrivial costs: it necessitates architectural modifications, such as an auxiliary discriminator head, and inherits the well-known optimization problems of adversarial training, such as non-stationary objectives, mode collapse, and sensitivity to training dynamics.

## 3 Universal distillation of matching models with real data

In this section, we present our novel RealUID approach for matching models enhanced by real data. First, we show that the previous data-free distillation methods can be unified under the single UID framework (§[3.1](https://arxiv.org/html/2509.22459v1#S3.SS1)). Then, we describe how this framework is connected to prior works (§[3.2](https://arxiv.org/html/2509.22459v1#S3.SS2)) and inverse optimization (§[3.3](https://arxiv.org/html/2509.22459v1#S3.SS3)). Using this intuition, we propose and discuss the real data modified UID framework (RealUID) with a natural way to incorporate real data without GANs (§[3.4](https://arxiv.org/html/2509.22459v1#S3.SS4)).

### 3.1 Universal Inverse Distillation

To learn a complex real data distribution $p_{0}^{*}$, one usually trains a teacher function $f^{*}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{*})$ which is then used in a multi-step sampling procedure (Def. [1](https://arxiv.org/html/2509.22459v1#Thmdefinition1)). To avoid time-consuming sampling, one can train a simple student generator $G_{\theta}:\mathcal{Z}\to\mathbb{R}^{D}$ with parameters $\theta$ to reproduce the real data $p_{0}^{*}$ from the distribution $p^{\mathcal{Z}}$ on the latent space $\mathcal{Z}$. The teacher function serves as a guide that shows how close the student distribution $p_{0}^{\theta}$ and the real data $p_{0}^{*}$ are. FGM and SiD methods (§[2.3](https://arxiv.org/html/2509.22459v1#S2.SS3)) train such generator via minimizing the squared $\ell_{2}$-distance between the known teacher function $f^{*}$ and an unknown student function $f^{\theta}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{\theta})$:

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}\|f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t})\|^{2}=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}\|f^{*}_{t}(x^{\theta}_{t})-\mathbb{E}_{x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\|^{2}$ $\displaystyle=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}[\|f^{*}_{t}(x^{\theta}_{t})\|^{2}]-2\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta},x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}[\langle f^{*}_{t}(x^{\theta}_{t}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle]$ $\displaystyle+\underbrace{\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}[\|\mathbb{E}_{x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}[f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})]\|^{2}]}_{\text{not tractable}},$ (10) $$

where $\{p_{t}^{\theta}\}_{t\in[0,T]}$ is the probability path constructed between generator distribution $p_{0}^{\theta}$ and noise $p_{T}$.
The problem is that the final term ([10](https://arxiv.org/html/2509.22459v1#S3.E10)) cannot be calculated directly. This is because it involves the math expectation inside the squared norm, unlike the other terms which are linear in the expectations. It means that a simple estimate of $\|f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\|^{2}$ using samples $x_{0}^{\theta}$ and $x^{\theta}_{t}$ will be biased. Moreover, to differentiate through the math expectation inside the norm, an explicit dependence of $p_{0}^{\theta}$ on $\theta$ is required, while, in practice, usually only dependence of samples $x_{0}^{\theta}$ on $\theta$ is known.

##### Making loss tractable via linearization.

To resolve this, we use a linearization technique. For a fixed point $x^{\theta}_{t}$ and time $t$, we reformulate the squared norm as a maximization problem. We achieve this by introducing an auxiliary function $\delta:[0,T]\times\mathbb{R}^{D}\to\mathbb{R}^{D}$ and using the identity

$$ $\displaystyle\|f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t})\|^{2}=\max_{\delta_{t}(x_{t}^{\theta})}\left\{-\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t})\rangle\right\}$ $\displaystyle=\max_{\delta_{t}(x_{t}^{\theta})}\mathbb{E}_{x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}\left\{-\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}.$ (11) $$

The reparameterization $\delta=f^{*}-f$ with a fake function $f:[0,T]\times\mathbb{R}^{D}\to\mathbb{R}^{D}$ allows to get:

$$ ( 11 ) $\displaystyle=\!\!\max_{f_{t}(x_{t}^{\theta})}\mathbb{E}_{x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}\!\!\left\{-\|f^{*}_{t}(x^{\theta}_{t})-f_{t}(x^{\theta}_{t})\|^{2}\!\!+2\langle f^{*}_{t}(x^{\theta}_{t})-f_{t}(x^{\theta}_{t}),f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}$ (12) $\displaystyle=\!\!\max_{f_{t}(x_{t}^{\theta})}\mathbb{E}_{x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}\!\bigl\{\underset{=\mathcal{L}_{\text{UM}}(f^{*},p_{0}^{\theta})}{\underbrace{\|f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\|^{2}}}-\underset{=\mathcal{L}_{\text{UM}}(f,p_{0}^{\theta})}{\underbrace{\|f_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\|^{2}}}\bigr\}.$ (13) $$

Since now all expectations are linear and can be estimated, the final step is to compute the expectation over all points $x_{t}^{\theta}$ and times $t$ and minimize it over the generator distribution $p_{\theta}$.

##### Summary.

We build a universal distillation framework as a single min-max optimization ([14](https://arxiv.org/html/2509.22459v1#S3.E14)), implicitly minimizing squared $\ell_{2}$-distance between teacher and student functions. When real and generated probability paths match, these functions match as well, and the distance attains its minimum.

###### Theorem 1 ( Real data generator minimizes UID loss ) .

Let teacher $f^{*}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{*})$ be the minimizer of UM loss (Def. [1](https://arxiv.org/html/2509.22459v1#Thmdefinition1)) on real data $p_{0}^{*}\in\mathcal{P}(\mathbb{R}^{D})$. Then real data generator $G_{\theta^{*}}$, s.t. $p_{0}^{\theta^{*}}=p_{0}^{*}$, is a solution to the min-max optimization of Universal Inverse Distillation (UID) loss $\mathcal{L}_{\text{UID}}(f,p_{0}^{\theta})$ over fake function $f$ and generator distribution $p_{0}^{\theta}$

$$ $\displaystyle\min{{}_{\theta}}\max{{}_{f}}\left\{\mathcal{L}_{\text{UID}}(f,p_{0}^{\theta}):=\mathcal{L}_{\text{UM}}(f^{*},p_{0}^{\theta})-\mathcal{L}_{\text{UM}}(f,p_{0}^{\theta})\right\}.$ (14) $$

###### Lemma 1 ( UID loss minimizes squared $\ell_{2}$ -distance ) .

Maximization of UID loss ([14](https://arxiv.org/html/2509.22459v1#S3.E14)) over fake function $f$ retrieves student function $f^{\theta}:=\operatorname*{arg\,min}{{}_{f}}\mathcal{L}_{\text{UM}}(f,p_{0}^{\theta})$ and represents the squared $\ell_{2}$-distance between it and the teacher $f^{*}$:

$$ $f^{\theta}=\operatorname*{arg\,max}_{f}\mathcal{L}_{\text{UID}}(f,p_{0}^{\theta}),\quad\max_{f}\mathcal{L}_{\text{UID}}(f,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}\|f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t})\|^{2}.$ (15) $$

Note that the distance ([15](https://arxiv.org/html/2509.22459v1#S3.E15)) mostly captures mismatches for the points from generator main domain which do not cover real data, i.e., points $x^{\theta}_{t}$ s.t. $p^{\theta}(x_{t}^{\theta})\gg 0,p^{*}(x_{t}^{\theta})\to 0$. For out-of-domain points $p_{t}^{\theta}(x^{\theta}_{t})\to 0$, the generator cannot receive feedback, because distance ([15](https://arxiv.org/html/2509.22459v1#S3.E15)) for $x^{\theta}_{t}$ also vanishes.

###### Theorem 1 ( Real data generator minimizes UID loss ) .

###### Lemma 1 ( UID loss minimizes squared ℓ 2 \ell_{2} -distance ) .

### 3.2 Relation to prior distillation works

FGM and SiD approaches formulate distillation as a constraint minimization of generator loss subject to the optimal fake model. For generator updates, the explicit UID loss ([12](https://arxiv.org/html/2509.22459v1#S3.E12)) exactly matches SiD loss ([4](https://arxiv.org/html/2509.22459v1#S2.E4)) with $\alpha_{SiD}=0.5$ and FGM loss ([2.3](https://arxiv.org/html/2509.22459v1#S2.Ex12)). For a fake model, it also simply minimizes the UM loss on the generated data. The work (Gushchin et al., 2025) was the first to formulate the distillation of Bridge Matching models in their IBMD framework as a min-max optimization of the single loss ([13](https://arxiv.org/html/2509.22459v1#S3.E13)).

Although previous works derive the same losses, we give a new, simple explanation using a linearization technique. This technique is more powerful and general for handling intractable math expectations than complex proofs for concrete models from FGM, SiD, IBMD. Furthermore, it allows adding real data directly into the distillation loss (see §[3.4](https://arxiv.org/html/2509.22459v1#S3.SS4) and Appendix [A.2](https://arxiv.org/html/2509.22459v1#A1.SS2)) and extending it, e.g., deriving a loss for minimizing the $\ell_{2}$-distance instead of the squared one (Appendix [A.4](https://arxiv.org/html/2509.22459v1#A1.SS4)).

### 3.3 Connection with Inverse Optimization

We derived UID loss ([14](https://arxiv.org/html/2509.22459v1#S3.E14)) by minimizing the squared $\ell_{2}$-distance between teacher and student functions. However, this loss admits another interpretation: its structure is typical for inverse optimization (Chan et al., 2025). In this framework, one considers a parametric family of optimization problems $\min_{f}\mathcal{L}(f,\theta)$ with objective loss $\mathcal{L}(f,\theta)$ depending on argument $f$ and parameters $\theta$. The goal is to find the parameters $\theta^{*}$ that yield a known, desired solution $f^{*}=\operatorname*{arg\,min}_{f}\mathcal{L}(f,\theta^{*})$. One standard way to recover the required parameters is to solve the same min-max problem as ([14](https://arxiv.org/html/2509.22459v1#S3.E14)):

$$ $\min{{}_{\theta}}\max{{}_{f}}\left\{\mathcal{L}(f^{*},\theta)-\mathcal{L}(f,\theta)\right\}\sim\min{{}_{\theta}}\bigl\{\mathcal{L}(f^{*},\theta)-\min{{}_{f}}\{\mathcal{L}(f,\theta)\}\bigr\}.$ (16) $$

The inverse problem ([16](https://arxiv.org/html/2509.22459v1#S3.E16)) always has minimum 0 which is attained when $\theta=\theta^{*}$.

Although the inverse optimization can handle arbitrary losses $\mathcal{L}$, it does not describe the properties of the optimized functions or how to find solutions. In our case, we show that all losses are tractable and minimize the distances between teacher and student functions (Lemmas [1](https://arxiv.org/html/2509.22459v1#Thmlemma1) and [2](https://arxiv.org/html/2509.22459v1#Thmlemma2)). Furthermore, in Appendix [A](https://arxiv.org/html/2509.22459v1#A1), we provide and justify a list of extensions of our framework that cannot be stated as inverse problems. All our proofs are self-contained and do not rely on inverse optimization, which only provides intuition and understanding.

### 3.4 RealUID: natural approach for real data incorporation

Figure: Figure 1: Pipeline of our RealUID distillation framework (§[3](https://arxiv.org/html/2509.22459v1#S3)) with the direct incorporation of real data $p^{*}_{0}$ adjusted by hyperparameters $\alpha,\beta\in(0,1]$. In the figure, it is depicted for Flow Matching models predicting denoised samples. It distills a costly frozen teacher model $f^{*}$ (blue) into a one-step generator $G_{\theta}$ (red) upon min-max optimization of $\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})$ loss over fake model $f$ (green) and generator distribution $p_{0}^{\theta}$ with parameters $\theta$. We use alternating optimization, updating the fake model several times per one generator update for stability. Algorithm’s pseudocode is located in Appendix [B](https://arxiv.org/html/2509.22459v1#A2).
Refer to caption: x1.png

Previous distillation methods add real data during training only via GANs with extra discriminator and adversarial loss. We propose a simpler, more natural way that requires no extra models or losses.

Based on intuition from inverse optimization (§[3.3](https://arxiv.org/html/2509.22459v1#S3.SS3)), we see that the min-max inverse problem ([16](https://arxiv.org/html/2509.22459v1#S3.E16)) is compatible with other losses. This allows us to redesign the UM loss ([2](https://arxiv.org/html/2509.22459v1#S2.E2)) to incorporate real data into it. A key constraint is that the loss must still yield the same teacher upon minimization on the real data. Thus, we derive a novel Unified Matching loss with real data - a weighted sum of two UM-like losses on generated and real data parameterized by $\alpha,\beta\in(0,1]$ which control the weights.

###### Definition 2 .

We define Universal Matching loss with real data on generated data $p^{\theta}_{0}\in\mathcal{P}(\mathbb{R}^{D})$ with $\alpha,\beta\in(0,1]$ (when $\alpha=1$ the real data term becomes $2(1-\beta)\langle f_{t}(x^{*}_{t}),f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\rangle$):

$$ $\displaystyle\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{\theta})$ $\displaystyle=$ $\displaystyle\underset{\text{generated data }p^{\theta}_{0}\text{ term}}{\underbrace{\alpha\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x^{\theta}_{t}\sim p^{\theta}_{t}(\cdot|x^{\theta}_{0})}\left[\|f_{t}(x^{\theta}_{t})-\frac{\beta}{\alpha}f^{\theta}(x^{\theta}_{t}|x^{\theta}_{0})\|^{2}\right]}}$ (17) $\displaystyle+$ $\displaystyle\underset{\text{real data }p^{*}_{0}\text{ term}}{\underbrace{(1-\alpha)\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{0}\sim p^{*}_{0},x_{t}^{*}\sim p_{t}^{*}(\cdot|x^{*}_{0})}\left[\|f_{t}(x^{*}_{t})-\frac{1-\beta}{1-\alpha}f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\|^{2}\right]}}.$ $$

RealUM loss ([17](https://arxiv.org/html/2509.22459v1#S3.E17)) for all $\alpha,\beta$ and UM loss ([2](https://arxiv.org/html/2509.22459v1#S2.E2)) yield the same teacher when input distribution is real data $p_{0}^{*}$, i.e., $\operatorname*{arg\,min}_{f}\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{*})=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{*})=f^{*}$.
Hence, the min-max inverse scheme ([16](https://arxiv.org/html/2509.22459v1#S3.E16)) with RealUM loss and the old teacher $f^{*}$ will still have a real data generator as a solution:

$$ $\min{{}_{\theta}}\{\underset{\geq 0}{\underbrace{\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f^{*},p_{0}^{\theta})-\min{{}_{f}}\{\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{\theta})\}}}\}=\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f^{*},p_{0}^{*})-\underset{=\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f^{*},p_{0}^{*})}{\underbrace{\min{{}_{f}}\{\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{*})\}}}=0.$ $$

But now distillation loss will incorporate real data through the real data terms of $\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{\theta})$.

###### Theorem 2 ( Real data generator minimizes RealUID loss ) .

Let teacher $f^{*}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{*})$ be the minimizer of UM loss on real data $p_{0}^{*}$. Then real data generator $G_{\theta^{*}}$, s.t. $p_{0}^{\theta^{*}}=p^{*}_{0}$, is a solution to the min-max optimization of Universal Inverse Distillation loss with real data (RealUID) $\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})$ over fake function $f$ and generator distribution $p_{0}^{\theta}$:

$$ $\displaystyle\min{{}_{\theta}}\max{{}_{f}}\left\{\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta}):=\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f^{*},p_{0}^{\theta})-\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{\theta})\right\}.$ (18) $$

We provide analysis of RealUID in Appendix [A.1](https://arxiv.org/html/2509.22459v1#A1.SS1), below we highlight the most important findings.

###### Definition 2 .

###### Theorem 2 ( Real data generator minimizes RealUID loss ) .

##### Role and choice of coefficients α , β \alpha,\beta .

The RealUID framework uses real data samples only to minimize RealUM loss for the fake model. As shown in Lemma [2](https://arxiv.org/html/2509.22459v1#Thmlemma2), RealUID also implicitly minimizes the rescaled distance between the teacher and generator functions. But now with the help of real data, this distance captures mismatches for both incorrectly generated points from the generator’s main domain and the real data points, which the generator fails to cover. Thus, unlike data-free UID loss (Lemma [1](https://arxiv.org/html/2509.22459v1#Thmlemma1)), RealUID loss provides the generator with feedback also on the real data domain it needs to cover (see Appendix [A.1.2](https://arxiv.org/html/2509.22459v1#A1.SS1.SSS2) for details).

The coefficients $\alpha$ and $\beta$ in RealUID loss control the balance between the generator’s and the real data domains. A zero difference ($\alpha=\beta$) means that only the generator’s domain is considered. Even when $\alpha=\beta<1$ and real data is formally added, it has no, or negative, effect on the generator. Meanwhile, a non-zero difference considers both domains: increasing $\alpha$ over $\beta$ ($\alpha>\beta$) puts less weight on the generator’s errors, while otherwise ($\beta<\alpha$) the weight grows. The optimal choice is a slightly different $\alpha\neq\beta$ close to $1.$ Excessively low $\alpha$ and $\beta$ diminish the effect of the generated data, leading to vanishing gradients. The same issue occurs with $\alpha\gg\beta$, while $\beta\gg\alpha$ eliminates the effect of the real data. Plus, configurations $\beta<\alpha=1$ are unstable due to out-of-domain samples.

##### Comparison with GAN-based methods.

Unlike SiD and FGM with GANs, we do not use extra adversarial losses and discriminator to incorporate real data. We only modify UM loss, preserving its core structure and fake model architecture. While general adversarial loss is unrelated to the main distillation loss and has uninterpretable scaling hyperparameters, our RealUID loss and weighting coefficients $\alpha,\beta\in(0,1]$ come naturally from the data-free UID loss. The original UID loss ([14](https://arxiv.org/html/2509.22459v1#S3.E14)), equivalent to SiD ([4](https://arxiv.org/html/2509.22459v1#S2.E4)) with $\alpha_{\text{SiD}}=0.5$ and FGM ([2.3](https://arxiv.org/html/2509.22459v1#S2.Ex12)), is obtained when $\alpha=\beta=1$.

Extension for Bridge Matching and Stochastic Interpolants framework. In Appendix [C](https://arxiv.org/html/2509.22459v1#A3), we demonstrate that our framework can be easily extended to other matching models by parametrizing the generated data coupling $\pi^{\theta}(x_{0},x_{T})$ instead of the data distribution $p^{\theta}_{0}$.

## 4 Experiments

Table: Table 1: Ablation studies on $(\alpha,\beta)$ coefficients on the left table and adversarial weighting parameters $(\lambda_{\text{adv}}^{G_{\theta}},\lambda_{\text{adv}}^{D})$ on the right table for CIFAR-10 in both unconditional and conditional settings. The baseline RealUID ($\alpha=1.0,\beta=1.0$) does not use real data. Configurations that outperform the baseline are highlighted. All values report FID $\downarrow$, where lower is better. The best configuration in each case is bolded.

All implementations were developed in PyTorch, and the code will be made publicly available.

This section provides an ablation study and evaluation of our RealUID, assessing both its performance and computational efficiency. We begin in (§[4.1](https://arxiv.org/html/2509.22459v1#S4.SS1)) by detailing the experimental setup. In (§[4.2](https://arxiv.org/html/2509.22459v1#S4.SS2)), we show that our incorporation of real data via coefficients $\alpha,\beta$ improves performance, speeds up convergence, and enables effective fine-tuning. In (§[4.3](https://arxiv.org/html/2509.22459v1#S4.SS3)), we assess the benchmark performance and computational demands of RealUID relative to SOTA methods. Additional experimental details and results are provided in Appendix [D](https://arxiv.org/html/2509.22459v1#A4) and Appendix [E](https://arxiv.org/html/2509.22459v1#A5), respectively.

### 4.1 Experimental Setup

##### Dataset and Evaluation Protocol.

Due to computational resource constraints, all experiments were conducted exclusively on the CIFAR-10 dataset with $32\times 32$ resolution (Krizhevsky et al., 2009), considering both conditional and unconditional settings. In line with prior works (Karras et al., 2019; 2022), we report test FID scores (Heusel et al., 2017), computed using 50k generated samples.

##### Implementation Details.

In contrast to prior studies (Zhou et al., 2024b; a; Huang et al., 2024), which employ the computationally demanding EDM architecture (Karras et al., 2022), our work adopts a more lightweight alternative (Tong et al., 2023) due to resource constraints (see (§[4.3](https://arxiv.org/html/2509.22459v1#S4.SS3)) for efficiency analysis). We also trained our own flow-matching model, denoted by $f^{*}$, which served as the teacher. Further implementation details are provided in Appendix [D](https://arxiv.org/html/2509.22459v1#A4).

### 4.2 Benchmarking Methods under a Unified Experimental Configuration

Table: Table 2: This table presents the results of our ablation study on the RealUID framework, evaluated using the FID metric under both unconditional and conditional generation setups. The Teacher Flow model with 100 NFE is reported as a reference. The performance of the baseline RealUID ($\alpha=1.0,\beta=1.0$) without real-data incorporation is indicated in italic. For emphasis, we underline the two counterparts that incorporate real data: the GAN-based and our RealUID methods. The best-performing configurations, obtained via an additional fine-tuning stage with adjusted $(\alpha_{\text{FT}},\beta_{\text{FT}})$, are highlighted in bold. Qualitative results are presented in § [E.2](https://arxiv.org/html/2509.22459v1#A5.SS2).

Figure: Figure 2: Evolution of FID during CIFAR-10 distillation for (i) the baseline RealUID ($\alpha=1.0$, $\beta=1.0$), (ii) the best-performing RealUID configurations, and (iii) subsequent fine-tuning, evaluated in both unconditional and conditional settings. The performances of Teacher Flow and UID+GAN are indicated by horizontal reference lines in their respective colors. Methods that incorporate real data—best-performing RealUID and UID+GAN—are highlighted in green to facilitate comparison.
Refer to caption: x2.png

We evaluate RealUID under a unified experimental protocol (fixed architecture and implementation). We begin by (i) conducting an ablation over $\alpha,\beta$ to assess the influence of real-data incorporation. We then (ii) compare RealUID to a GAN-based alternative, showing that RealUID achieves comparable or superior accuracy. Furthermore, (iii) we analyze convergence, indicating that RealUID variants with real data train substantially faster than baselines without real-data. Finally, (iv) we explore a fine-tuning stage initialized from strong RealUID checkpoints, showing further performance gains.

##### Ablation study of coefficients α , β \alpha,\beta .

The search for optimal $\alpha$ and $\beta$ parameters was restricted to values near $1$, specifically $\alpha,\beta\in[0.9,1.0]$ with increments of $0.02$ to cover the full grid. Setting these parameters too low prevents the student from accurately capturing the true generator gradient, which in turn leads the generator to produce noisy samples. The results are reported in Table [1](https://arxiv.org/html/2509.22459v1#S4.T1). As a baseline, we highlight the model without data incorporation our RealUID ($\alpha=1.0,\beta=1.0$). As shown in the table, using real data with $\alpha=\beta<1.0$ or with substantially different $\alpha$ and $\beta$ consistently degraded performance. In contrast, parameter settings close to the diagonal (with the exception of $\alpha=1.0,\beta=0.98$) produced improved results, with the best performance achieved by our RealUID ($\alpha=0.94,\beta=0.96$) for the unconditional case and our RealUID ($\alpha=0.98,\beta=0.96$) for the conditional case. Note that the practical results for various $\alpha,\beta$ match the theoretical description from (§[3.4](https://arxiv.org/html/2509.22459v1#S3.SS4).)

##### Comparison with GAN-based method.

We integrated the GAN-based approach proposed by Zhou et al. (2024a) into our experimental framework as an alternative method for incorporating real data, enabling a direct comparison with our RealUID formulation. Specifically, we combined the GAN loss with the baseline RealUID ($\alpha=1.0,\beta=1.0$). As shown in Table [1](https://arxiv.org/html/2509.22459v1#S4.T1), the best-performing configurations are achieved with GAN losses ($\lambda_{\text{adv}}^{G_{\theta}}=0.3$, $\lambda_{\text{adv}}^{D}=1$). While this setup performs comparably to RealUID ($\alpha=0.94,\beta=0.96$) in the unconditional setting, it remains clearly inferior to RealUID ($\alpha=0.98,\beta=0.96$) in the conditional case.

##### Convergence Speed.

Our RealUID ($\alpha,\beta$), which outperform the baseline RealUID ($\alpha=1.0,\beta=1.0$) and are highlighted in Table [1](https://arxiv.org/html/2509.22459v1#S4.T1), demonstrate rapid convergence. For clarity, we present qualitative comparisons of the best-performing configurations against their respective baselines in Figure [2](https://arxiv.org/html/2509.22459v1#S4.F2). As shown in figure, the best RealUID configurations reach the saturated performance level of the baseline after $\sim$100k iterations, whereas the baseline requires $\sim$300k iterations to achieve comparable metrics. These results demonstrate that incorporating real data substantially accelerates convergence.

##### Fine-tuning stage.

We observe that the RealUID framework offers substantial flexibility for fine-tuning. In this procedure, the generator $G_{\theta}$ is initialized from the best-performing RealUID checkpoint obtained during training from scratch, while the fake model $f$ is initialized from the teacher model $f^{*}$. Fine-tuning then proceeds with new hyperparameter values $\alpha_{\text{FT}}$ and $\beta_{\text{FT}}$, allowing for refined control over the degree of real-data incorporation during this stage. We find that the configurations RealUID ($\alpha=0.94,\beta=0.96\mid\alpha_{\text{FT}}=0.94,\beta_{\text{FT}}=1.0$) and RealUID ($\alpha=0.98,\beta=0.96\mid\alpha_{\text{FT}}=0.94,\beta_{\text{FT}}=1.0$) produced the best results in the unconditional and conditional cases, respectively, as shown in Tables [2](https://arxiv.org/html/2509.22459v1#S4.T2). Ablation studies analyzing the effect of $\alpha_{\text{FT}}$ and $\beta_{\text{FT}}$ are provided in Appendix [E.1](https://arxiv.org/html/2509.22459v1#A5.SS1).

### 4.3 Benchmark performance and Computational comparisons

Table: Table 3: Comparison of *unconditional* generation on CIFAR-10. The best method under the FID metric in each section is highlighted with bold.

Table: Table 5: Inference complexity on an Ascend 910B3 (65 GB) NPU. For each method, we report (i) the mean inference time per image (bs=1, fp32), averaged over 10,000 iterations; (ii) the total number of parameters (Millions); and (iii) peak NPU memory usage (maximum allocated and reserved, in MB). Best values are bolded.

As shown in Tables [4](https://arxiv.org/html/2509.22459v1#S4.T4) and [4](https://arxiv.org/html/2509.22459v1#S4.T4), RealUID consistently outperforms all prior flow-based models on CIFAR-10, significantly surpassing the strongest flow distillation baseline, FGM. Despite its compact architecture (§[4.1](https://arxiv.org/html/2509.22459v1#S4.SS1)), it achieves performance comparable to leading diffusion distillation methods-matching SiD ($\alpha{=}1.0$) and closely approaching SiD ($\alpha{=}1.2$), while falling short of adversarially enhanced models such as $\text{SiD}^{2}\text{A}$. Based on ablation studies and comparisons with GANs (§[4.2](https://arxiv.org/html/2509.22459v1#S4.SS2)), we hypothesis that this performance gap is attributed to architectural and teacher capacity differences rather than the lack of adversarial loss. In terms of efficiency, RealUID leverages a lightweight architecture based on Tong et al. (2023). Therefore, as summarized in Table [5](https://arxiv.org/html/2509.22459v1#S4.T5), it achieves nearly $2\times$ faster inference, lower memory usage, and reduced model size compared to recent distillation approaches (Zhou et al., 2024b; a; Huang et al., 2024). The results indicate that our approach achieves competitive performance while maintaining a lower computational footprint.

## 5 Discussion, extension, future works

##### Extensions.

Our RealUID (§[3.4](https://arxiv.org/html/2509.22459v1#S3.SS4)) framework can distill Flow/Bridge Matching, Diffusion models, and Stochastic Interpolants enhanced by a novel natural way to incorporate real data.
In Appendix [A](https://arxiv.org/html/2509.22459v1#A1), we provide three extensions of our RealUID beyond the inverse scheme: General RealUID with 3 coefficients (Appendix [A.2](https://arxiv.org/html/2509.22459v1#A1.SS2)), SiD framework with real data for $\alpha_{\text{SiD}}\neq\frac{1}{2}$ (Appendix [A.3](https://arxiv.org/html/2509.22459v1#A1.SS3)) and Normalized RealUID for minimizing non-squared $\ell_{2}$-distance between teacher and student (Appendix [A.4](https://arxiv.org/html/2509.22459v1#A1.SS4)).

##### Relation to DMD.

Instead of minimizing the squared $\ell_{2}$-distance between the score functions, Distribution Matching Distillation (Luo et al., 2023; Wang et al., 2023; Yin et al., 2024b; a) (DMD) approach minimizes the KL divergence between the real and generated data. Its gradients are computed using the generator and teacher score functions, leading to the similar alternating updates. We would like to highlight that DMD does not fit UID framework.
Nevertheless, we investigated an opportunity to incorporate real data into DMD without GANs in Appendix [A.5](https://arxiv.org/html/2509.22459v1#A1.SS5).

## References

- Albergo et al. (2023)
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden.
Stochastic interpolants: A unifying framework for flows and diffusions.
*arXiv preprint arXiv:2303.08797*, 2023.
- Brock et al. (2018)
Andrew Brock, Jeff Donahue, and Karen Simonyan.
Large scale gan training for high fidelity natural image synthesis.
*arXiv preprint arXiv:1809.11096*, 2018.
- Chan et al. (2025)
Timothy CY Chan, Rafid Mahmood, and Ian Yihang Zhu.
Inverse optimization: Theory and applications.
*Operations Research*, 73(2):1046–1074, 2025.
- Gao et al. (2025)
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin Patrick Murphy, and Tim Salimans.
Diffusion models and gaussian flow matching: Two sides of the same coin.
In *The Fourth Blogpost Track at ICLR 2025*, 2025.
- Geng et al. (2023)
Zhengyang Geng, Ashwini Pokle, and J Zico Kolter.
One-step diffusion distillation via deep equilibrium models.
*Advances in Neural Information Processing Systems*, 36:41914–41931, 2023.
- Goodfellow et al. (2014)
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
*Advances in neural information processing systems*, 27, 2014.
- Gushchin et al. (2024)
Nikita Gushchin, Alexander Kolesov, Alexander Korotin, Dmitry P Vetrov, and Evgeny Burnaev.
Entropic neural optimal transport via diffusion processes.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Gushchin et al. (2025)
Nikita Gushchin, David Li, Daniil Selikhanovych, Evgeny Burnaev, Dmitry Baranchuk, and Alexander Korotin.
Inverse bridge matching distillation.
2025.
- Heusel et al. (2017)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
*Advances in neural information processing systems*, 30, 2017.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*, 33:6840–6851, 2020.
- Holderrieth et al. (2024)
Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky TQ Chen, and Yaron Lipman.
Generator matching: Generative modeling with arbitrary markov processes.
*arXiv preprint arXiv:2410.20587*, 2024.
- Huang et al. (2024)
Zemin Huang, Zhengyang Geng, Weijian Luo, and Guo-jun Qi.
Flow generator matching.
*arXiv preprint arXiv:2410.19310*, 2024.
- Karras et al. (2019)
Tero Karras, Samuli Laine, and Timo Aila.
A style-based generator architecture for generative adversarial networks.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 4401–4410, 2019.
- Karras et al. (2020)
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.
Analyzing and improving the image quality of stylegan.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 8110–8119, 2020.
- Karras et al. (2022)
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.
Elucidating the design space of diffusion-based generative models.
*Advances in neural information processing systems*, 35:26565–26577, 2022.
- Kim et al. (2023)
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon.
Consistency trajectory models: Learning probability flow ode trajectory of diffusion.
*arXiv preprint arXiv:2310.02279*, 2023.
- Krizhevsky et al. (2009)
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.
- Lipman et al. (2023)
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le.
Flow matching for generative modeling.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=PqvMRDCJT9t](https://openreview.net/forum?id=PqvMRDCJT9t).
- Liu (2022)
Qiang Liu.
Rectified flow: A marginal preserving approach to optimal transport.
*arXiv preprint arXiv:2209.14577*, 2022.
- Liu et al. (2022a)
Xingchao Liu, Chengyue Gong, and Qiang Liu.
Flow straight and fast: Learning to generate and transfer data with rectified flow.
*arXiv preprint arXiv:2209.03003*, 2022a.
- Liu et al. (2022b)
Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu.
Let us build bridges: Understanding and extending diffusion generative models.
*arXiv preprint arXiv:2208.14699*, 2022b.
- Liu et al. (2023)
Xingchao Liu, Chengyue Gong, and qiang liu.
Flow straight and fast: Learning to generate and transfer data with rectified flow.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=XVjTT1nw5z](https://openreview.net/forum?id=XVjTT1nw5z).
- Loshchilov & Hutter (2017)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
*arXiv preprint arXiv:1711.05101*, 2017.
- Luo et al. (2023)
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang.
Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models.
*Advances in Neural Information Processing Systems*, 36:76525–76546, 2023.
- Peluchetti (2023)
Stefano Peluchetti.
Non-denoising forward-time diffusions.
*arXiv preprint arXiv:2312.14589*, 2023.
- Sauer et al. (2022)
A Sauer, K Schwarz, and A StyleGAN-XL Geiger.
scaling stylegan to large diverse datasets.
In *Proceedings of the SIGGRAPH Conference. ACM*, pp. 1–10, 2022.
- Sohl-Dickstein et al. (2015)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In *International conference on machine learning*, pp. 2256–2265. pmlr, 2015.
- Song & Dhariwal (2023)
Yang Song and Prafulla Dhariwal.
Improved techniques for training consistency models.
*arXiv preprint arXiv:2310.14189*, 2023.
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=PxTIG12RRHS](https://openreview.net/forum?id=PxTIG12RRHS).
- Takida et al. (2023)
Yuhta Takida, Masaaki Imaizumi, Takashi Shibuya, Chieh-Hsin Lai, Toshimitsu Uesaka, Naoki Murata, and Yuki Mitsufuji.
San: Inducing metrizability of gan with discriminative normalized linear layer.
*arXiv preprint arXiv:2301.12811*, 2023.
- Tong et al. (2023)
Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio.
Improving and generalizing flow-based generative models with minibatch optimal transport.
*arXiv preprint arXiv:2302.00482*, 2023.
- Wang et al. (2022)
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou.
Diffusion-gan: Training gans with diffusion.
*arXiv preprint arXiv:2206.02262*, 2022.
- Wang et al. (2023)
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.
Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.
*Advances in neural information processing systems*, 36:8406–8441, 2023.
- Yang et al. (2024)
Ling Yang, Zixiang Zhang, Zhilong Zhang, Xingchao Liu, Minkai Xu, Wentao Zhang, Chenlin Meng, Stefano Ermon, and Bin Cui.
Consistency flow matching: Defining straight flows with velocity consistency.
*arXiv preprint arXiv:2407.02398*, 2024.
- Yin et al. (2024a)
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman.
Improved distribution matching distillation for fast image synthesis.
*Advances in neural information processing systems*, 37:47455–47487, 2024a.
- Yin et al. (2024b)
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park.
One-step diffusion with distribution matching distillation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 6613–6623, 2024b.
- Zheng et al. (2024)
Kaiwen Zheng, Guande He, Jianfei Chen, Fan Bao, and Jun Zhu.
Diffusion bridge implicit models.
*arXiv preprint arXiv:2405.15885*, 2024.
- Zhou et al. (2024a)
Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang.
Adversarial score identity distillation: Rapidly surpassing the teacher in one step.
*arXiv preprint arXiv:2410.14919*, 2024a.
- Zhou et al. (2024b)
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang.
Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation.
In *Forty-first International Conference on Machine Learning*, 2024b.

## Appendix A Theoretical proofs and extensions

In this appendix, we discuss our RealUID framework (Appendix [A.1](https://arxiv.org/html/2509.22459v1#A1.SS1)) in theoretical details and provide three extensions of it: General RealUID framework with 3 degrees of freedom (Appendix [A.2](https://arxiv.org/html/2509.22459v1#A1.SS2)), SiD framework with real data (Appendix [A.3](https://arxiv.org/html/2509.22459v1#A1.SS3)) and Normalized RealUID framework for minimizing $\ell_{2}$-distance between teacher and student functions instead of the squared one (Appendix [A.4](https://arxiv.org/html/2509.22459v1#A1.SS4)). All proofs are based on the linearization technique and splitting terms in linearized decomposition between real and generated data.

We also propose an approach to incorporate real data into DMD framework, which is unsuitable for our RealUID Appendix [A.5](https://arxiv.org/html/2509.22459v1#A1.SS5).

### A.1 RealUID theoretical properties

In this section, we discuss our RealUID loss in detail. We begin by presenting its explicit form and how it connects linearization technique and real data incorporation. We then demonstrate that the loss minimizes a squared $\ell_{2}$-distance between the rescaled teacher and student functions (Appendix [A.1.1](https://arxiv.org/html/2509.22459v1#A1.SS1.SSS1)). Finally, we provide guidance on selecting the coefficients $\alpha$ and $\beta$ to optimize the impact from the real data (Appendix [A.1.2](https://arxiv.org/html/2509.22459v1#A1.SS1.SSS2)).

#### A.1.1 RealUID Distance Lemma 2

Putting explicit values for RealUM loss ([17](https://arxiv.org/html/2509.22459v1#S3.E17)) in RealUID loss ([18](https://arxiv.org/html/2509.22459v1#S3.E18)) and denoting $\delta_{t}=f^{*}_{t}-f_{t}$, we get:

$$ $\displaystyle\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}[-\alpha\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\alpha\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\beta\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle]$ $\displaystyle+\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{0}\sim p^{*}_{0},x_{t}^{*}\sim p_{t}^{*}(\cdot|x^{*}_{t})}[-(1-\alpha)\|\delta_{t}(x_{t}^{*})\|^{2}+2(1-\alpha)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2(1-\beta)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\rangle].$ $$

This form provides an alternative definition of coefficients $\alpha$ and $\beta$: they define the proportion in which each summand in the data-free linearized representation ([11](https://arxiv.org/html/2509.22459v1#S3.E11)) of the squared $\ell_{2}$-distance is split between the real and generated data. The idea of splitting coefficients between two data types helps extend RealUID to extra coefficients (Appendix [A.2](https://arxiv.org/html/2509.22459v1#A1.SS2)), new distances (Appendix [A.4](https://arxiv.org/html/2509.22459v1#A1.SS4)) and SiD framework with $\alpha_{\text{SiD}}\neq\frac{1}{2}$ (Appendix [A.3](https://arxiv.org/html/2509.22459v1#A1.SS3)).

While UID loss implicitly represents a squared $\ell_{2}$-distance between the teacher and student functions (Lemma [1](https://arxiv.org/html/2509.22459v1#Thmlemma1)), the RealUID loss minimizes a squared $\ell_{2}$-distance between the rescaled functions. The distance is still minimal when $p_{0}^{\theta}=p_{0}^{*}$, alternatively proving Theorem [2](https://arxiv.org/html/2509.22459v1#Thmtheorem2).

###### Lemma 2 ( Distance minimized by RealUID loss ) .

Maximization of RealUID loss $\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}$ ([17](https://arxiv.org/html/2509.22459v1#S3.E17)) over fake function $f$ returns the weighted sum between the teacher $f^{*}$ and student function $f^{\theta}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{\theta})$ and represents the weighted squared $\ell_{2}$-distance between them:

$$ $\left[\operatorname*{arg\,max}_{f}\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})\right](t,x_{t})=\frac{(1-\beta)p_{t}^{*}(x_{t})\cdot f^{*}_{t}(x_{t})+\beta p_{t}^{\theta}(x_{t})\cdot f^{\theta}_{t}(x_{t})}{(1-\alpha)p_{t}^{*}(x_{t})+\alpha p_{t}^{\theta}(x_{t})},$ (19) $$

$$ $\max_{f}\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})\!=\!\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p_{t}^{*}}\!\!\left[\!\!\frac{\|(p_{t}^{*}(x_{t}^{*})(\beta-\alpha)+\alpha p_{t}^{\theta}(x_{t}^{*}))\cdot f^{*}_{t}(x_{t}^{*})-\beta p_{t}^{\theta}(x_{t}^{*})\cdot f^{\theta}_{t}(x_{t}^{*})\|^{2}}{p_{t}^{*}(x_{t}^{*})((1-\alpha)p_{t}^{*}(x_{t}^{*})+\alpha p_{t}^{\theta}(x_{t}^{*}))}\!\!\right].$ $$

###### Proof.

First, we write down RealUID loss in an explicit form with $\delta_{t}=f^{*}_{t}-f_{t}$

$$ $\displaystyle\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p^{\theta}_{t}}[-\alpha\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\alpha\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\beta\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t})\rangle]$ $\displaystyle+\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t}}[-(1-\alpha)\|\delta_{t}(x_{t}^{*})\|^{2}+2(1-\alpha)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2(1-\beta)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle].$ (20) $$

Then, we rescale the generated data terms in RealUID loss ([20](https://arxiv.org/html/2509.22459v1#A1.E20)) using the equality $p^{\theta}_{t}(x_{t})=\frac{p^{\theta}_{t}(x_{t})}{p^{*}_{t}(x_{t})}p^{*}_{t}(x_{t})$ for $x_{t}\in\mathbb{R}^{D}$ (we assume $p_{t}^{*}(x_{t})>0,\forall x_{t},t$) leaving only math expectation w.r.t. the real data, i.e, $\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(\delta,p_{0}^{\theta})=$

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t}}\!\!\left[-[(1-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]\|\delta_{t}(x_{t}^{*})\|^{2}+2[(\beta-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2\beta\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}\langle\delta_{t}(x_{t}^{*}),f^{\theta}_{t}(x^{*}_{t})\rangle\right].$ $$

Finally, we maximize the loss w.r.t. $\delta_{t}(x_{t}^{*})$ for each $x_{t}^{*}$ and $t$ as a quadratic function. The maximum is achieved when

$$ $\delta_{t}(x_{t}^{*})=\frac{[(\beta-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]f^{*}_{t}(x^{*}_{t})-\beta\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}f^{\theta}_{t}(x^{*}_{t})}{[(1-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]}$ $$

or in terms of the fake model $f=f^{*}-\delta$

$$ $\left(\operatorname*{arg\,max}_{f}\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})\right)(t,x_{t})=\frac{f^{*}_{t}(x_{t})\cdot(1-\beta)+f^{\theta}_{t}(x_{t})\cdot\beta\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}}{(1-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}}.$ $$

The maximum itself equals to

$$ $\max_{f}\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p_{t}^{*}}\left[\frac{\|f^{*}_{t}(x_{t}^{*})\cdot((\beta-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})})-f^{\theta}_{t}(x_{t}^{*})\cdot\beta\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}\|^{2}}{(1-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}}\right].$ $$

It is easy to see that when $p_{0}^{\theta}=p^{*}_{0}$ and $f^{\theta}=f^{*}$ this distance achieves its minimal value 0. Moreover, optimal fake model in this case matches the teacher $f^{*}$, i.e.,

$$ $\left(\operatorname*{arg\,max}_{f}\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{*})\right)(t,x_{t})=\frac{f^{*}_{t}(x_{t})\cdot(1-\beta)+f^{*}_{t}(x_{t})\cdot\beta\frac{p_{t}^{*}(x_{t})}{p_{t}^{*}(x_{t})}}{(1-\alpha)+\alpha\frac{p_{t}^{*}(x_{t})}{p_{t}^{*}(x_{t})}}=f^{*}_{t}(x_{t}).$ $$

∎

###### Lemma 2 ( Distance minimized by RealUID loss ) .

###### Proof.

#### A.1.2 Explanation of the choice of coefficients α \alpha and β \beta

Following Lemma [2](https://arxiv.org/html/2509.22459v1#Thmlemma2), we know exactly what distance our RealUID loss implicitly minimizes and can examine it for various $\alpha,\beta\in(0,1]$:

$$ $\displaystyle\max_{f}\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})$ $\displaystyle=$ $\displaystyle\int_{x_{t}}l_{t}(x_{t},\beta,\alpha)dx_{t},$ $\displaystyle l_{t}(x_{t},\beta,\alpha)$ $\displaystyle:=$ $\displaystyle\frac{\|(p_{t}^{*}(x_{t})(\beta-\alpha)+\alpha p_{t}^{\theta}(x_{t}))\cdot f^{*}_{t}(x_{t})-\beta p_{t}^{\theta}(x_{t})\cdot f^{\theta}_{t}(x_{t})\|^{2}}{(1-\alpha)p_{t}^{*}(x_{t})+\alpha p_{t}^{\theta}(x_{t})},$ $$

where $l_{t}(x_{t},\beta,\alpha)$ denotes the distance for the particular point $x_{t}$.

The total distance mostly sums up from the two groups of points: incorrectly generated points from the generator’s main domain, i.e., $p_{t}^{\theta}(x_{t})\gg 0,p^{*}(x_{t})\to 0,$ and real data points which are not covered by the generator, i.e., $p_{t}^{\theta}(x_{t})\to 0,p^{*}(x_{t})\gg 0.$ For the points out of both domains $p_{t}^{\theta}(x_{t})\to 0,p_{t}^{*}(x_{t})\to 0$, the distance tends to 0, as well as for matching points $p_{t}^{\theta}(x_{t})\approx p_{t}^{*}(x_{t}).$

Next, we consider various coefficients $\alpha,\beta\in(0,1]$ and how they effect two main groups of points.

- •
All configurations affect the incorrectly generated points $x_{t}:p_{t}^{*}(x_{t})\to 0,p^{\theta}(x_{t})\gg 0$:
$\displaystyle l_{t}(x_{t},\beta,\alpha)\approx\frac{\|\alpha p_{t}^{\theta}(x_{t})\cdot f^{*}_{t}(x_{t})-\beta p_{t}^{\theta}(x_{t})\cdot f^{\theta}_{t}(x_{t})\|^{2}}{\alpha p_{t}^{\theta}(x_{t})}\approx\frac{\beta^{2}\|f^{\theta}_{t}(x_{t})\|^{2}}{\alpha}p_{t}^{\theta}(x_{t})\gg 0.$
(21)
Note that increasing $\beta$ over $\alpha$ ($\beta>\alpha$) will diminish the weight of the distance in comparison with $\alpha=\beta=1$, while decreasing otherwise will lift the weight up.
- •
Configuration $\beta<\alpha=1$ is unstable for uncovered real data points $x_{t}:p_{t}^{\theta}(x_{t})\to 0,p^{*}(x_{t})\gg 0$:
$\displaystyle l_{t}(x_{t},\beta,\alpha)\approx\frac{\|p_{t}^{*}(x_{t})(\beta-1)\cdot f^{*}_{t}(x_{t})-\beta p_{t}^{\theta}(x_{t})\cdot f^{\theta}_{t}(x_{t})\|^{2}}{p_{t}^{\theta}(x_{t})}\to\infty.$
- •
Configuration $\beta=\alpha=1$ (UID loss) does not affect uncovered real data points $x_{t}:p_{t}^{\theta}(x_{t})\to 0,p^{*}(x_{t})\gg 0$:
$\displaystyle l_{t}(x_{t},\beta,\alpha)\approx\frac{\|p_{t}^{\theta}(x_{t})\cdot f^{*}_{t}(x_{t})-p_{t}^{\theta}(x_{t})\cdot f^{\theta}_{t}(x_{t})\|^{2}}{p_{t}^{\theta}(x_{t})}=\|f^{*}_{t}(x_{t})-f^{\theta}_{t}(x_{t})\|^{2}p_{t}^{\theta}(x_{t})\to 0.$
- •
Configuration $\beta=\alpha<1$ does not affect uncovered real data points $x_{t}:p_{t}^{\theta}(x_{t})\to 0,p^{*}(x_{t})\gg 0$:
$\displaystyle l_{t}(x_{t},\beta,\alpha)\approx\frac{\|\alpha p_{t}^{\theta}(x_{t})\cdot f^{*}_{t}(x_{t})-\beta p_{t}^{\theta}(x_{t})\cdot f^{\theta}_{t}(x_{t})\|^{2}}{(1-\alpha)p_{t}^{*}(x_{t})}=\frac{\|\alpha\cdot f^{*}_{t}(x_{t})-\beta f^{\theta}_{t}(x_{t})\|^{2}}{(1-\alpha)}\frac{(p_{t}^{\theta}(x_{t}))^{2}}{p_{t}^{*}(x_{t})}\to 0.$
Notably, in this configuration, the distance drops even faster than when $\alpha=\beta=1$, what makes it even less preferable.
- •
Configuration $\beta\neq\alpha$ affects the uncovered real data points $x_{t}:p_{t}^{\theta}(x_{t})\to 0,p^{*}(x_{t})\gg 0$:
$\displaystyle l_{t}(x_{t},\beta,\alpha)\approx\frac{\|p_{t}^{*}(x_{t})(\beta-\alpha)\cdot f^{*}_{t}(x_{t})-\beta p_{t}^{\theta}(x_{t})\cdot f^{\theta}_{t}(x_{t})\|^{2}}{(1-\alpha)p_{t}^{*}(x_{t})}\gg 0.$

For visual illustration, we analytically calculated the loss surface $l_{t}(x_{t},\alpha,\beta)$ between the FM models transforming one-dimensional real data Gaussian $\mathcal{N}(\mu^{*},1)$ and generated Gaussian $\mathcal{N}(\mu^{\theta},1)$ to noise $\mathcal{N}(0,1)$ on the time interval $[0,1]$. In this case, the generated and real data interpolations are $p_{t}^{\theta}(x_{t})=\mathcal{N}(x_{t}|\mu^{\theta}(1-t),t^{2}+(1-t)^{2})$ and $p_{t}^{*}(x_{t})=\mathcal{N}(x_{t}|\mu^{*}(1-t),t^{2}+(1-t)^{2}).$ The unconditional vector field $u$ between $\mathcal{N}(0,1)$ and $\mathcal{N}(\mu,1)$ can be calculated as

$$ $\displaystyle u_{t}(x_{t})$ $\displaystyle=$ $\displaystyle\mathbb{E}_{x_{0}\sim p_{0}(\cdot|x_{t})}\left[\frac{x_{t}-x_{0}}{t}\right]=\int_{x_{0}}\left(\frac{x_{t}-x_{0}}{t}\right)\cdot\mathcal{N}\left(\frac{x_{t}-x_{0}(1-t)}{t}|0,1\right)\cdot\mathcal{N}(x_{0}|\mu,1)dx_{0}$ (22) $\displaystyle=$ $\displaystyle\frac{a(2t^{2}-2t)-bt^{2}}{\sqrt{2\pi}(1-2t+2t^{2})^{\frac{3}{2}}}\exp\left(-\frac{(x_{t}-\mu(1-t))^{2}}{2(1-2t+2t^{2})^{2}}\right).$ $$

In Figure [3](https://arxiv.org/html/2509.22459v1#A1.F3), we can see that configurations $\alpha=\beta$ do not notice the real data sample, while $\alpha\neq\beta$ actually spots both domains, increasing the weight of generator domain when $\beta>\alpha$ and decreasing it otherwise.

Figure: Figure 3: RealUID loss for $1D$-Gaussians under various coefficients $(\alpha,\beta)$.
Refer to caption: x4.png

### A.2 General RealUID loss

We recall that UID loss (Theorem [1](https://arxiv.org/html/2509.22459v1#Thmtheorem1)) can be restated via linearization technique with $\delta=f^{*}-f$ as:

$$ $\mathcal{L}_{\text{UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}\left\{-\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}.$ $$

In turn, after real data incorporation, we obtain our RealUID loss (Theorem [2](https://arxiv.org/html/2509.22459v1#Thmtheorem2)). Putting the explicit values for RealUM loss ([17](https://arxiv.org/html/2509.22459v1#S3.E17)) in RealUID loss ([18](https://arxiv.org/html/2509.22459v1#S3.E18)), we get the explicit formula:

$$ $\displaystyle\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}[-\alpha\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\alpha\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\beta\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle]$ $\displaystyle+\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{0}\sim p^{*}_{0},x_{t}^{*}\sim p_{t}^{*}(\cdot|x^{*}_{0})}[-(1-\alpha)\|\delta_{t}(x_{t}^{*})\|^{2}+2(1-\alpha)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2(1-\beta)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\rangle].$ $$

These two formulas give us alternative explanation on how to add real data into arbitrary losses: we need to split each term in the linearized representation of the data-free loss between real and generated data. For example, in RealUID loss, its three terms are split with proportions $\alpha,\alpha,\beta,$ respectively. We can go even further and split the first quadratic coefficient $-\|\delta_{t}(\cdot)\|^{2}$ using a new parameter $\gamma\in(0,1]$ to create one more degree of freedom. Moreover, we can use other parametrization of $\delta$, since its form does not change the proofs.

###### Definition 3 .

We introduce General RealUID loss $\mathcal{L}^{\alpha,\beta,\gamma}_{\text{R-UID}}(\delta,p_{0}^{\theta})$ on generated data $p^{\theta}_{0}\in\mathcal{P}(\mathbb{R}^{D})$ with coefficients $\alpha,\beta,\gamma\in(0,1]$:

$$ $\displaystyle\mathcal{L}^{\alpha,\beta,\gamma}_{\text{R-UID}}(\delta,p_{0}^{\theta}):=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}[-\gamma\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\alpha\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\beta\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle]$ $\displaystyle+\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{0}\sim p^{*}_{0},x_{t}^{*}\sim p_{t}^{*}(\cdot|x^{*}_{0})}[-(1-\gamma)\|\delta_{t}(x_{t}^{*})\|^{2}+2(1-\alpha)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2(1-\beta)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\rangle].$ $$

Optionally, one can change default reparameterization $\delta=f^{*}-f$ or substitute sampled real data term $f^{*}_{t}(x^{*}_{t}|x_{0}^{*})$ with the unconditional teacher $f^{*}_{t}(x^{*}_{t})$.

In case of $\delta=f^{*}-f$ and $\gamma\neq\alpha$, the General RealUID loss cannot be expressed as inverse min-max problem ([16](https://arxiv.org/html/2509.22459v1#S3.E16)) for simple losses, since some scalar products do not eliminate each other. Nevertheless, min-max optimization of $\mathcal{L}^{\alpha,\beta,\gamma}_{\text{R-UID}}$ still minimizes the similar squared $\ell_{2}$-distance between the weighted teacher and generator-induced functions, attaining minimum when $p^{\theta}_{0}=p^{*}_{0}$.

###### Lemma 3 ( Distance minimized by General RealUID loss ) .

Maximization of General RealUID loss $\mathcal{L}^{\alpha,\beta,\gamma}_{\text{R-UID}}$ over $\delta$ represents the squared $\ell_{2}$-distance between the weighted teacher $f^{*}$ and student function $f^{\theta}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{\theta})$:

$$ $\max_{\delta}\mathcal{L}^{\alpha,\beta,\gamma}_{\text{R-UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p_{t}^{*}}\left[\frac{\|(p_{t}^{*}(x_{t}^{*})(\beta-\alpha)+\alpha p_{t}^{\theta}(x_{t}^{*}))\cdot f^{*}_{t}(x_{t}^{*})-\beta p_{t}^{\theta}(x_{t}^{*})\cdot f^{\theta}_{t}(x_{t}^{*})\|^{2}}{p_{t}^{*}(x_{t}^{*})((1-\gamma)p_{t}^{*}(x_{t}^{*})+\gamma p_{t}^{\theta}(x_{t}^{*}))}\right].$ $$

The distances being minimized for RealUID (Lemma [2](https://arxiv.org/html/2509.22459v1#Thmlemma2)) and General RealUID (Lemma [3](https://arxiv.org/html/2509.22459v1#Thmlemma3)) are almost identical except the scale factor in the denominator. Thus, we keep the same recommendations for choosing coefficients $\alpha,\beta$ as we discuss in §[A.1.2](https://arxiv.org/html/2509.22459v1#A1.SS1.SSS2). The optimal choice is slightly different $\alpha\neq\beta$ which are close to $1$. Coefficient $\gamma$ should also be close to $1$, but not exactly $1.$

###### Proof.

First, we write down General RealUID loss (Def. [3](https://arxiv.org/html/2509.22459v1#Thmdefinition3)) in an explicit form with $\delta_{t}=f^{*}_{t}-f_{t}$

$$ $\displaystyle\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p^{\theta}_{t}}[-\gamma\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\alpha\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\beta\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t})\rangle]$ $\displaystyle+\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t}}[-(1-\gamma)\|\delta_{t}(x_{t}^{*})\|^{2}+2(1-\alpha)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2(1-\beta)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle].$ $$

Then, we rescale the generated data terms in the General RealUID loss using the equality $p^{\theta}_{t}(x_{t})=\frac{p^{\theta}_{t}(x_{t})}{p^{*}_{t}(x_{t})}p^{*}_{t}(x_{t})$ for $x_{t}\in\mathbb{R}^{D}$ (we assume $p_{t}^{*}(x_{t})>0,\forall x_{t},t$) leaving only math expectation w.r.t. the real data, i.e, $\mathcal{L}^{\alpha,\beta,\gamma}_{\text{R-UID}}(\delta,p_{0}^{\theta})=$

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t}}\left[-[(1-\gamma)+\gamma\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]\|\delta_{t}(x_{t}^{*})\|^{2}+2[(\beta-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2\beta\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}\langle\delta_{t}(x_{t}^{*}),f^{\theta}_{t}(x^{*}_{t})\rangle\right].$ $$

Then we maximize the loss w.r.t. $\delta_{t}(x_{t}^{*})$ for each $x_{t}^{*}$ and $t$ as a quadratic function. The maximum is achieved when

$$ $\delta_{t}(x_{t}^{*})=\frac{[(\beta-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]f^{*}_{t}(x^{*}_{t})-\beta\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}f^{\theta}_{t}(x^{*}_{t})}{[(1-\gamma)+\gamma\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}]}.$ $$

The maximum itself equals to

$$ $\max_{\delta}\mathcal{L}^{\alpha,\beta,\gamma}_{\text{R-UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p_{t}^{*}}\left[\frac{\|f^{*}_{t}(x_{t}^{*})\cdot((\beta-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})})-f^{\theta}_{t}(x_{t}^{*})\cdot\beta\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}\|^{2}}{(1-\gamma)+\gamma\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}}\right].$ $$

∎

###### Definition 3 .

###### Lemma 3 ( Distance minimized by General RealUID loss ) .

###### Proof.

### A.3 SiD with real data

We recall that data-free UID loss (Theorem [1](https://arxiv.org/html/2509.22459v1#Thmtheorem1)), which is equivalent to SiD loss with $\alpha_{\text{SiD}}=1/2$, can be restated via linearization technique with $\delta=f-f^{*}$ as

$$ $\mathcal{L}_{\text{UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}\left\{-\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}.$ (23) $$

In turn, after real data incorporation, we obtain our RealUID loss (Theorem [2](https://arxiv.org/html/2509.22459v1#Thmtheorem2)). Putting the explicit values for RealUM loss ([17](https://arxiv.org/html/2509.22459v1#S3.E17)) in RealUID loss ([18](https://arxiv.org/html/2509.22459v1#S3.E18)), we get the explicit formula:

$$ $\displaystyle\mathcal{L}^{\alpha,\beta}_{\text{R-UID}}(\delta,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}[-\alpha\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\alpha\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\beta\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle]$ $\displaystyle+\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{0}\sim p^{*}_{0},x_{t}^{*}\sim p_{t}^{*}(\cdot|x^{*}_{0})}[-(1-\alpha)\|\delta_{t}(x_{t}^{*})\|^{2}+2(1-\alpha)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t})\rangle-2(1-\beta)\langle\delta_{t}(x_{t}^{*}),f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\rangle].$ $$

These two formulas give us alternative explanation on how to add real data into arbitrary losses: we need to split each term in the linearized representation of the data-free loss between real and generated data. For example, in RealUID loss, its three terms are split with proportions $\alpha,\alpha,\beta,$ respectively.

In SiD framework (Zhou et al., 2024a; b), the authors notice that UID loss ([23](https://arxiv.org/html/2509.22459v1#A1.E23)), in which the first coefficient $-\|\delta_{t}(x_{t}^{\theta})\|^{2}$ is scaled by $2\alpha_{\text{SiD}}$ during generator updates, empirically yields better performance. Namely, the SiD loss for generator with parameter $\alpha_{\text{SiD}}$ is

$$ $\mathcal{L}_{\alpha_{\text{SiD}}}(p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}\left\{-2\alpha_{\text{SiD}}\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}.$ $$

Following the structure of generator SiD loss, we propose to scale the first coefficient in our RealUID loss during generator updates. The whole SiD pipeline with real data determined by coefficients $\alpha,\beta\in(0,1],\alpha_{\text{SiD}}$ and teacher $f^{*}$ is two alternating steps:

- 1.
Minimize the real data modified UM loss $\mathcal{L}^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{\theta})$ (Def. [2](https://arxiv.org/html/2509.22459v1#Thmdefinition2)) for the fake model $f$ via several update steps:
$\displaystyle L^{\alpha,\beta}_{\text{R-UM}}(f,p_{0}^{\theta})$
$\displaystyle=$
$\displaystyle\underset{\text{generated data }p^{\theta}_{0}\text{ term}}{\underbrace{\alpha\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}\left[\|f_{t}(x^{\theta}_{t})-\frac{\beta}{\alpha}f^{\theta}(x^{\theta}_{t}|x^{\theta}_{0})\|^{2}\right]}}$
$\displaystyle+$
$\displaystyle\underset{\text{real data }p^{*}_{0}\text{ term}}{\underbrace{(1-\alpha)\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{0}\sim p^{*}_{0},x_{t}^{*}\sim p_{t}^{*}(\cdot|x^{*}_{0})}\left[\|f_{t}(x^{*}_{t})-\frac{1-\beta}{1-\alpha}f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\|^{2}\right]}}.$
- 2.
Make generator update step minimizing the loss $\mathcal{L}^{\alpha,\beta}_{\text{R-UID},\alpha_{\text{SiD}}}(p_{0}^{\theta})$ with $\delta=f-f^{*}:$
$\mathcal{L}^{\alpha,\beta}_{\text{R-UID},\alpha_{\text{SiD}}}(p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}\left\{-2\alpha_{\text{SiD}}\cdot\alpha\cdot\|\delta_{t}(x_{t}^{\theta})\|^{2}+2\alpha\langle\delta_{t}(x_{t}^{\theta}),f^{*}_{t}(x^{\theta}_{t})\rangle-2\beta\langle\delta_{t}(x_{t}^{\theta}),f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}.$
We keep the same recommendations for choosing coefficients $\alpha,\beta$ as we discuss in Appendix [A.1.2](https://arxiv.org/html/2509.22459v1#A1.SS1.SSS2). The optimal choice is slightly different $\alpha\neq\beta$ which are close to $1$. Following (Zhou et al., 2024a), the best choice for $\alpha_{\text{SiD}}$ is $\alpha_{\text{SiD}}\in[1,1.2].$

### A.4 Normalized UID and RealUID losses for minimizing ℓ 2 \ell_{2} -distance

Using the linearization technique from Section [3.1](https://arxiv.org/html/2509.22459v1#S3.SS1), we can estimate the non-squared $\ell_{2}$-distance between the teacher $f^{*}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{*})$ and student $f^{\theta}:=\operatorname*{arg\,min}_{f}\mathcal{L}_{\text{UM}}(f,p_{0}^{\theta})$ functions. In this case, the connection with the inverse optimization disappears.

For a fixed point $x^{\theta}_{t}$ and time $t$, we derive:

$$ $\displaystyle\|f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t})\|$ $\displaystyle=$ $\displaystyle\max_{\delta_{t}(x_{t}^{\theta})}\left\{\langle\frac{\delta_{t}(x_{t}^{\theta})}{\|\delta_{t}(x_{t}^{\theta})\|},f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t})\rangle\right\}$ (24) $\displaystyle=$ $\displaystyle\max_{\delta_{t}(x_{t}^{\theta})}\mathbb{E}_{x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}\left\{\langle\frac{\delta_{t}(x_{t}^{\theta})}{\|\delta_{t}(x_{t}^{\theta})\|},f^{*}_{t}(x^{\theta}_{t})\rangle-\langle\frac{\delta_{t}(x_{t}^{\theta})}{\|\delta_{t}(x_{t}^{\theta})\|},f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}.$ $$

With the reparameterization $\delta_{t}=f^{*}_{t}-f_{t}$, the Normalized UID loss $\hat{\mathcal{L}}_{\text{UID}}(f,p_{0}^{\theta})$ for min-max optimization to solve $\min_{\theta}\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}\|f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t})\|$ is:

$$ $\displaystyle\min_{\theta}\max_{f}\left\{\hat{\mathcal{L}}_{\text{UID}}(f,p_{0}^{\theta}):=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x_{t}^{\theta}\sim p_{t}^{\theta}(\cdot|x^{\theta}_{0})}\left[\langle\frac{f^{*}_{t}(x_{t}^{\theta})-f_{t}(x_{t}^{\theta})}{\|f^{*}_{t}(x_{t}^{\theta})-f_{t}(x_{t}^{\theta})\|},f^{*}_{t}(x^{\theta}_{t})-f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right]\right\}.$ (25) $$

##### Adding real data.

Following the intuition from the proof for RealUID in Appendix [A.1.1](https://arxiv.org/html/2509.22459v1#A1.SS1.SSS1), we can incorporate real data in Normalized UID loss ([25](https://arxiv.org/html/2509.22459v1#A1.E25)) as well. We need to split two summands in the linearized representation ([24](https://arxiv.org/html/2509.22459v1#A1.E24)) into generated and real data parts with weights $\alpha,(1-\alpha)$ and
$\beta,(1-\beta)$.

###### Definition 4 .

We introduce Normalized RealUID loss $\hat{\mathcal{L}}_{\text{R-UID}}^{\alpha,\beta}(f,p_{0}^{\theta})$ on generated data $p^{\theta}_{0}\in\mathcal{P}(\mathbb{R}^{D})$ with coefficients $\alpha,\beta\in(0,1]$:

$$ $\displaystyle\hat{\mathcal{L}}_{\text{R-UID}}^{\alpha,\beta}(f,p_{0}^{\theta}):=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p^{\theta}_{t},x_{0}^{\theta}\sim p_{0}^{\theta}(\cdot|x^{\theta}_{t})}\left\{\langle\frac{f^{*}_{t}(x_{t}^{\theta})-f_{t}(x_{t}^{\theta})}{\|f^{*}_{t}(x_{t}^{\theta})-f_{t}(x_{t}^{\theta})\|},\alpha\cdot f^{*}_{t}(x^{\theta}_{t})-\beta\cdot f^{\theta}_{t}(x^{\theta}_{t}|x_{0}^{\theta})\rangle\right\}$ $\displaystyle+\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t},x_{0}^{*}\sim p_{0}^{*}(\cdot|x^{*}_{t})}\left\{\langle\frac{f^{*}_{t}(x_{t}^{*})-f_{t}(x_{t}^{*})}{\|f^{*}_{t}(x_{t}^{*})-f_{t}(x_{t}^{*})\|},(1-\alpha)\cdot f^{*}_{t}(x^{*}_{t})-(1-\beta)\cdot f^{*}_{t}(x^{*}_{t}|x_{0}^{*})\rangle\right\}.$ $$

Similar to the proof of RealUID distance Lemma [2](https://arxiv.org/html/2509.22459v1#Thmlemma2), we can show that min-max optimization of Normalized RealUID loss minimizes the non-squared $\ell_{2}$-norm between the similar weighted student $f^{\theta}$ and teacher $f^{*}$ functions:

$$ $\max_{f}\hat{\mathcal{L}}^{\alpha,\beta}_{\text{R-UID}}(f,p_{0}^{\theta})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p_{t}^{*}}\left[\|((\beta-\alpha)+\alpha\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})})\cdot f^{*}_{t}(x_{t}^{*})-\beta\frac{p_{t}^{\theta}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}\cdot f^{\theta}_{t}(x_{t}^{*})\|\right].$ $$

This distance attains minimum when $p^{\theta}_{0}=p^{*}_{0}$, justifying the procedure.

###### Definition 4 .

### A.5 DMD approach with real data

Distribution Matching Distillation (Luo et al., 2023; Wang et al., 2023; Yin et al., 2024b; a) (DMD) approach distills Gaussian diffusion models with forward process $x_{t}=x_{0}+\sigma_{t}\epsilon,\epsilon\sim\mathcal{N}(0,I).$

This approach minimizes KL divergence $\mathbb{E}_{t\sim[0,T]}D_{KL}(p_{t}^{\theta}||p_{t}^{*})=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x_{t}^{\theta}\sim p_{t}^{\theta}}\left[\log\left(\frac{p_{t}^{\theta}(x_{t}^{\theta})}{p_{t}^{*}(x_{t}^{\theta})}\right)\right]$ between the generated data $p_{t}^{\theta}$ and the real data $p_{t}^{*}$. The authors show the true gradient of $\mathbb{E}_{t\sim[0,T]}D_{KL}(p_{t}^{\theta}||p_{t}^{*})$ w.r.t. $\theta$ can be computed via the score functions:

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\left[\frac{dD_{KL}(p_{t}^{\theta}||p_{t}^{*})}{d\theta}\right]=\mathbb{E}_{z\sim p^{\mathcal{Z}},x_{0}^{\theta}=G(z),x^{\theta}_{t}\sim p_{t}^{\theta}}\left[(\nabla_{x_{t}^{\theta}}\ln p_{t}^{\theta}(x_{t}^{\theta})-\nabla_{x_{t}^{\theta}}\ln p_{t}^{*}(x_{t}^{\theta}))\frac{dG_{\theta}(z)}{d\theta}\right].$ $$

Then, this true gradient is estimated with the teacher score function $s^{*}:=\operatorname*{arg\,min}_{s}\mathcal{L}_{\text{DSM}}(s,p_{0}^{*})$ and student score $s^{\theta}=\operatorname*{arg\,min}_{s}\mathcal{L}_{\text{DSM}}(s,p_{0}^{\theta})$ at each time moment:

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\left[\frac{dD_{KL}(p_{t}^{\theta}||p_{t}^{*})}{d\theta}\right]=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{z\sim p^{\mathcal{Z}},x_{0}^{\theta}=G_{\theta}(z),x^{\theta}_{t}\sim p_{t}^{\theta}}\left[(s_{t}^{\theta}(x_{t}^{\theta})-s_{t}^{*}(x_{t}^{\theta}))\frac{dG_{\theta}}{d\theta}\right].$ $$

The final algorithm alternates updates for the fake model and the generator similar to SiD approach.

We would like to highlight that DMD does not fit our UID framework. The UID loss is uniquely determined by its input UM loss. In the case of Diffusion models and DMD, the UM loss is the $\mathcal{L}_{DSM}(s,p_{0}^{\theta})$ loss. With this loss, the resulting UID loss becomes exactly the SiD loss, not DMD.

Nevertheless, we investigated an opportunity to incorporate real data into the DMD framework. We found that we can use the Modified DSM loss ([17](https://arxiv.org/html/2509.22459v1#S3.E17)) to train the modified student score function $s_{t}^{\theta,\alpha}=\operatorname*{arg\,min}_{s}\mathcal{L}^{\alpha,\alpha}_{M-DSM}(s,p_{0}^{\theta})$ with coefficients $\alpha=\beta$:

$$ $\displaystyle\mathcal{L}^{\alpha,\alpha}_{M-DSM}(s,p_{0}^{\theta})$ $\displaystyle:=$ $\displaystyle\underset{\text{generated data }p^{\theta}_{0}\text{ term}}{\underbrace{\alpha\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{0}\sim p^{\theta}_{0},x^{\theta}_{t}\sim p^{\theta}_{t}(\cdot|x_{0})}\left[\|s_{t}(x^{\theta}_{t})-s^{\theta}(x^{\theta}_{t}|x^{\theta}_{0})\|^{2}\right]}}$ $\displaystyle+$ $\displaystyle\underset{\text{real data }p^{*}_{0}\text{ term}}{\underbrace{(1-\alpha)\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{0}\sim p^{*}_{0},x_{t}^{*}\sim p_{t}^{*}(\cdot|x^{*}_{0})}\left[\|s_{t}(x^{*}_{t})-s^{*}_{t}(x^{*}_{t}|x_{0}^{*})\|^{2}\right]}}.$ $$

Then apply the generator parameters update based on the KL divergence between mixed distributions.

###### Lemma 4 ( DMD with real data ) .

Consider real data distribution $p^{*}_{0}\in\mathcal{P}(\mathbb{R}^{D})$ and generated by generator $G_{\theta}$ distribution $p_{0}^{\theta}\in\mathcal{P}(\mathbb{R}^{D}).$ Then, KL divergence between mixed and real data for $\alpha\in(0,1]$ has the following gradients with modified student score $s_{t}^{\theta,\alpha}:=\operatorname*{arg\,min}_{s}\mathcal{L}^{\alpha,\alpha}_{\text{M-DSM}}(s,p_{0}^{\theta})$ and teacher score $s_{t}^{*}:=\operatorname*{arg\,min}_{s}\mathcal{L}_{\text{DSM}}(s,p_{0}^{*})$:

$$ $\displaystyle\mathbb{E}_{t\sim[0,T]}\left[\frac{dD_{KL}(\alpha\cdot p_{t}^{\theta}+(1-\alpha)\cdot p_{t}^{*}||p_{t}^{*})}{d\theta}\right]=\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{z\sim p^{\mathcal{Z}},x_{0}^{\theta}=G_{\theta}(z),x^{\theta}_{t}\sim p_{t}^{\theta}}\left[\alpha(s_{t}^{\theta,\alpha}(x_{t}^{\theta})-s_{t}^{*}(x_{t}^{\theta}))\frac{dG_{\theta}}{d\theta}\right].$ $$

In the proof below, we show that use of coefficients $\alpha\neq\beta$ in the fake model loss leads to the total collapse of a generator. The proof itself follows the work (Wang et al., 2023).

###### Proof.

We aim to minimize KL divergence between generated distribution $p^{\theta}_{0}$ and the real data $p_{0}^{*}$

$$ $\min_{p_{0}^{\theta}}E(p_{0}^{\theta}):=\mathbb{E}_{t\sim[0,T]}\left[D_{KL}(\alpha\cdot p^{\theta}_{t}+(1-\alpha)\cdot p_{t}^{*}||p_{t}^{*})\right].$ $$

First, we use (Wang et al., 2023, Lemma 1) which says that, for any two distributions $p,q\in\mathcal{P}(\mathbb{R}^{D})$ and point $x\in\mathbb{R}^{D}$, we have

$$ $\left(\frac{\delta D_{KL}(q||p)}{\delta q}\right)[x]=\log q(x)-\log p(x)+1.$ $$

Second, for the parametrization $x_{0}^{\theta}=G_{\theta}(z),z\sim p^{\mathcal{Z}}$ and a fixed point $x_{t}$, we have (Wang et al., 2023, Lemma 2)

$$ $\frac{\delta p_{t}^{\theta}(x_{t})}{\delta p_{0}^{\theta}}[\theta]=\int_{z}p_{t}^{\theta}(x_{t}|x^{\theta}_{0})p^{\mathcal{Z}}(z)dz.$ $$

It allows us to obtain

$$ $\displaystyle\frac{\delta E(p_{0}^{\theta})}{\delta p_{0}^{\theta}}[\theta]$ $\displaystyle=$ $\displaystyle\mathbb{E}_{t}\frac{\delta D_{KL}(\overset{=:q_{t}}{\overbrace{\alpha\cdot p_{t}^{\theta}(\cdot)+(1-\alpha)\cdot p_{t}^{*}(\cdot)}}||p_{t}^{*}(\cdot))}{\delta p_{0}^{\theta}}[\theta]$ (26) $\displaystyle=$ $\displaystyle\mathbb{E}_{t}\int\frac{\delta D_{KL}(q_{t}||p_{t}^{*})}{\delta q_{t}}[x_{t}]\cdot\frac{\delta q_{t}}{\delta p^{\theta}_{t}}[x_{t}]\cdot\frac{\delta p_{t}^{\theta}(x_{t})}{\delta p_{0}^{\theta}}[\theta]\cdot dx_{t}$ $\displaystyle=$ $\displaystyle\mathbb{E}_{t}\int\left[\log(\alpha\cdot p^{\theta}_{t}(x_{t})+(1-\alpha)\cdot p_{t}^{*}(x_{t}))-\log(p_{t}^{*}(x_{t}))+1\right]\cdot\alpha\cdot\int_{z}p_{t}^{\theta}(x_{t}|x^{\theta}_{0})p^{\mathcal{Z}}(z)dz\cdot dx_{t}$ $\displaystyle=$ $\displaystyle\mathbb{E}_{t,\epsilon,z}[\alpha\log(\alpha\cdot p^{\theta}_{t}(x^{\theta}_{t})+(1-\alpha)\cdot p_{t}^{*}(x^{\theta}_{t}))-\alpha\log(p_{t}^{*}(x^{\theta}_{t}))+\alpha]$ $\displaystyle=$ $\displaystyle\mathbb{E}_{t,\epsilon,z}[\alpha\log\left(\alpha\cdot\frac{p^{\theta}_{t}(x^{\theta}_{t})}{p_{t}^{*}(x_{t}^{\theta})}+(1-\alpha)\right)+\alpha],$ $$

where $x^{\theta}_{0}=G_{\theta}(z),x^{\theta}_{t}=x^{\theta}_{0}+\sigma_{t}\epsilon,\epsilon\sim\mathcal{N}(0,I).$ Finally, we take derivative w.r.t. $\theta$ from ([26](https://arxiv.org/html/2509.22459v1#A1.E26)):

$$ $\displaystyle\nabla_{\theta}\frac{\delta E(p_{0}^{\theta})}{\delta p_{0}^{\theta}}[\theta]$ $\displaystyle=$ $\displaystyle\mathbb{E}_{t,\epsilon,z}\left[\alpha\cdot\nabla_{x^{\theta}_{t}}\log\left(\alpha\cdot\frac{p^{\theta}_{t}(x^{\theta}_{t})}{p_{t}^{*}(x^{\theta}_{t})}+(1-\alpha)\right)\cdot\frac{\partial x_{t}^{\theta}}{\partial\theta}\right]$ (27) $\displaystyle=$ $\displaystyle\mathbb{E}_{t,\epsilon,z}\left[\alpha\cdot\nabla_{x^{\theta}_{t}}\log\left(\alpha\cdot\frac{p^{\theta}_{t}(x^{\theta}_{t})}{p_{t}^{*}(x^{\theta}_{t})}+(1-\alpha)\right)\cdot\frac{\partial G_{\theta}(z)}{\partial\theta}\right]$ $\displaystyle=$ $\displaystyle\mathbb{E}_{t,\epsilon,z}\left[\alpha^{2}\frac{\nabla_{x^{\theta}_{t}}\nicefrac{{p^{\theta}_{t}(x^{\theta}_{t})}}{{p_{t}^{*}(x^{\theta}_{t})}}}{\alpha\cdot\frac{p^{\theta}_{t}(x^{\theta}_{t})}{p^{*}(x_{t}^{\theta})}+(1-\alpha)}\cdot\frac{\partial G_{\theta}(z)}{\partial\theta}\right].$ $$

Now, we show how to obtain unbiased estimate of this gradient. We minimize the following loss function over the fake model $s$:

$$ $\displaystyle\mathcal{L}^{\alpha,\alpha}_{M-DSM}(s,p_{0}^{\theta})$ $\displaystyle:=$ $\displaystyle\alpha\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p^{\theta}_{t},x^{\theta}_{0}\sim p^{\theta}_{0}(\cdot|x_{t})}\left[\|s_{t}(x^{\theta}_{t})-s^{\theta}(x^{\theta}_{t}|x^{\theta}_{0})\|^{2}\right]$ $\displaystyle+$ $\displaystyle(1-\alpha)\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t},x_{0}^{*}\sim p_{0}^{*}(\cdot|x^{*}_{t})}\left[\|s_{t}(x^{*}_{t})-s^{*}_{t}(x^{*}_{t}|x_{0}^{*})\|^{2}\right].$ $$

This loss is equivalent to the following sequence

$$ $\displaystyle\min_{s}\left\{\alpha\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}\|s_{t}(x^{\theta}_{t})-s^{\theta}_{t}(x_{t}^{\theta})\|^{2}+(1-\alpha)\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t}}\|s_{t}(x_{t}^{*})-s^{*}_{t}(x_{t}^{*})\|^{2}\right\},$ $$

$$ $\displaystyle\min_{s}\left\{\alpha\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{\theta}_{t}\sim p_{t}^{\theta}}\|s_{t}(x^{\theta}_{t})-\nabla_{x_{t}^{\theta}}\log p^{\theta}_{t}(x^{\theta}_{t})\|^{2}+(1-\alpha)\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t}}\|s_{t}(x_{t}^{*})-\nabla_{x_{t}^{*}}\log p_{t}^{*}(x^{*}_{t})\|^{2}\right\},$ $$

$$ $\displaystyle\min_{s}\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x^{*}_{t}\sim p^{*}_{t}}\left[\alpha\|s_{t}(x^{*}_{t})-\nabla\log p^{\theta}_{t}(x_{t}^{*})\|^{2}\frac{p^{\theta}_{t}(x_{t}^{*})}{p_{t}^{*}(x_{t}^{*})}+(1-\alpha)\|s_{t}(x_{t}^{*})-\nabla\log p_{t}^{*}(x_{t}^{*})\|^{2}\right].$ $$

The optimal solution $s^{\theta,\alpha}$ of this quadratic minimization for each point $x_{t}$ and time moment $t$ is

$$ $s^{\theta,\alpha}_{t}(x_{t})=\frac{\alpha\frac{p^{\theta}_{t}(x_{t})}{p_{t}^{*}(x_{t})}\nabla_{x_{t}}\log p_{t}^{\theta}(x_{t})+(1-\alpha)\nabla_{x_{t}}\log p_{t}^{*}(x_{t})}{\alpha\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}+(1-\alpha)}.$ $$

Thus, we have the following estimate with modified student score $s^{\theta,\alpha}$ and teacher score $s^{*}_{t}(x_{t}):=\nabla_{x_{t}}\log p_{t}^{*}(x_{t})$

$$ $\displaystyle s^{\theta,\alpha}_{t}(x_{t})-s^{*}_{t}(x_{t})$ $\displaystyle=$ $\displaystyle\frac{\alpha\frac{p^{\theta}_{t}(x_{t})}{p_{t}^{*}(x_{t})}\nabla_{x_{t}}\log p^{\theta}_{t}(x_{t})+(1-\alpha)\nabla_{x_{t}}\log p_{t}^{*}(x_{t})}{\alpha\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}+(1-\alpha)}-\nabla_{x_{t}}\log p_{t}^{*}(x_{t})$ $\displaystyle=$ $\displaystyle\frac{\alpha\frac{p^{\theta}_{t}(x_{t})}{p_{t}^{*}(x_{t})}(\nabla_{x_{t}}\log p_{t}^{\theta}(x_{t})-\nabla_{x_{t}}\log p_{t}^{*}(x_{t}))}{\alpha\frac{p^{\theta}_{t}(x_{t})}{p_{t}^{*}(x_{t})}+(1-\alpha)}$ $\displaystyle=$ $\displaystyle\frac{\alpha\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}\nabla_{x_{t}}\log\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}}{\alpha\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}+(1-\alpha)}=\frac{\alpha\nabla_{x_{t}}\nicefrac{{p^{\theta}_{t}(x_{t})}}{{p_{t}^{*}(x_{t})}}}{\alpha\frac{p_{t}^{\theta}(x_{t})}{p_{t}^{*}(x_{t})}+(1-\alpha)}.$ $$

Hence, this estimate completely matches with required gradient ([27](https://arxiv.org/html/2509.22459v1#A1.E27)):

$$ $\displaystyle\eqref{eq: DMD grad}=\mathbb{E}_{t,\epsilon,z}\left[\alpha\cdot(s^{\theta,\alpha}(x^{\theta}_{t})-s^{*}_{t}(x_{t}^{\theta}))\cdot\frac{\partial G_{\theta}(z)}{\partial\theta}\right].$ $$

Use of other coefficients during student score optimization does not work. For the other student scores $s_{t}^{\theta,\alpha,\beta}:=\operatorname*{arg\,min}_{s}\mathcal{L}^{\alpha,\beta}_{\text{M-DSM}}(s,p_{0}^{\theta})$, the estimate $s_{t}^{\theta,\alpha,\beta}(x_{t})-\nabla_{x_{t}}\log p_{t}^{*}(x_{t})$ does not lead to the necessary difference $\nabla_{x_{t}}\log p_{t}^{\theta}(x_{t})-\nabla_{x_{t}}\log p_{t}^{*}(x_{t}|c)=0$. And the optimal generator collapses due to large bias.

∎

###### Lemma 4 ( DMD with real data ) .

###### Proof.

## Appendix B RealUID Algorithm for Flow Matching models

We provide a practical implementation of our RealUID approach for FM in Algorithm [1](https://arxiv.org/html/2509.22459v1#alg1). In the loss functions, we retain only the terms dependent on the target parameters. For the fake model, we reformulate the maximization objective as a minimization. We use alternating optimization, updating the fake model $K$ times per one student update for stability.

Figure: Algorithm 1 Real data modified Unified Inversion Distillation (RealUID) for Flow Matching

## Appendix C Unified Inverse Disillation for Bridge Matching and Stochastic Interpolants

### C.1 Bridge Matching

Bridge Matching (Liu et al., 2022b; Peluchetti, 2023) is an extension of diffusion models specifically design to solve data-to-data, e.g. image-to-image problems. Typically, the distribution $p_{T}$ is the distribution of ”corrupted data” and $p_{0}$ is the distribution of clean data, furthermore, there is some coupling of clean and corrupted data $\pi(x_{0},x_{T})$ with marginals $p_{0}(x_{0})$ and $p_{T}(x_{T})$. To construct the diffusion which recovers clean data given a corrupted data, one first needs to build prior process (which often is the same forward process used in diffusions):

$$ $dx_{t}=f_{t}(x_{t})+g_{t}dw_{t},$ $$

where $f_{t}(\cdot)$ is a drift function and $g_{t}$ is a time-dependent scalar noise scheduler. This prior process defines conditional density $p_{t}(x_{t}|x_{0})$ and the posterior density $p_{t}(x_{t}|x_{0},x_{T})$ called ”diffusion bridge”. To recover $p_{0}$ from $p_{T}$, one can use reverse-time SDE

$$ $dx_{t}=\left(f_{t}(x_{t})-g^{2}_{t}\cdot v^{\pi}(x_{t})\right)dt+g_{t}d\bar{w}_{t},$ $$

where the drift $v^{\pi}_{t}(x_{t})$ is learned via solving of the bridge matching problem:

$$ $\mathcal{L}_{\text{BM}}(v,\pi)=\mathbb{E}_{t\sim[0,T],(x_{0},x_{T})\sim\pi(x_{0},x_{T}),x_{t}\sim p_{t}(x_{t}|x_{0},x_{T})}\left[w_{t}\|v_{t}(x_{t})-\nabla_{x_{t}}\log p_{t}(x_{t}|x_{0})\|^{2}\right].$ (28) $$

However, this reverse-time diffusion in general does not guarantee that the produced samples come from the same coupling $\pi(x_{0},x_{T})$ used for training. This happens only if $\pi(x_{0},x_{T})$ solves entropic optimal transport between $p_{0}$ and $p_{T}$. To guarantee the preservance of the coupling $\pi(x_{0},x_{T})$, there exists another version of Bridge Matching called either Augmented Bridge Matching or Conditional Bridge Matching, which differs only by addition of a condition on $x_{T}$ to the drift function $v_{t}(x_{t},x_{T})$:

$$ $\mathcal{L}_{\text{ABM}}(v,\pi)=\mathbb{E}_{t\sim[0,T],(x_{0},x_{T})\sim\pi(x_{0},x_{T}),x_{t}\sim p(x_{t}|x_{0},x_{T})}\left[w_{t}\|v_{t}(x_{t},x_{T})-\nabla_{x_{t}}\log p_{t}(x_{t}|x_{0})\|_{2}^{2}\right].$ $$

The learned conditional drift is then used for sampling via the reverse-time SDE starting from a given $x_{T}\sim p_{T}$:

$$ $dx_{t}=\left(f_{t}(x_{t})-g^{2}_{t}\cdot v^{\pi}_{t}(x_{t},x_{T})\right)dt+g_{t}d\bar{w}_{t}.$ $$

### C.2 Stochastic Interpolants

The Stochastic Interpolants framework generalizes Flow Matching and diffusion models, constructing a diffusion or flow between two given distributions $p_{0}$ and $p_{T}$. To do so, one needs to consider the interpolation between any pair of points $(x_{0},x_{T})$ which are sampled from the coupling $\pi(x_{0},x_{T})$ with marginals $p_{0}$ and $p_{T}$. The interpolation itself is given by formula

$$ $x_{t}=I(t,x_{0},x_{T})+\gamma_{t}\epsilon,\quad\epsilon\sim\mathcal{N}(0,\mathbf{I}),\quad t\in[0,T],$ $$

where $I(0,x_{0},x_{T})=x_{0}$, $I(T,x_{0},x_{T})=x_{T}$, $\gamma_{0}=\gamma_{T}=0$ and $\gamma_{t}>0$ for all $t\in(0,T)$. This interpolant defines a conditional Gaussian path $p_{t}(x_{t}|x_{0},x_{T})$. Note that in the original paper (Albergo et al., 2023), the authors consider the time interval $[0,1]$, but those two intervals are interchangeable by using a change of variable $t^{\prime}=\frac{T}{t}$. Thus, the ODE interpolation between $p_{0}$ and $p_{T}$ is given by:

$$ $dx_{t}=u_{t}(x_{t})dt,\quad x_{0}\sim p_{0},$ $$

where $u_{t}(x,x_{T}):=\mathbb{E}[\dot{x}_{t}|x_{t}=x]=\mathbb{E}[\partial_{t}I(t,x_{0},x_{T})+\dot{\gamma}\epsilon|x_{t}=x]$ is the unique minimizer of the quadratic objective:

$$ $\mathcal{L}_{\text{SI}}(v,\pi)=\mathbb{E}_{t\sim[0,T],(x_{0},x_{T})\sim\pi(x_{0},x_{T}),(x_{t},\epsilon)\sim p(x_{t}|x_{0},x_{T})}\left[w_{t}\|v_{t}(x_{t},x_{T})-(\partial_{t}I(t,x_{0},x_{T})+\dot{\gamma}_{t}\epsilon))\|^{2}\right].$ (29) $$

The authors also provide a way of matching the score and the SDE drift of the reverse process by solving similar MSE matching problems.

### C.3 Objective for Unified Inverse Distillation for general data coupling

The essential difference of Bridge Matching and Stochastic Interpolants from diffusion models and Flow Matching with a Gaussian path is that they additionally introduce coupling $\pi(x_{0},x_{T})$ used to sample $x_{t}$ and can work with conditional drifts.

This difference can be easily incorporated to our RealUID distillation framework just by parametrizing the generator $G_{\theta}$ to output not the samples from the initial distribution $p_{0}^{\theta}$, but from the coupling $\pi^{\theta}$. One can do it by setting $\pi^{\theta}(x_{0},x_{T})=p_{T}(x_{T})\pi_{0}^{\theta}(x_{0}|x_{T})$, where conditional data distribution $\pi_{0}^{\theta}(x_{0}|x_{T})$ is parametrized by the student generator $G_{\theta}:\mathcal{Z}\times\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$ conditioned on a sample $x_{T}\sim p_{T}$. This approach is specifically used in Inverse Bridge Matching Distillation (IBMD) (Gushchin et al., 2024). Hence, our Universal Inverse Distillation objective can be written just by substituting student distribution $p_{0}^{\theta}$ by student coupling $\pi^{\theta}$, substituting real data $p_{0}^{*}$ by real data coupling $\pi^{*}$ and adding extra conditions.

###### Definition 5 .

We define Universal Matching loss with real data for general coupling on generated data coupling $\pi^{\theta}\in\mathcal{P}(\mathbb{R}^{D}\times\mathbb{R}^{D})$ with $\alpha,\beta\in(0,1]$:

$$ $\displaystyle\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f,\pi^{\theta})$ $\displaystyle=$ $\displaystyle\underset{\text{generated data }\pi^{\theta}\text{ term}}{\underbrace{\alpha\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x_{T}\sim p_{T},x^{\theta}_{0}\sim\pi_{0}^{\theta}(\cdot|x_{T}),x^{\theta}_{t}\sim p^{\theta}_{t}(\cdot|x^{\theta}_{0},x_{T})}\left[\|f_{t}(x^{\theta}_{t},x_{T})-\frac{\beta}{\alpha}f^{\theta}(x^{\theta}_{t}|x^{\theta}_{0},x_{T})\|^{2}\right]}}$ $\displaystyle+$ $\displaystyle\underset{\text{real data }\pi^{*}\text{ term}}{\underbrace{(1-\alpha)\cdot\mathbb{E}_{t\sim[0,T]}\mathbb{E}_{x_{T}\sim p_{T},x^{*}_{0}\sim\pi_{0}^{*}(\cdot|x_{T}),x^{*}_{t}\sim p^{*}_{t}(\cdot|x_{0},x_{T})}\left[\|f_{t}(x^{*}_{t},x_{T})-\frac{1-\beta}{1-\alpha}f^{*}_{t}(x^{*}_{t}|x_{0}^{*},x_{T})\|^{2}\right]}}.$ $$

And the corresponding Universal Inverse Distillation loss with real data for general coupling is:

$$ $\min_{\theta}\max_{f}\{\mathcal{L}^{\alpha,\beta}_{\text{R-UID-coup}}(f,\pi^{\theta}):=\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f^{*},\pi^{\theta})-\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f,\pi^{\theta})\}.$ $$

In case of coupling match$\pi^{\theta}=\pi^{*}$, the RealUID loss for couplings attains its minimum, .i.e.,

$$ $\min{{}_{\theta}}\{\underset{\geq 0}{\underbrace{\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f^{*},\pi^{\theta})-\min{{}_{f}}\{\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f,\pi^{\theta})\}}}\}=\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f^{*},\pi^{*})-\underset{=\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f^{*},\pi^{*})}{\underbrace{\min{{}_{f}}\{\mathcal{L}^{\alpha,\beta}_{\text{R-UM-coup}}(f,\pi^{*})\}}}=0.$ $$

###### Definition 5 .

## Appendix D Experimental details

Table: Table 6: Ablation of the fine-tuning for $\alpha_{\text{FT}}$ and $\beta_{\text{FT}}$ for unconditional (left) and conditional (right) generation. Each cell reports the resulting FID score for the corresponding $(\alpha_{\text{FT}},\beta_{\text{FT}})$; “–” indicates the method did not converge. Best results are bolded.

##### Training hyperparameters.

We train with AdamW (Loshchilov & Hutter, 2017), using $(\beta_{1},\beta_{2})=(0,0.999)$. The learning rate is $3\times 10^{-5}$ for training from scratch and $1\times 10^{-5}$ for fine-tuning. A 500-step linear warm-up is applied only when training from scratch. We use a batch size of 256 and maintain an EMA of the generator parameters with decay $0.999$. To regulate adaptation between the generator and the fake model, the generator is updated once for every $K=5$ updates of the fake model, following DMD2 (Yin et al., 2024a). Additionally, at each optimization step we apply $\ell_{2}$ gradient-norm clipping with threshold $1.0$ to both the generator and the fake model.

##### Training time.

All distillation experiments were trained for 400,000 gradient updates, corresponding to approximately 4.5 days. All finetuning experiments were conducted for 100,000 gradient updates, which took a little more than 1 day, starting from the best distillation checkpoints.
All experiments were executed on a single Ascend910B NPU with 65 GB of VRAM memory.
The reported results are based on the checkpoints that achieved the best Fréchet Inception Distance (FID) during training.

##### Codebase and Dataset.

Building on the reference codebase of Tong et al. (2023), which serves as our primary experimental infrastructure, we integrate the training algorithm described in Algorithm [1](https://arxiv.org/html/2509.22459v1#alg1). We evaluate the resulting approach on CIFAR-10 (32×32) under both conditional and unconditional settings, benchmarking against established baselines.

##### Models Initialization and Generator Parametrization.

The generator $G_{\theta}$ is initialized by replicating both the architecture and parameters of the teacher model $f^{*}$, while the fake model $f$ is initialized with random weights. We parameterize the generator using a residual formulation:

$$ $G_{\theta}(z)=z+g_{\theta}(0,z),$ $$

where the input $t=0$ corresponds to the fixed control input used in the teacher model $f^{*}$. Empirically, we observe that this initialization strategy and parameterization lead to improved performance.

##### GAN details

We integrate a GAN loss into our framework in line with SiD^2A and DMD2
(Zhou et al., 2024a; Yin et al., 2024a). In the original setup of Zhou et al. (2024a), the adversarial loss employs a coefficient ratio of $\lambda_{\text{adv}}^{D}/\lambda_{\text{adv}}^{G_{\theta}}=10^{2}$ (see Table 6 in Zhou et al. (2024a)), a choice that poses practical difficulties due to the extreme imbalance between generator and discriminator losses. To mitigate this issue, we adopt the formulation of Yin et al. (2024a), where the ratio is $\approx 3$, and evaluate different coefficient scales (see result in the Table [1](https://arxiv.org/html/2509.22459v1#S4.T1)).

##### Evaluation protocol.

We evaluate image quality using the Fréchet Inception Distance (FID; Heusel et al., 2017), computed from 50,000 generated samples following Karras et al. (2022; 2020; 2019). In line with SiD (Zhou et al., 2024b), we periodically compute FID during distillation and select the checkpoint achieving the minimum value. To ensure statistical reliability, we repeat the evaluation over 3 independent runs—rather than 10, as in SiD—because the empirical variance of FID in our experiments was below $0.01$.

## Appendix E Additional Results

### E.1 Fine-tuning ablation study of coefficients α FT , β FT \alpha_{\text{FT}},\beta_{\text{FT}} .

This section presents an ablation of the fine-tuning stage over the loss-balancing coefficients $\alpha_{\text{FT}}$ and $\beta_{\text{FT}}$. Results are summarized in Table [6](https://arxiv.org/html/2509.22459v1#A4.T6), where “–” denotes non-convergence. We observe that training is highly sensitive to the choice of $(\alpha_{\text{FT}},\beta_{\text{FT}})$: many configurations do not converge, underscoring the need for careful selection. Notably, the same set of $(\alpha_{\text{FT}},\beta_{\text{FT}})$ exhibit stable optimization and yield improved FID for both conditional and unconditional CIFAR-10 generation.

### E.2 Example of samples for different methods.

This section presents representative sample outputs from various studies conducted within the RealUID framework.

Figure: Figure 4: Uncurated samples for *unconditional* generation by the one-step RealUID ($\alpha=1.0,\beta=1.0$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/baseline_uncond.png

Figure: Figure 5: Uncurated samples for *unconditional* generation by the one-step RealUID ($\alpha=1.0,\beta=1.0$) + GAN ($\lambda_{\text{adv}}^{G_{\theta}}=0.3,\lambda_{\text{adv}}^{D}=1$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/gan_uncond.png

Figure: Figure 6: Uncurated samples for *unconditional* generation by the one-step RealUID ($\alpha=0.94,\beta=0.96$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/best_unconditional.png

Figure: Figure 7: Uncurated samples for *unconditional* generation by the one-step RealUID ($\alpha=0.94,\beta=0.96\mid\alpha_{\text{FT}}=0.94,\beta_{\text{FT}}=1.0$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/ft_uncond_best.png

Figure: Figure 8: Uncurated samples for *conditional* generation by the one-step RealUID ($\alpha=1.0,\beta=1.0$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/baseline_cond.png

Figure: Figure 9: Uncurated samples for *conditional* generation by the one-step RealUID ($\alpha=1.0,\beta=1.0$) + GAN ($\lambda_{\text{adv}}^{G_{\theta}}=0.3,\lambda_{\text{adv}}^{D}=1$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/gan_cond.png

Figure: Figure 10: Uncurated samples for *conditional* generation by the one-step RealUID ($\alpha=0.98,\beta=0.96$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/best_conditional.png

Figure: Figure 11: Uncurated samples for *conditional* generation by the one-step RealUID ($\alpha=0.98,\beta=0.96\mid\alpha_{\text{FT}}=0.94,\beta_{\text{FT}}=1.0$) trained on CIFAR-10. Quantitative results are reported in Table [2](https://arxiv.org/html/2509.22459v1#S4.T2).
Refer to caption: images/ft_cond_best.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. **On Scalability (W1)**: Can the authors provide even preliminary results on a single higher-resolution dataset (e.g., FFHQ-64 or CelebA-256)? How does RealUID converge, and are the $\alpha, \beta$ hyperparameters stable across different resolutions?
2. **On $K=5$ (W3)**: The experimental details state the fake model $f$ is updated 5 times for every 1 generator update ($K=5$). How critical is this ratio? Does the RealUID framework become unstable at $K=1$? This seems to hide a 5x computational cost for the fake model's training loop relative to the generator's.
3. **On $\mathcal{L}_{R-UM}^{\alpha,\beta}$ Coefficient Design (S2)**: The specific coefficients $\frac{\beta}{\alpha}$ and $\frac{1-\beta}{1-\alpha}$ in Eq. 17 are clearly deliberate. The paper claims this is to ensure $f=f^\*$ is the optimal solution when $p_0^\theta = p_0^*$.
Is this specific mathematical form the only one that satisfies this property? Would the framework fail if simpler, unscaled $L_2$ losses were used (e.g., setting $\beta/\alpha = 1$ and $(1-\beta)/(1-\alpha) = 1$)? This seems highly relevant given the hyperparameter sensitivity seen in (W3).
4. **On "Correcting" the Teacher (S2)**: A potential benefit of RealUID is that the fake model $f$ sees both the teacher $f^\*$ and the real data $p_0^\*$. If the teacher $f^\*$ is flawed (e.g., suffers from mode collapse) but $p_0^\*$ is complete, $f$ faces a conflict during its training: should it match $f^\*$ or the function implied by $p_0^\*$?  Can RealUID truly "correct" a flawed teacher, or would this conflict lead to training instability?
soundness: 3
strengths: 1. **Significant Methodological Innovation (GAN-free Supervision)**: The paper's strongest contribution is RealUID, a novel and elegant non-adversarial method to incorporate real-data supervision into the distillation process. This directly addresses a major pain point of previous work, avoiding the need for extra discriminator architectures and the instability of adversarial training.
2. **Clever Loss Function Design ($\mathcal{L}_{R-UM}^{\alpha,\beta}$)**: The mechanism of innovation is particularly commendable. Instead of simply adding a new top-level loss, the authors redesigned the inner-loop loss of the min-max game (Eq. 17). The "fake model" $f$ is forced to optimize on both $p_0^\theta$ (generated data) and $p_0^*$ (real data) during its $\max_f$ step. This allows $f$ to serve as a "bridge," passing the supervisory signal from the real data to the student generator $G_\theta$ in a natural, non-adversarial way.
3. **Strong Theoretical Unification (UID)**: Prior to RealUID, the paper's UID framework is a valuable theoretical contribution in its own right. It unifies seemingly separate works (SiD, FGM) under a single min-max perspective using a clear and general mathematical derivation (the linearization technique). It clarifies the underlying game structure: the "fake model" $f$ (the $\max_f$ player) tries to approximate the "theoretical student function" $f^\theta$, while the generator $G_\theta$ (the $\min_\theta$ player) tries to minimize the distance between $f^\theta$ and the teacher $f^*$.
4. **Excellent Empirical Efficiency**: The experimental results strongly support the practical utility of this new method. RealUID not only achieves strong FID scores but also demonstrates significantly faster convergence (approx. 3x) compared to the data-free baseline (Fig. 2), proving that this GAN-free supervision is both effective and efficient。
summary: This paper addresses a core problem in generative model distillation: existing data-free methods (e.g., SiD, FGM) cannot leverage real data to improve the student model, and the only current way to do so relies on complex and unstable GANs.The paper's contributions are twofold:
1. **Theoretical Unification (UID)**: It introduces a general min-max distillation framework (UID) and demonstrates that recent methods like SiD and FGM are special cases of this unified structure. 
2. **Methodological Innovation (RealUID)**: This is the paper's main contribution. The authors propose RealUID, a novel, non-adversarial (GAN-free) mechanism to integrate real-data supervision. This is achieved by cleverly redesigning the inner loss function of the min-max game ($\mathcal{L}_{R-UM}^{\alpha,\beta}$) to include terms from both the generated distribution ($p_0^\theta$) and the real-data distribution ($p_0^*$).

Experiments (on CIFAR-10) show that RealUID not only outperforms the data-free baseline in FID but also achieves comparable or superior results to GAN-based methods, all while significantly accelerating convergence and avoiding the complexities of adversarial training.
weaknesses: 1. **Severely Limited Experimental Scope (Main Flaw)**: All experiments are conducted exclusively on the CIFAR-10 (32x32) dataset. While resource constraints are mentioned, this leaves the scalability of RealUID to higher resolutions (e.g., 256x256 or 512x512) completely unknown. The need for distillation is most acute for these larger, slower models.
2. **Not State-of-the-Art (SOTA) Performance**： While the results are strong, the best-reported FID (e.g., 1.91 for conditional generation) still lags behind SOTA adversarial distillation methods (e.g., $SiD^2A$ @ 1.39). The authors hypothesize this is due to their lightweight architecture and teacher model, but this is unproven. It is unclear if the performance gap is due to the architecture or an inherent ceiling of the RealUID method itself compared to adversarial approaches.
3. **New Hyperparameter Sensitivity**: RealUID avoids GAN instability but introduces new hyperparameters $\alpha$ and $\beta$. The fine-tuning ablation (Table 6) shows high sensitivity to these, with many configurations failing to converge (marked with "-"). This may offset the claimed simplicity, as tuning $\alpha$ and $\beta$ could be as difficult as tuning a GAN.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: - Why does the setting too low values for the coefficients $\alpha,\beta$ lead to collapse? Is this just experimentally seen or is there a theoretical justification for it?
soundness: 4
strengths: - The premise of the paper is very interesting and important. Distillation methods that rely on Reverse-KL-like objectives (SiD, FGM, DMD, …) are widely used in practice but are known to suffer from a diversity collapse problem. This issue arises mainly because the loss is computed only on student samples (reverse-KL). Therefore, being able to effectively incorporate real data into training without resorting to GANs (which have famously unstable training dynamics) is potentially very valuable.
- The reasoning that leads to the RealUM and RealUID objectives is clever.
- The supplementary material is extensive and includes additional variants of the approach, as well as theorems and proofs.
summary: The paper tackles the challenge of distilling diffusion/flow-based models into fast one-step generators. The authors build on works such as SiD and FGM, which optimize the student by minimizing the $\ell_2$ distance between the teacher and student score/velocity functions, referred to as $f$-functions in the paper. The authors propose a simple but clever modification to the training objective of the student $f$-function that includes both real and generated data, allowing for an alternative way to incorporate real data into the distillation pipeline of these methods. They theoretically justify this change by proving that it has a proper global minimizer and experimentally demonstrate its usefulness on CIFAR-10, achieving impressive results.
weaknesses: - As mentioned above, loss of diversity is a common drawback of distillation approaches with Reverse-KL-like objectives. Incorporating real data into the training pipeline can, in principle, fix this issue to some extent. Therefore, including diversity metrics in the experiments would be very helpful and could strengthen the paper’s argument.
- The paper broadly argues that recent diffusion distillation methods either rely solely on synthetic data or use real data only in GAN-like objectives. While this is true for most Reverse-KL-like methods, it does not hold for consistency- or flowmap-based methods[1,2,3,4,5]. Including a discussion of such methods and adding them as additional baselines would help the paper better position itself within the literature.
- Although the authors explicitly note that the limited scope of the experiments is due to computational constraints, the fact remains that the proposed approach is primarily evaluated on a small dataset using a lightweight network architecture. As a result, it is unclear whether the conclusions drawn will generalize to larger datasets and models. For example, as shown in Table 4, the proposed approach performs considerably worse than prior SiD and SiDA baselines.
- The method seems highly sensitive to the values of $\alpha,\beta$, as shown in Table 1.


[1] Song, Yang, et al. "Consistency models." (2023). \
[2] Lu, Cheng, and Yang Song. "Simplifying, stabilizing and scaling continuous-time consistency models." arXiv preprint arXiv:2410.11081 (2024). \
[3] Frans, Kevin, et al. "One step diffusion via shortcut models." arXiv preprint arXiv:2410.12557 (2024). \
[4] Geng, Zhengyang, et al. "Mean flows for one-step generative modeling." arXiv preprint arXiv:2505.13447 (2025). \
[5] Sabour, Amirmojtaba, Sanja Fidler, and Karsten Kreis. "Align Your Flow: Scaling Continuous-Time Flow Map Distillation." arXiv preprint arXiv:2506.14603 (2025).

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1.Beyond mathematical re-formulation, what are the new insights and practical improvements uniquely enabled by your unified UID loss perspective that were not apparent or easily achievable within the prior frameworks.
2.Is there any adaptive criterion for tuning α and β? What is the rationale behind using different parameter values during the training versus fine-tuning stages? Do these hyperparameters generalize across datasets, or do they require costly re-tuning for each new dataset?
3.Why does RealUID use a lightweight model architecture different from the baseline methods? Could increasing the parameter count further improve performance?
4.Could the authors provide additional results of RealUID on other datasets and models? This is my primary concern.
soundness: 2
strengths: 1.The authors constructed a concise unified form through a linearization technique, and the theoretical derivation is fundamentally sound; 
2.The extension of the framework to bridge matching and stochastic interpolants is commendable; 
3.Replacing complex GANs with simple loss interpolation to incorporate real data is a straightforward yet effective approach.
summary: This paper introduces the RealUID framework, with two primary contributions: (1) it theoretically unifies prior distillation methods for different matching models (e.g., SiD, FGM) and extends them to Bridge Matching and Stochastic Interpolants via a novel UID loss based on a linearization technique; (2) it proposes a novel method for incorporating real data into the distillation process without relying on GANs. Experiments on CIFAR-10 provide empirical support for these theoretical claims.
weaknesses: 1.The proposed unified framework appears more as a formal re-formulation than a substantive theoretical breakthrough. This raises concerns that the core contribution may essentially be a "re-parameterization" of prior methods like SID and FGM, thereby undermining its claimed novelty and independence.
2.The proposed RealUID method suffers from a critical dependence on two newly introduced hyperparameters, α and β. Firstly, the selection of these parameters lacks clear theoretical guidance, and their design appears more as a heuristic afterthought rather than a principled derivation. Secondly, as evidenced in Table 1 and Appendix Table 6, the method exhibits extreme sensitivity and pronounced randomness to these parameters.
3.Lemma 2 relies on the assumption that the data probability density is positive everywhere (pt∗(xt) > 0, ∀xt, t). This assumption typically fails in high-dimensional spaces.
4.The most prominent weakness of this work is its narrow experimental setup:
1)The evaluation is narrowly based on small-scale datasets like CIFAR-10. The absence of results on standard large-scale, high-resolution benchmarks such as AFHQ and ImageNet severely limits the empirical grounding of the paper's claims.
2)The experimental evaluation is confined to  flow-based models, omitting tests on diffusion models. This limitation raises serious concerns about the method's competitive advantage and the framework's generalizability.
3)The experiments lack comparisons against state-of-the-art baselines from 2025.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please see weaknesses.
It is unclear why on line 294, the paper requires that "A key constraint is that the loss must still yield the same teacher upon minimization on the real data." With the inclusion of real data, shouldn't the teacher model be somehow different from the teacher model without using the real data? 
The goal of equation 13 is to maximize it, which is the last term of equation 10. Does maximizing equation 13 guarantee that equation 10, as a whole, will be minimized as equation 10 has two other terms that depend on f_t^star and f_t^theta?
soundness: 3
strengths: Stengths include the incorporation of real data into the distillation loss, thus avoiding the need for extra adversarial loss, which in turn, requires a discriminator network as in GAN. This is a logic and novel idea of implementing the distillation loss, as compared with existing approaches. Another strength is the reduced inference time, as shown in Table 3, second row from the bottom.
summary: This is a paper about generative modeling, in which the main contribution is the incorporation of real data into the loss function or the training process. The goal is to overcome the slow sampling process in diffusion models and flow matching models, which are collectively called match models in the paper. The proposed method incorporates real data into the distillation loss, thus, it removes the need to use GAN for assessing the adversarial loss. Experiments were performed on CIFAR-10 and showed that the proposed method did have faster inference.
weaknesses: It seems Equation 17 is the key formula of the work as it defines the new distillation loss, which weighs the loss or contribution from real data by hyperparameters alpha and beta. It is unclear why alpha and beta are, first, confined to a small range such as between 0.94 and 1.0, and, second, why alpha and beta are so close to each other in most cases. What does that mean for understanding the distillation loss? Does it mean that the contribution from the real data is typically given a small weight since it is multiplied by (1-alpha)? 
In the ablation study, Table 2, second row, when alpha = 1, beta=1 in this case, shouldn't the RealUID's performance equal that of the reference (the first row, Teacher Flow) since according to equation 17, when alpha=1, the second term is 0?


Decision: Accept (Oral)