Continuous Semi-Implicit Models
Authors: Longlin Yu
Jiajun Zha
Tong Yang
Tianyu Xie
Xiangyu Zhang
S.-H. Chan
Cheng Zhang
Venue: ICML 2025 poster (2025)
Keywords: Semi-Implicit Variational Inference
Score-Based Generative Models
Diffusion Models Distillation
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Semi-implicit distributions have shown great promise in variational inference and generative modeling.
    Hierarchical semi-implicit models, which stack multiple semi-implicit layers, enhance the expressiveness of semi-implicit distributions and can be used to accelerate diffusion models given pretrained score networks. 
    However, their sequential training often suffers from slow convergence.
    In this paper, we introduce CoSIM, a continuous semi-implicit model that extends hierarchical semi-implicit models into a continuous framework.
    By incorporating a continuous transition kernel, CoSIM enables efficient, simulation-free training.
    Furthermore, we show that CoSIM achieves consistency with a carefully designed transition kernel, offering a novel approach for multistep distillation of generative models at the distributional level.
    Extensive experiments on image generation demonstrate that CoSIM performs on par or better than existing diffusion model acceleration methods, achieving superior performance on FD-DINOv2.

Continuous Semi-Implicit Models
Longlin Yu"!
Jiajun Zha
Tong Yang 34
Tianyu Xie
Xiangyu Zhang* S.-H. Gary Chan 2 Cheng Zhang 15
Abstract
Semi-implicit distributions have shown great promise in variational inference and generative modeling. Hierarchical semi-implicit models, which stack multiple semi-implicit layers, en- hance the expressiveness of semi-implicit distri- butions and can be used to accelcrate diffusion models given pretrained score networks. How- ever, their sequential training often suffers from slow convergence. In this paper, we introduce CoSIM, continuous semi-implicit model that extends hierarchical semi- semi-implicit models into a continuous framework. By incorporating a contin- uous transition kernel, CoSIM enables efficient, simulation-free training. Furthermore, we show that CoSIM achieves consistency with care- fully designed transition kernel, offering a novel approach for multistep distillation of generative models at the distributional level. Extensive ex- periments on image generation demonstrate that CoSIM performs on par or better than existing diffusion model acceleration methods, achieving superior performance on FD-DINOv2.
Figure 1. Selected generated images on Imagenet 512 512 using I. model from Section 4.2.
imation accuracy ( (Yin & Zhou, 2018; Titsias & Ruiz, 2019; Moens al., 2021; Yu & Zhang, 2023; Cheng et al., 2024). Beyond variational inference, semi-implicit architectures have been successfully integrated into deep generative mod- els. including variational autoencoders (VAEs) (Kingma & Welling, 2014; 4; Rezende et al., 2014) and diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song al., 2021; 2020).
1. Introduction
Semi-implicit distributions, which are constructed through the convolution of explicit conditional distributions and im- plicit mixing distributions, have gained significant traction in variational inference and generative modeling. Unlike traditional approximating distributions with explicit density forms, semi-implicit distributions enable a more expressive family of variational posteriors, leading to improved approx-
In VAEs, the generator typically employs single-layer semi-implicit construction, pushing forward a simple noise through conditional factorized Gaussian distribution pa- rameterized by neural network. However, despite their widespread adoption, VAE-generated samples often suffer from blurriness and fail to capture high-frequency details in the data distribution (Dosovitskiy & Brox, 2016). To address these limitations, several VAE variants have been proposed (Rezende & Mohamed, 2016; Razavi et al., 2019; Vahdat & Kautz, 2021). ) A prominent approach is hierar- chical VAEs (Gregor et al., 2015; Ranganath et al., 2016; Sønderby et al., 2016; Kingma et al., 2017; Vahdat & Kautz, 2021). which enhances the expressiveness the single layer vallina VAEs through the introduction of multiple latent vari-
"Equal contribution. This work was done during internship StepFun. 'Shoool of Mathematical Sciences, Peking University, Beijing, China Deppatmment of Computer Science and Engineering, Hong Kong University of Science and Technology 'School of Computer Science. Fudan University, Shanghai, China "Megvii Technology Inc., Beijing, China Center for Statistical Science, Peking University, Beijing, China. Correspondence to: Cheng Zhang <chengzhange @ math.pku.cdu.ccm
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Continuous Semi-Impliee Models
ables, enabling multistep generation. Similarly, diffusion models have emerged as a leading framework for generat- ing high-quality and diverse samples (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021; 2020). These models can also be interpreted through the lens of semi-implicit distributions, where a fixed noise injection process is used to generate latent variables instead of learnable posterior distributions as in hierarchical VAEs.
Furthermore, CoSIM outperforms existing methods on the
FD-DINOV2 metrie (Stein al., 2024), which employs the larger DINOv2 encoder (Oquab et al., 2024) instead of the Inception 33 encoder (Szegedy et al., 2016) to better align with human perception.
2. Background
2.1. Diffusion Models
Although multistep generative models promising di- rection for improving sample quality, they often involve a trade-off between generation quality and computational cost. Drawing on the connection between diffusion models, hierarchical VAEs, and semi-implicit distributions, recent works have explored distilling diffusion models using tools from semi-implicit variational inference (SIVI) (Yin al., 2023; Luo et al., 2023; Yu et al., 2023; Zhou et al., 2024b;a). These methods can be broadly categorized into two types: (1) one-step deterministic distillation methods, which learn direct mapping to generate samples from the target distribu- tion (Yin et al., 2023; Luo et al., 2023; Zhou et al., 2024b;a), and (2) stochastic multi-step models, which generate more diverse samples in fewer steps. In particular, hierarchical semi-implicit variational inference (HSIVI) falls into the second category, recursively constructing the variational distribution over fixed sequence of time points (Yu et al., 2023). While HSIVI can generate more diverse samples than one-step models, its discretized design results in slow convergence during training due to the sequential simulation process.
Diffusion models (Sohl-Dickstein et al., 2015; Song & Er- mon, 2019; Ho et al., 2020; Song et al., 2021; 2020) perturb the clean data to noise in the forward process and then gen- erate the data from noise by multiple denoising steps in the backward process. The forward process can be described by a stochastic differential equation (SDE)
dæ. = f s)ds + g(s)d
(1)
where E [0. T 0 is a fixed terminating time, B, is a standard Brownian motion, and f(x;; and g(s) are the drift and diffusion coefficients respectively. The starting To ~ p(::0) is the data distribution. Denote the density law of the forward process as {p(:;s)) the SDE in (1) is designed a variance preserving (VP) or variance exploding (VE) scheme (Song al., 2020; Kar- ras et al., 2022), and the samples of p(ax am;; t, 0) can be reparameterized as
Typically,
=a(()) + o()), 1 LE [0
(2)
Drawing inspiration from continuous-time diffusion pro- cesses (Song et al., 2020), we introduce a Continuous Semi- Implicit Model (CoSIM), which extends hierarchical semi- implicit variational inference (HSIVI) into a continuous framework. By leveraging a continuous transition kernel, CoSIM generates samples of the mixing layer in single fush-forward operation, significantly accelerating the train- ing process. The design of the continuous transition kernel shares some similarities with consistency distillation (CD) (Song et al., 2023; Kim et al., 2023; Song & Dhariwal, 2024;
where e is Gaussian noise, a(t) is non-increasing function, and (t) is a monotonically increasing function. In the
backward process, one can run the following reversed-tin SDE from 7 to 0 to generate the samples of p(æ0; 0)
dx, {f(z;;s) g²( p(x.; s) + g())dB..
where V Vlogr(æ,; 8) is the score function of p(æ,i 8), B, is a standard Brownian motion. The score func- tion V logg((;;;s) is often estimated with a score model Sa(z,; = V Floop(xs: via denoising score match- ing (Vincent, 2011; Song et al., 2020).
Geng et al., 2024; Heek et al., 2024; Lu & Song, 2024), which learn a consistency function to map noisy distribu- tions back to clean target distributions. While Salimans et al. (2024) explore distilling multi-step diffusion models using moment-matching losses, CoSIM distinguishes itself by em- ploying semi-implicit variational inference (SIVI) training criteria. This approach enables learning the consistency function directly at the distributional level, bypassing the need to recover the reverse process of diffusion models at the sample or moment level. As a result, CoSIM signifi- cantly reduces the number of iterations required for distilla- tion. In experiments, we demonstrate that CoSIM achieves comparable results on the Fréchet Inception Distance (FID) (Heusel et al., 2017) while incurring lower training costs.
2.2. Semi-Implicit Models and Diffusion Distillation
Semi-implicit is a mixture distribution expressed as (x)) = f qa (=)2))(=)dz, which can be used to approxi- mate the target distribution via variational inference (Yin & Zhou, 2018; Titsias& & Ruiz, 2019). The conditional layer qo ()) is required to be explicit but the mixing layer q(z) can be implicit, where ø is the variational parameter. Recent works have explored diffusion model distillation within the semi-implicit framework ( (Wang et al., 2023b). These meth- ods are primarily distinguished by their training objectives: those utilizing density-basee divergences (e.g., JSD and KI.
2

---

Continuous Semi-Implicit Models
divergence) (Luo et al., 2023; Wang et al., 2023a; Yin al., 2023), and those employing score-based divergences (e.g., Fisher divergence) (Yu et al., 2023; Zhou et al., 2024b). Following SiD (Zhou et al., 2024b), which demonstrated superior performance and faster convergence in terms of FID results than Diff-Instruct (Luo et al., 2023), we adopt the score-based training objective in this work.
tives
min
(di(()) (x mid) (0(a)))) (V logs(æss:)
Va, log 9o (zx |z: ()] -allo 1113..
(6)
where a E R+ is a given hyperparameter used to balance the cross term and squared term in (6). The lower-level optimization problem for v( remains consistent with the maximization of (ST) Empirically, SiD with a > 2/2 demonstrates significantly better performance than both the baseline of a 1 and density-based distillation methods in one-step image generation tasks (Zhou et al., 2024b).
Hierarchical Semi-Implicit Variational Inference Yu et al. (2023) propose a hierarchical semi-implicit structure recursively defined from the top layer k = K- to the base layer l 0 for multistep diffusion distillation. This struc- ture employs the conditional layer (x [20411111 tx) and the variational prior q(xr: T), defined by
3. Continuous Semi-Implicit Models
(o(( = f qo (at f4 (a)s
Unlike one-step generation models, stochastic multi-step approaches such as HSIVI provide a systematic framework for progressive image quality enhancement while preserving sample diversity. However, HSIVI suffers from slow con- vergence in its training process. Inspired by the evolution of noise conditional score network models (NCSN) (Song & Ermon, 2019) to continuous time score-based generative models (Song et al., 2020), extending the sequential train- ing to continuous- time training framework promises to enhance the training efficiency. Following Yu al. (2023), extend the hierarchical semi-implicn model with fixed number of layers to a continuous framework. Our goal is to learn a transition kernel guans (2:, (z;; s, t) that maps the dis- tribution p( t) to x)) for 0 I 8 < t S T as a contin- uous generalization of the conditional layer q ( fx) in HSIVI. In the context of diffusion models, we assume p(: t) follows the density of the forward process of (1).
(3) tx = T and qs (::tk) := (): Moreover, the forward process (I) of diffusion models naturally provides sequence of intermediate bridge distributions {p(æ; t& )/= which can be combined with (3) for diffusion models distillation. In the training procedure, Yu et al. (2023) introduce joint training scheme to minimize a weighted sum of the semi-implicit variational inference (SIVI) objectives
where k- = 0, 00,,...,K 1, <
K-1
(HSIVI-
Σ C(k) (go(: (oo(s)))))((3))) (4) K=
where ((k) tO...., function and ff represents a distance criterion. In the applica- tion to diffusion model acceleration, / can be chosen as the Fisher divergence and the resulting HSIVI-SM optimizes go (2 f) via
T 1} - R4 is positive weighting
3.1. Continuous Transition Kernel
To construct the density of variational distribution at timestep 8, we denote the marginal distribution q(0s;; 8, follows
min max (o)) = Eggtor, (e) o(ar,, te)" (logp(mnnttt) v(-1a) Vay, log (0 (1 (at.l [6) (+0))))))))))
(5)
/ (ar.am(a ((0.)8,))=
8, fmix (2:1; f)dae,
(7)
where va (4) is an auxiliary vector-valued function and qo (Z )(z41) = qo ( i ( ix+1).
where Gmix (aix( I) is chosen as a role of the mixing distri- bution in semi-implicit variational distribution. Within the paradigm of hierarchical semi-implicit distribution (Yu et al., 2023), gmix ( i&)) is obtained by progressively sampling from the conditional layers {go(z,, ( :t) h2) as indi- cated in (3). Instead, we construct gmix (-:t) through single push-forward operation using the conditional distribution p|+|: 6, 0) defined (2)
Score Identity Distillation For the optimization of ø in (5), the coefficient & is not a unique choice. Zhou et al. (2024b) introduced Score Identity Distillation (SiD), a one- step distillation method for diffusion models. Assuming the variational semi-impll cemi-implicit distribution qo (z,, z)+)
p(a, Jarz/Gole 42)))) 0)ū(z), where G. is learnable neural net- work generator mapping from simple distribution q(z) ) to
gmix (xnxx t) = / p(zzz) wo; t, 0)p(2z; 0)
(8)
data distribution and the conditional layer p(-l-st.0) 0) follows
the definition in (2). SiD employs a fused loss function, which can be viewed as the training objective in SIVI objec-
which allows us to sample De ~ qmix (wy; f) instantancouslyy With the continuous timesteps and 8, we can train the tran-
3

---

Continuous Semi-Implicit Models
sition kernel Utrans (1r, la 8, 1) via the following continuous generalization of (4)
A Although this additional regularization introduces bias into the second optimization in (10), found that this bias in v does not transfer to é, and the Nash equilibrium of ø remains consistent with (11). We provide comprehensive statement in the theorem 3.1.
Lest-s(y) = So' ((s, CSSVE (a(#)g ss (1lp(( dids, (9)
where x(s, - (())) is a joint time schedule. A detailed discussion of the design principles for (t)) (t) is presented in Appendix A. Note that the marginal dis- tribution 5 t) in (7) depends on t. Thus models of the continuous transition kernel Grrans (ae,) (mz; 8, 1) will lead to consistency, as it convert 9mix (2;; t) to the same distri- bution p(z,; s) for all t E (s, T]. We call a continuous semi-implicit model (CoSIM).
Reformulation of Fused Loss In the fused training objec- tive (6) of SiD, there is mismatch in the two-stages training objective (10) when a is not 2. Furthermore, when a > 1, the fused loss function exhibits pathological behavior as it becomes negative when the inner optimization of v(:tt) converges to its optimal solution
( = (a - a)ll logp(z.;s) V log (2(())))))) t)
3.2. The Training Objectives
However, within the framework of shifting fu. we can reformulate the SiD training objective to achieve unbi- asedness while eliminating the aforementioned patholog- ical behavior. Consider the scaled Fisher divergence Da (pl: (00(ss))l999 (+) defined as
Let ( 8, t) be the corresponding variational distribu- tion for the parameterized transition kernel (0(22. s, t). We adopt the score matching objective LSIVI-SM introduced in Section 2.2 for training. More specifically, we parameter- ize the auxiliary vector-valued function as (2, s, t) == V log p(æ,;; 8) - L (Zzi 8, () and reformulate the optimiza- tion into two stages
Da Da 8, (.)((((ss))) = Beslesst,)) 4a (0( = V log 8) V Voo go (x.; (13)
(0 1)
min (o, us (m 8, t)a (V log p(ms;))
Va, log (2, s,;s,
2 ( t)
1
(10)
Then we rewrite the above Da, (p(:: (gg((; t)) the maximum value of the optimization problem
min E (a mm m ff
log go | 112.
max V2
E (00( (p t) [V log
where co(2,, Dz t) Qtrans (trans(2s (255 e 8, gmix (wt; t).
Vog g0 (o,)x,s - a w s, 1)]
(14)
Shifting fo by Regularization The Nash-equilibrium of the two-stage optimization problem (10) is given by
Similarly, we utilize (14) to reformulate the (13).
Theorem 3.1. Optimization of Da (p(:; s) ()los((sss
is
f st =V log ao s,),
equivalent to the two-stages optimization
o² E arrgmnn ( )) 08 (xsi s, 1)2)
(II)
min E (g [V 8, f)7 [V pog.(
(z( s) S (z V Vog g0 (xgis,t)
Va, (0g
u ()
(15)
A natural initialization, therefore, would be use the pre- trained score network fo. st) s So- (z) Dur- ing training, the optimal f is given by (o) (x,, V log (o(#;; s, t). When V log (0x s, t) deviates signifi- cantly from the target score model Sg- (æ.;;) 8), this initial- ization strategy becomes inefficient for the second-stage optimization in (10). To address this mismatch, we follow Salimans et al. (2024) and adopt regularization strategy
min Ems(a. (w.i f)a V log (o (o s
s)
V Vlogp(asss)) s)] + (1 + a) t)]
where X 2 0 is a given regularization strength hyperpa- rameter. The optimal (1) of the second-sted stage
optimization is given by
fsr (e)((((s 3V (oggo(ss,t))) (o Vogg 8), (16)
min E Eggte, V
[lI f s, t) - log qo (x., (2, s, 1)
where 3 = 2a(1+x)- The equilibrium of 6 remains consis- tent with (11).
1
fallf((zzsss So. (ax;: s))
(12)
where x 2 0 a hyperparameter controlling the strength of regularization. It can shown that the optimal f of (12) is shifted towards Sg- as follows
It is straightforward to observe that (15) is the same as the SiD loss in (6). If V logp(zz;s) is approximated by So- (z:;; 8), the optimal f is given by
1
A
(xo; = 1+
V (
So (2))))))
for (38,t) = BV log f) + +11 V)ggop(())). (17)
1

---

Continuous Semi-Implicit Models
(17) indicates that when a optimal (Z2;; 1) in the second-stage optimization also shifts towards Sp- (ars:)). 8). This alignment facilitates learning when f is initialized as Sg.. The proof of Theo- rem 3.1 and its general version are shown in Appendix B.1. The training process of CoSIM is summarized in Algorithm 2 in Appendix A.
バパス then 8 < 1. The
Algorithm 1 Inference of Continuous Semi-Implicit Models
Input: Continuous transition kernel Trans (2's (2, t)
and a sequence of time points = to > t1 >
4. = 0.
Output: The estimated samples of p(æ0)).
Initialize To ~ p(ær:
for = 1 to k do
Sample In, ~ Quanss (- tu 1n-
3.3. Parameterization of CoSIM
end for Output: æk
Consistency model (Song et al., 2023) is a family of gen- erative models that learn a consistency function Go (ar,1)) to map the samples of distribution p(æ;;; f) back to p( 0). Once the consistency function is well-trained, the consis- tency model can generate samples using multiple steps by iterating with < f as follows
where [Cst, is defined in (9) with the choice of criterion
Da (p(:: s) (: s)) discussed in section 3.2.
We first introduce the following assumptions with a fixed À > 0 and an early-stopping time 0 < 8 <.
vz = (aass()Goll () + (s)e,
(18)
As illustrated in Algorithm 2, the approximation errors arise from the inexact second-stage optimization of f We de- note it as follows.
which approximates zy in (2) by t), and NNO,T), I). Intuitively, this provides a parameterized form of the continuous transition kernel Otrans (x: (2r/ 8, t). We hereafter adopt this setting for diffusion model distillation. One can expect that once (a t) is perfectly trained with f (0)) then Go t) will also map the samples of dis- tribution p( a 1) back to p( (o 0). We present this result in Proposition 3.2. See detailed proof of Proposition 3.2 in Appendix B.2.
Assumption 3.3 (fe function error). The estimated auxil- iary vector-valued function fa (aa;; t) in the two-stage optimization problem (15) is eraccuratee, that is for all 8, E [6, 7] with s < and & E $:
(a 1oll (ao (o) of. (20)
Proposition 3.2 (Consistent Similarity). Let the continuous transition kernel Grrmss ( (2); s, t) be parameterized as
Let the g. (, be the obtained by the first stage optimization problem with the er-accuraee L
F, = [o((s) f) + (s)ee ~ 40(x0 8, s,t), where 0 < < < e~ NoO, I). and a(-).o(() are defined in (2). Then the optimal (-((,t) obtained from the two-stage alternating optimization problem (15) also serves as a consistency function.
9 8, t)e arg (no U s.) (So (So
-va, log (x. (x(is t)] a (rios) (x t)
With a well-trained continuous transition kernel Orans (2r, 8, l), we can iteratively sample from it over multiple steps to ultimately generate samples from p(xo)) To select the sequence of time points, we use the sampling scheme of EDM (Karras al., 2022) as an initial choice and set scale parameter for tuning with greedy algorithm. We summarize the multistep sampling procedure in Algorithm 1.
where (5(0)) (I.: s, == log p(x,;s) I (o) (a., Tz; s,t.
Then we are ready to state the following result.
Proposition 3.4 (Error bound of one-step map. Yu & Zhang (2023)). Suppose the assumptions [3.3] hold. Then for any 0 < 8 < 1 < T, the Fisher divergence between the target distribution p(æs) and the estimated distribution of CoSIM is bounded as follows
Error Bound of CoSIM Building on the parameter- ized setting (18) of quans (xa [215 8, 1), we denote the vari- { qo (x,,ss)
Fligg (3 & (100) + o7.
ational distribution family Q
The above results show that the approximation error of inex- act f only attribute extra term Ef to the approximation accuracy of CoSIM. As long as the approximation error Er is small, the two stages optimization of CoSIM with regular- ization strength À > 0 will provide the same approximation accuracy as the optimization with the original training ob- jective (13).
my; 8, 1m (a.) )aa,, @ E } where is the fea- sible domain of neural network parameters of Go and quix follows (8). Then we can define the optimal variational
I (ax))
distribution in Q as
Jo- E arg min LEst, (qo);
(19)
5

---

Continuous Semi-Implicit Models
3.4. Insights the Benefits of Multi-Step Methods
Song al., 2023), we employ the consistency function as C () as follows
As discussed in section 3.2, we adopt the same type of SIVI objective ( y as in SiD (Zhou et al., 2024b) for diffusion model distillation. However, unlike the deter- ministic one-step generation scheme of SiD, continuous semi-implicit models (CoSIM) allow us to generate samples from p( through multiple steps. A notable advantage of CoSIM is that it injects randomness at each generation step, producing more diverse samples an observation consistent with the benefits of stochastic diffusion models (Song al., 2020). Moreover, we show next, CoSIM can generate samples closer to the target distribution than the one-step distillation model by employing multistep sampling.
Go( t) (t)ip(t + Cour (a t), (22)
where Cskip modulates the skip connection and Cin- Cout are the scaling factors of network input and output.
Assumption 3.7 (Smoothness of neural network). The neu- ral network Fa I) in (22) is L--Liscchizz continuous on the variable with constant L > 1 for all E [].
The Lipschitz continuous assumption is naturally used in previous works (Song et al., 2023; Lyu et al., 2024; Li et al., 2024). However, we adopt practical preconditioning and impose the Lipschitz condition on F& rather than Go- Moreover, these two Lipschitz conditions are equivalent under the variance preserving (VP) scheme, which shown in Appendix B.4.
Since the optimal variational distribution gø- (ægis, s, t) is de- fined on Q, this family itself will introduce approximation gaps due to the approximation capacity of (z t). Since (22,, t) maps the distribution p(ar;; t) back to p( (o; 0), the approximation error is expected to grow with t. Follow- ing the assumption in previous work Chen et al. (2023), we scale the error term of (æ t) accordingly.
Based on the above error bound in proposition 3.4, we can give convergence bound for the multistep sampling of CoSIM similar to Lyu et al. (2024).
Assumption 3.5 (Consistency function error). For all e [6, Ti. the approximation gaps between V log qo- (Is, 6, and V log p(a:;; 5) is bounded in (.
Proposition 3.8 (Multistep Wasserstein Distance Error). Under Assumptions [3.3-377,, consider a sequence of time points T to 2 4 = =... = lK-1- = lmmid >> l O. Let (.) denote the K-step estimated distribution by CoSIM. Then for variance preserving scheme of (1), there exists mid Ollo Ly)) for variance preserving scheme of (1) such that
Bag- (8 V log 8) - V log qo (xss 5, f)1l S Eg(t)),
à
where the error term is scaled by (t) == 2: Flp((( 5) [lp(::)) 6)), and error &, characterizes the approximation capacity of the consistency function family {Gol E $). FI(-II-) denotes the Fisher divergence.
(K) W2(0 .P(:0) S &dd+ ( + E (tmdt
We note that the divergence Fl(p(:st t + ()])p(,))) grows as }) con-
where Ewy (t) denotes the Wasserstein distance bound be- tween the one-step CoSIM estimate (=(:;;;0 and pl:; 5)) This bound is controlled by the error terms from Assump- tions 13.3-3.7|, with the explicit formulation detailed in Appendix B.4. For variance exploding scheme of (1), the above results hold for some midd = (L)))
t increases to T. Furthermore, when {G tains only the identity mapping, the aforementioned upper bound holds with e? = 1, validating the reasonableness of assumption 3.5. More details can be found in the proof of Proposition 3.4 in Appendix B.2.
Assumption 3.6. The perturbed distribution p(æ;;; 5) satis- fies logarithmic Sobolev inequality (LSI) as follows.
Intuitively, since the Wasserstein distance bound (tg(t) is increasing with 1, the benefit of multistep sampling lies in reducing the error bound from E (T) of the one-step model smaller one (Wgg(mmaa) We provide detailed proof of Proposition 3.8 in Appendix B.4.
ent(f" (f2) < 2L
vff E C (RR",
(21)
where ent( e2t( = Fp (log f2 E. o) 22) is the entropy of f², r(f) = f/ and LISI is the LSI constant.
The LSI assumption is a standard assumption in the anal- ysis of diffusion models (Lee et al., 2023), and previous works show that the LSI inequality holds when p(zu; 0) is a bounded distribution (Chen et al., 2021). More details can be found in the proof of proposition 3.4.
4. Experiments
In this section, we explore the performance of CoSIM on the unconditional image generation task on CIFAR-10 (32 x 32) and the conditional image generation task on ImageNet (64 x 64) and ImageNet (512 x 512). For all experiments, the network architecture of the transition kernel go (-|,,;; 8, t) and the auxiliary function u (xg; s, t) is almost identical to the pre-trained score network Sg. (Ix;; 8) (Karras et al.,
Then we investigate the smoothness assumption of the con- sistency function (2 t). Following the preconditioning strategy of neural network proposed in (Karras et al., 2022;
6

---

Continuous Semi-Implaee Models
2022), except for an additional time embedding network of t. For the purpose of minimal changes, this is done by duplicating the time embedding network of in the score network urchitecture and summing up the time embedding of & and together, leading to only a 4% increase in pa- rameters. The implementation of CoSIM is available at https: : /t//ttttubbb cwwm wwwginnunnooooooooooooooooo (uw.nggiin//CCMM.....
Table I. Unconditional generation quality on CIFAR-10 (32 32). Results with asterisks (*) are tested by ourselves with the official codes and checkpoints. ECT does not provide official checkpoints, so its FD-DINOv2 score (") comes from our re-implementation. The other results are from the original papers. The best result marked in black bold font and the second best result is marked in brown bold font.
We use the well-established metric Frechet Inception Dis- tance (Salimans et al., 2016) (FID) to measure the quality of the generated images. Furthermore, as the FID score often unfairly favors the models trained with GAN losses and penalizes the diffusion models, we consider an additional metric- FD-DINOv2 (Stein et al., 2024), which replaces the Inception V3 (Szegedy et al., 2016) encoder of FID by DINOv2 (Oquab al., 2024) to better align with human per- ception. Both FID and FD-DINOv2 are evaluated on 50K generated images and the whole training set, which means 50K images from CIFARIO (Krizhevsky & Hinton, 2009) training split and 1, 1,281,167 images from ImageNet (Deng et al., 2009).
Method
PParams NFE
FI
FD-DINOV2
CIPAR: Spiis
3:15
31.07
DDPM (Ho et al., 2020)
1000
3.17
DDIM (Song et al., 2021)
100
4.16
TDFM- (2heng al., 2023a)
2.85
Solver (La el 2022)
18
2.65
VP EDM (Karras 2022)
56M
1.977
168.0L
VE-EDM-LEGG-PPE (Zheng et 2023h)
1.85
PFGM+ (Ku et al., 2023)
35
1.91
141.65
HSSII-SM (Ya all., 2023)
15
4.17
CTSL (Srin al., 2023)
3.55
2.95
ICT (Song & Dhariwal, 2024)
2.83
2,46
SCT (Lu & Song. 2024)
2.97
2,06
CTM (Kim 2013))
324M
1.98
08*
CTM (Kim al., 2023)
531M
1.87
210.64
CTM (Kim al., 2023)
324M
1.84
197.30"
BCT (Geng et al., 2024)
56M
2.11
211:171
SiD (Zhou et al., 2024bb
56M
1.92
148.17"
The training setup for CoSIM is basically kept the same as the underlying pre-trained score models, only with a different optimization objective in Section 3.2. In practice, we adopt the setting a = 1.2 as in Zhou et al. (2024b). In contrast to the SiD loss, which uses a(1 + A) = 0.5, we choose to increase the regularization strength À such that a(1 + A) = for simplicity. The time schedule is detailed in Appendix 1. We re-use most of the optimizer settings from the pretrained score models with some slight tweakings such as learning rate and batch size, more details can be found in Appendix C.
CoSIM
56M
2,40
116.02
CoSIM (ours)
56M
1.97
113.51
CoSEM)ours)
5GM
1.96
111.98
FD-DINOV2. Finally, we observe consistent decrease in both FID and FD-DINOV2 as the NFE increases from 2 to 4, which accords with our theoretical analysis in Section 3.4.
4.2. Conditional Image Generation
ImageNet (64 x 64) Similarly to Section 4,1, the network structure and the checkpoint of the pre-trained score model Sg- (I. 8) are from Karras et al. (2022).
4.1. Unconditional Image Generation
We report the conditional generation quality of different models in Table 3. We see that the VP-EDM (the teacher model) can get a worse FID with 999 NFE, while the FD- DINOv2 decreases monotonically as NFE increases. This consistency in FD-DINOV2 further verifies the reasonability of introducing FD-DINOv2 for measuring the generation quality of diffusion- based models. After distillation, our CoSIM produces an FID of 1.46 with 4 NFE, reaching a similar performance to VP-EDM with 999 NFE and outper- forming most methods in Table 3. Our CoSIM also achieves state-of of-the--rttttt- results on ED-DINOv2, setting the lowest record at 56.66 with only 4 NFE. We also notice that Mo- ment Matching (Salimans et al., 2024) reaches a better FID score compared to our CoSIM, which we attribute to the smaller network architecture employed by us.
CIFAR-10 (32 32) On the unconditional image gen- eration task CIFAR-10, the pre-trained score network Sg- (xx;;)) s) is from Karras et al. (2022).
Table 1 reports the generation quality measured by FID and FD-DINOv2. We see that our CoSIM reaches the same FID as the teacher (VP-EDM) with only NFE. Although the 2-step CoSIM reaches FID above 2, this is still con- siderably lower than many of the leading methods. The reasons for CoSIM's Is lagging behind other methods can be attributed to our model size, which is significantly smaller than others. On the FD-DINOV2, our CoSIM achieves state- of-the-art results compared to both seiindd-roo-scrattee and distilled models. Specifially, CoSIM attains an FD-DINOv2 of 113.51 with 4 NFE, outperforming the best-ever PFGM++ (Xu et al., 2023, FD-DINOv2 of 141.65) by a large mar- gin. We also note that this leading status of CoSIM still holds with 2 NFE. Furthermore, the FID metric's 's inclination towards GAN-based models (Stein et al., 2024) may partly explain why CTM achieves the best FID score, rather than
ImageNet (512 x 512) FD-DINOv2 is a new metric pro- posed very recently (Stein et al., 2024). Most of the models published before FD-DINOv2 naturally did not consider metric their experiments and thus may not set up their
7

---

Continuous Semi-Implicit Models
Figure 2. Conditionally generated images ImageNet (512 512). The two batches of images are generated from 4-step (left) and 2-step (right) CoSIM model respectively, with identical initial noise and class labels.
Table Conditional generation quality measured by FD-DINOv2 on ImageNet (512 512). Results of SiD are reported from Zhou et al. (2025).
Table 3. Conditional generation quality on ImageNet (64 x 64). Results with asterisks (") are tested by ourselves with the official codes and checkpoints. The other results are from the original papers. The best result marked black bold font and the second best result is marked in brown bold font.
Method
#Params
NPE
FD-DINOV2 (L)
EDM2 S. DINOv2 (Karra et al., 2024)
280M
63
68.64
EDM2-M-DINOv2 (Karas al. 2024)
498M
#Params
(.)
FD (IN
EDM2-L-DINOR (Karras al., 2024)
778M
52.25
DDPM (Ho al., 2000)
250
11.00
EDM2-XL-DINOv2 (Karra et al., 2024)
1.1B
63
15.96
TDPM+ (Zheng al., 2023g)
1000
1.62
EDM2-XXI DDN.---DOOO2.. (Karras et al., 2024)
1.5B
DPM-Soler3 (L. et 2022)
50
17.52
VP-EDM (Kamas el al. 2022)
290M
70
2.64
107 07
SiD-S (Zhou et al., 2024b)
280M
293M
611
1.36
70.82"
SiD M. (Zhou el ul., 2024bb
498M
55.92
296M
889
1.41"
72.67
SID-L (Zhou et al., 2024b)
177M
56.25
VP-EDM-LEGO-AE (Zheng et al., 20236)
250
2.16
CoSIM-S (ours)
28UM
HSIVL-SM (Yu al., 2023)
15.48
28IM
67.71
CDPP (Song al., 2023)
1
6.20
CoSIM-M (ours)
198M
53.35
2
4.70
198M
iCT (Sing Dhuriwal, 2024)
CoSIM-L (ours)
778M
46.41
sCT (Lu Song. 2024)
2
1
3,20
2.04
778M
41.79
CTM (Kim al., 2023)
80M
1.92
163-47
604M
2
1,73
150.00
Moment Matching (Salimons al., 2034)
200M
3.0
200M
2
3.86
models perfectly for getting a good FD-DINOv2 result. To make more convincing comparison, we further test CoSIM on ImageNet (512x512) against EDM2 (Karras et al., 2024) which also incorporates FD-DINOv2 benchmark in their original paper. This experiment also validates the scalability of our approach.
200M
1.50
200M
1.24
BCT (Geng et al., 2024)
293M
1.67
150
Diff-Instract (L.s 2023)
295M
SID (Zhou et al., 2024b)
296M
1.52
79.15"
CoSIM (ours) CaSIM (ours)
293M
2
3.35
108.99
1.48
58.66
We test our method on three different model sizes (S, M, L) from Karras al. (2024), which are significantly larger than those models tested in CIFAR 32 x 32 and Imagenet 64 64. Also, with steady increase in parameter numbers from S to M and L, we showcase the scalability of our approach by generating superior results on larger models.
We also provide the conditionally generated images from CoSIM in Figure We ensure that the 4-step generation and 2-step generation start from the same initial noise and class labels, and show that there is clearly observable difference. Specifically, 2-step samples are good enough to capture the majority of contents, but 4-step samples tend to capture more fine-grained details, e.g., the human eyes, cricket legs, and snake bodies shown in this figure.
Tables [2-4] report the conditional generation quality mea- sured by FD-DINOv2 and FID of different models. As the model size grows, the generation quality of both models improves consistently. For all the model sizes (S,,.L,, our CoSIM with only 2 NEE surpasses the teacher with 63 NFE, demonstrating the effective distillation ability of CoSIM for handling nigh-dimensional samples and larger models. Due to the GPU memory limit, we do not test CoSIM on XL and XXL models but expect its similarly superior performance, since CoSIM-L with 4 NFE already beats 1-M2-XXI-ddino.
5. Conclusion
We presented CoSIM, continuous-time semi-implicit framework for accelerating diffusion models through stochastic multi-step generation. Unlike the discrete-tin sequential training in hicrarchical semi-immlice variational inference, CoSIM leverages a continuous-tin framework
8

---

Continuous Semi-Im Semi-Implicit Models
specifically highlighted here.
Table Conditional generation quality measured by FID on Ima-
geNet (512 512).
A Acknowledgements
Method
#Paramss
NFE
FID (U)
EDM2-S-FID (Karras ul., 2024)
280M
63
2.56
This work was supported by National Natural Science Foun- dation of China (grant no. 12201014, grant no. 12292980 and grant no. 12292983). The research of Cheng Zhang was support in part by National Engineering Laboratory for Big Data Analysis and Applications, the Key Laboratory of Mathematics and Its Applications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQF) of Peking University. The research of S.-H. Gary Chan was supported, in part, by Research Grants Council Collaborative Research Fund (under grant number C1045-23G). The authors appreciate the anonymous ICML reviewers for their constructive feedback.
EDM2-M-FID (Karras al., 2024)
496M
63
2.25
EDM2-L-FID (Karras et al., 2024)
778M
63
2.06
EDM2-XL-FID (Karras al., 2024)
1.IB
63
1.96
EDM2-XXL-FID (Karras ct ul., 2024)
1.5B
63
1.91
sCD-S & Song. 2024)
280M
2
2.50
sCD-M (Lu& & Song, 2024)
49SM
xCD-L. (I.u & Song, 2024)
77SM
2
2.04
SID-S (Zhou et al., 2024b)
280M
1
2.71
SID-M (Zhou et al, 2024b)
498M
I
2.06
SiD-L (Zhou et al.. 2024b)
778M
1
1.91
CoSIM-S (ours)
280M
2
2.66
280M
4
2.56
CoSIM-M (ours)
498M
2
1.95
498M
4
1.93
CoSIM-L (ours)
778M
2
1.84
References
778M
1.83
Bardet, J.-B., Gozlan, N., Malrieu, F., and Zitt, P.-A. Func- tional inequalities for gaussian convolutions of compactly supported measures: explicit bounds and dimension de- pendence. Bernoulli, 24(1):333 2018.
to approximate continuous transition kernels. By introduc- ing a framework of equilibrium point shifting, we establish theoretical guarantees for unbiased two-stage optimization while resolving the pathological behavior inherent in SiD training objectives. Through careful parameterization of continuous transition kernels, we provide a novel method for multistep distillation of generative models training on a distributional level. In experiments, we demonstrate that CoSIM achieves comparable or superior FID while only requiring fewer training iterations and utilizing similar smaller neural networks compared to existing one-step dis- tillation methods and consistency model variants. Further- more, CoSIM achieves state-of-the-e performance on both unconditional and conditional image generation tasks as measured by FD-DINOv2.
Cattiaux, and Guillin, A. Functional inequalities for per- turbed measures with applications log-concave mea- sures and to some bayesian problems. Bernoulli, 28(4): 2022.
Chen, H., Lee, H., and Lu, J. Improved analysis of score- based generative modeling: User-friendly bounds under minimal smoothness assumptions, 2023. URL https: //arxiv. org/ abs 221/22211111 01916.
Chen, H.-B., Chewi, S., and Niles- Niles-Weed, Dimension freee log-sobolev inequalities for mixture distributions, 2021, URL https:tp///// http://axxvvvvv. org/ 1116.
Limitations For diffusion distillation, CoSIM training involves three models: the generator Go, the auxiliary func- tion fg and the target score model Sg.. As a result, CoSIM incurs high memory consumption due the involvement of multiple models. Additionally, since CoSIM initializes the generator G and the function fo from pre-trained target score model, it limits the flexibility of G to leverage larger architectures for modeling the continuous transition kernels. Consequently, the one-step generation quality CoSIM is lower than that of modern one-step distillation models such as SiD and sCD, defer this aspect to future research.
Cheng, Z., Zhang, S., Yu, L., and Zhang, C. Particle-based variational inference with generalized wasserstein gradi- ent flow. In Thiirty-eventth Conference on Neural Infor- mation Processing Systems, 2023.
Cheng, Z., Yu, L., Xic, T., Zhang, S., and Zhang, C. Ker- nel semi-implicit variational inference. In Forty-First International Conference on Machine Learning, 2024.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 245-255. leee, 2009.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be
Dosovitskiy, A. and Brox, T. Generating images with per- ceptual similarity metrics based on deep networks, 2016. URL https: //arxiv..vvvv///////////rre wwwrriii.gg///eee. wwwp.ors.///..............o... 602. 02644.

---

Continuous Semi-Implics Models
Geng, Z., Pokle, A., Luo, W. Lin, J., and Kolter, Z. Consistency models made easy, 2024. URL https: //arxiv.org/ abs 14548.
Lee, H., Lu, J., and Tan, Y. Convergence for score-based generative modeling with polynomial complexity, 2023. URL http::/: htt//aiivv..rro/ org /ass///////aa aba//b//.... 06227.
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., and Wierstra, D. Draw: A recurrent neural network for image generation, 2015. URL https:: //tp://aaavv..... org/ abs/ 1502. 04623.
Li, G., Huang, Z., and Wei, Y. Towards a mathematical theory for consistency training in diffusion models. ArXiv, abs/2402.07802, 2024.
Lu, C. and Song, Y. Simplifying, stabilizing and scaling continuous-time consistency models, 2024. URL https: / wwwpxw..rg//bs///11....
Gronwall, T. H. Note on the derivatives with respect to parameter of the solutions of system of differential equations. Annals of Mathematics, 20:292, 1919.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. DPM-solver: A fast ODE solver for diffusion probabilis- tic model sampling in around 10 steps. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
Heek, J., Hoogeboom, E., and Salimans, T. Multistep con- sistency models, 2024. URL /tttp::://axx..../////
abs/2403. 06807.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by two time-scale update rule converge to local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637 2017.
Luo, W., Hu, T. Zhang, S., Sun, J., Li, 7.,, and Zhang, Z. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: www.reeiew.nnee....eeeeeeeeeeeee forum?dd-MLIsSiRRo
Ho, J., Jain, A., and Abbeel, P. Denoising diffusson proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
Lyu, J., Chen, Z., and Feng, S. Sampling is as easy as keeping the consistency: convergence guarantee for consistency models. In Forty-firs International Con- ference on Machine Learning, 2024. URL https: / /openreview.. net forum? wtw.mmiddddiiiiippoooo
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidat- ing the design space of diffusion- generative mod- els. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https:t/t//c /ttp///ppeeeeveeeeeeeeeeeeeeeeee www.wwewwee... net./ forum?
Moens, V., Ren, H., Maraval, A., Tutunov, R., Wang. J., and Ammar, H. Efficient semi-implicit variational inference. arXiv preprint 10X.x:2111.06770 2021.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14174-24184, 2024.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec. M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y,, Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Je- gou, H., Mairal, J., Labatut, P., Joulin, A., and Bo- janowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL ht https: openreview net / forum idd 6zFt. Fea- tured Certification.
Kim, D., Lai, C.-H., Liao, W.H.., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consis- tency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprini arXiv:2310.02279 2023.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In ICLR, 2014.
Otto, F. and Villani, C. Generalization of an inequality by ta- lagrand and links with the logarithmic sobolev inequality. Journal of Functional Analysis, 173:361-400, 2000.
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improving variational inference with inverse autoregressive flow, 2017. URL https: arxiv. org/ abs/ 1bss11 66666.60 04934.
Ranganath, R., Tran, D., and Blei, D. M. Hierarchical vari- ational models, 2016. URL https://arxiv/: abs/1511. 0238661..0288.
Krizhevsky, A. and Hinton, G. Learning multiple lay- ers of features from tiny images. https: / /www.w cs. toronto. eeconto.edu/kkeee -ratl///rii/ee far. html, 2009.
Razavi, A., van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vae-ae 2019. URL https: http//rxvvv.rrrrre// 606. 00446.
10

---

Continuous Semi-Implicit Models
Rezende, D. J. and Mohamed, S. Variational inference with normalizing flows, 2016. UtL.tttps:/aaxix
Sønderby, C. K., Raiko, T., Maaløe, L., Sønderby, S. K., and Winther, O. Ladder variational autoencoders, 2016. URL httphttt/////m http://axivvvoooo parg/ati /abs///////a 0tt.//1122.2222.8
/ 1105. 05770.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models, 2014. URL https::/p///t...... atxii.iii... org/ abs/1401. 4082.
Titsias, M. K. and Ruiz, F. J. R. Unbiased implicit varia- tional inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 167-176. PMLR, 2019.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Vahdat, A. and Kautz, J. Nvae: A deep hierarchical vari- ational autoencoder, 2021. URL https://arxiv. org abs www.aba0000 http//20070..9... 03898.
Vincent, P. connection between score matching and de- noising autoencoders. Neural Computation, 23(7):1661- 1674, 201 1. doi: 10. 1062/ECO a 00142.
Salimans, T., Mensink, T., Heek, J., and Hoogeboom, E. Multistep distillation of diffusion models via mo- ment matching. In The Thirty-eighth Annual Con- ference Neural Information Processing Systems, wwt. forum?
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, J. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2023a
2024. URL ttttp:://operreee 1d=C62d2nS3KO.
Sohl-Dickstein, N., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. ArXiv, abs/1503.03585, 2015.
Wang. Z., Zheng, H., Hc, P., Chen, W., and Zhou, M. Diffusion Diffusion-GAN: Training GANs with diffusion. In The Eleventh International Conference on Learning Represen- tations, 2023b. URL https: openreview .nt// forum 2orum?dd=42EUUppWWi FUUDHHUA.
Song. J., Meng. C., and Ermon, S. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
Xu, Y., Liu, Z., Tian, Y., Tong, S., Tegmark, M., and Jaakkola, T. Pfgm++: Unlocking the potential physics- inspired generative models. In International Conference on Machine Learning, pp. 38566-38591, PMLR, 2023.
Song, Y. and Dhariwal, P. Improved techniques for training consistency models. The Twelfth International Confer- on Learning Representations, 2024. URL https: / //ppereevie net forum? d 9bRDVG.
Yin, M. and Zhou, M. Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5646-5655, 2018
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Neural Information Processing Systems, 2019.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.1344 2020.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F,, Freeman, W. T., and Park, One-step diffusion with distribution matching distillation. arXiv preprint arXiv: 188828, 2023.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models, arXiv preprint wwX.::233330044000 2023.
Yu, L. and Zhang, C. Semi-implicit variational inference via score matching. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / //openrevien wew....w. net. www.mmiiciiooooo (crrum?id=sd90a2yet.
Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., Liu, 7.,, Caterini, A. I.., Taylor, E., and Loaiza-Ganem, G. Exposing flaws of generative model evaluation metrics and their unfair treatment diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Yu, L.,, Xie, T., Zhu, Y., Yang, T., Zhang, X., and Zhang, C. Hierarchical semi-implicit variational inference with application to diffusion model acceleration. In Thirty- seventh Conference Neural Information Processing Systems, 2023. URL https: openreviereeeeeeeeeeeeeeee .nt// forum? forum?id=ghe d=ghhiiixghhhpxs
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2818-2826, 2016.
Zheng, H., He, P., Chen, W., and Zhou, M. Truncated diffu- sion prohabilistic models and diffusion- based adversarial

---

Continuous Semi-Implicit Models
auto-encoders. In The Eleventh International Confer-
Learning Representations, 2023a. https: //opennereee.net/ooonoeta hi-.rummmmddDDDD
Zheng, H., Wang. Z., Yuan, J., Ning, G., He, P., You, Q.. Yang, H., and Zhou, M. Learning stackable and skippable LEGO bricks for efficient, reconfigurable, and variable- resolution diffuusion modeling, 2023b.
Zhou, M., Wang, Z., Zheng, H., and Huang, H. Long and short guidance in score identity distillation for one-step text-to-image generation. ArXiv 2406. 0611,, 2024a. URL https://arxi:: org/ abs/ 2406. 01561.
Zhou, M., Zheng. H., Wang. Z., Yin, M., and Huang, H. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference Machine Learning, 2024b.
Zhou, M., Zheng, H., Gu, Y., Wang. Z. and Huang, H. Adversarial score identity distillation: Rapidly sur- passing the teacher in one step. In The Thirteenth In- ternational Conference on Learning Representations, net/ forum?
2025. URL ptttps://operreeeee id=1s2stfwizd.
12

---

Continuous Semi-Implicit Models
Algorithm 2 Training procedure of Continuous Semi-Implicit Models (CoSIM)
Input: Dataset D, pretrained score model Sg- (xs; 8), continuous transition kernel go (2./ 8, t), auxiliary vector-valued function (u( t) and X, a s.t. a = 1.2, a: X) = 1.0.
Output: Continuous transition kernel 44 ( [4; 8, t).
Initialization: o 0*, O with require all=False and iteration number 0.
repeat
Sample continuous time points (s) and t (t|ss) (o:
Sample 2o ~ 'D and
Sample æt ~ qu (2 ( 8, l).
uy (agg; s, - Sg- (Zsi s) - (z
if odd then
a ( Update @ t Φ nw (s) V ue (2:s: s, t)² [So- (2x: Ve, (o -)la 1)11
else
Update w + + nw2(8) (2 t)a Vv, log as (vc, ,, - Sg. (x.;; 8]] + ar11 + A) un 9, (1s,,)
end if
n1 1.
until convergence
A. SIVI Objectives
A.1. Details of training procedure
Choice of Time Schedule We now discuss the design principles of the time schedule a(s, t)) defined over |ő, 7]. First, we decompose (ss i) into the marginal distribution x zss) and the conditional distribution (T[ss). For m(s), we adopt the EDM time schedule (Karras et al., 2022):
(a min
+T, (o min)) )), T, Uniform (0, 1],
where min 6, max and
7.0. The conditional distribution 7 (r((s) is parameterized as:
1 t = (a min - r: (o o (mmm)) W,, = min + + 1};
where y m)llss). Since the continuous transition kernel Grrans (a.//=/ 8, learns the mapping from distribution p(:; to s):ss, the learning complexity increases with the time difference between f and s. Therefore, we design the distribution ग (t(rrs) to assign higher probabilities to larger time difference
y = B Uniform(( (0, Uniform 1 1
) +1 B), B Bernoulli )
where is hyperparameter controlling the probability of sampling smaller values of
Leveraging the aforementioned time schedule and adopting the weight functions (s) 1,2) from SiD (Zhou et al., 2024b), we present our complete training procedure in Algorithm 2.
B. Theoretical Result
B.1. Proof of Theorem 3.1
Here we provide general version of theorem 3.1 and illustrate the scaled Fisher divergence within the framework of generalized Fisher divergence (Cheng al., 2023). Consider the generalized Fisher divergence equipped with strictly smooth convex function h(x)))
GFI[h]([(s (Iga = Eystm, (V (Vlogp(z,;s) log (xx
where (0) and h(x))= > 0 for æ + 0.
13

---

Continuous Semi-Implicit Models
Theorem B.1. Let the variational distribution (a 8, i) defined in (7) and Pa(ss 6, 1) == V (og f ( 8, D).
Then the optimization of GFI(i)) (p(:s5) is equivalent the two-Stages alternating optimization problem as follows
min
Ba 22:0.))) uy (1s; 8,1" (V logp(r:;;)) - Vz, log Do (T, [2g 8, ()] h (12
(23)
@
min E Boo 20:0.))) (2,; t)² (V log (o 84880 1) V p 8)] + +
where X 2 0 a given regularization strength hyperparameter and h² (-) the Legendre transformation of h(.). Specifically,
if h(x)) = ta | and V Flogp(a.; p( is approximated by S (a: 8), (23) becomes the SiD loss (6). The optimal
(o) (Izis, s,t) of the second-stag stage optimization is given by
f 3V log (o(;s,t) (1 (0-8))5... (::8)
(24)
where B = 2a(1+X] Similarly, the equilibrium of 6 remains consistent with (11).
Proof of Theorem B.I. We can write the optimization problem with the generalized Fisher divergence GFI(h) as
min
E
V go
(25)
Since the Legendre transformation of h(x) is that hy) max - and h+*(-)) = h(-), we can rewrite the
optimization problem
min max (m. (sssss [ logp(z,;s)) V log (o.so( - h" (w,ss (=300)
(26)
By the trick on the score function V (o s, s,t), we have
(m.ss)))) (w, s, t), V log g0 (z.; s, t)) Egg (0a( s, V log go (1, sx)ss))
(27)
Therefore, the optimization problem (25) can be rewritten as these two stages optimization problem
min (go(
logp(w..;s) Ve,, lo. 8, 0) hi ()
(28)
2e
min 0g
(x. (211.))) (p( V a,, log (2, V Plggp s)] h1" (ww t))
(29)
the regularization strength parameter 0, the optimization problem with regularized term is formed as
min Pg
Beo((n (aggi s, s) [V log (x,;stt) V Vlogp(z,:s) s)] (1 + (os (o.(zz;s,
(30)
We can view the above minization formulation as the Legendre transformation with the convex function (1 A)h*(.) (.), then the optimal function (a s, of the above optimization problem satisfies
Eg (2)2) (0)) t)a [V log 4s (z.:8 V log p(r,; s)] + (1 + A)A" (1000))())))))
(31)
=- (x..:.)))) (1 2)h" ]
( (Vlogp(z.;s) V (oggo
(32)
1
(1 A)h (logp(z,;5) Vz. log (z.;
(33)
Bring the above optimal function (;-(6)(;8,t)) back the first stage optimization problem (28), then have
(m, U (2,s))))sss (oogp(e,;)) Ve, log (2, h (ow
(34)
-E (am))))))
A)h
(log p(z.:s)))) Vn. log (og + Ah (1(0000))(2208
(35)
1+
14

---

Continuous Semi-Implicit Models
As is strongly smooth convex function, the Legendre transformation h* is also strongly convex function, and h((=) 0 if and only if 0. When (d) 8, 0, the first derivate condition of the optimal function (o) is satisfied
V v, log (2,; s,t) logp( s)] (1 + A) V log qo s t) logp(æz;: 0.
Therefore, the global optimal distribution of (34) is qo- (z,; s,t) = V log p s) that similar with the original optimization problem (25), So the two stage optimization problem with regularization term has the similar optimal solution as the original optimization problem.
In practice, choose the function h(x:))
4a ||ar [|g, then the Legendre transformation of h(x) is that
11 hr(y)) m 1 4a
[,=1|]] al|y||
Substituting the Legendre transformation h=))) back into the second stage optimization problem (30), we obtain
E (2) Uas (a,)is ()
[V (oggs(
V Vogg(æ.,; 8)] + (1
8,
V Vlogp(..; x) || C(o),
-11 + 2) A) (0 05 (2z;
log 9o s
2(1 Aja
where (²)) (x..s.))) V log To of the second stage optimization problem (30) is that
t) V pogp s) 112 and is independent of tb. Therefore, the optimal function
(a) (, =V log logp( 8)
1
| (a + Vlogp(æsis)) s]]
2(1 Aja
Replacing the target score function V log p(z,; s) with the estimated score model S (xx; 8), eventually write the optimal fai ( () of the second stage optimization problem with the regularization term as
U (6) s,)) = Se- (x (o)
(36)
1 2a(1 + +))
f ( (Zs)8,)) 2xx1 + A)
V Vloggo
) S0- (2)
(37)
Moreover, (34) is rewritten
(a. ( (0): 8, (So (2x;;));) - Vn, log (a, 8, f)] ail(u (m, 10)
(1+ 2A) 4a X)²
E (ws:s.))) So- ( V log (2.s 8, 1).
Therefore, the equilibrium of the two-stage optimization problem for e remains consistent with (11).
B.2. Proof of Proposition 3.2
Proof. As discussed in the proof of theorem 3.1, the two-stages optimization problem in (15) is equivalent to optimizing
(1 + 2A)
min And the continuous transition kernel gurns (x, [z; s, t) parameterized as
4a(1 A)2 (ys( VV - V log (I
(38)
a (xr me; 8, t) = N( (Z(
(39)
where 0 8 S. a(8), (x) E + are defined in (2). Then we have that the marginal distribution go 8, L) is supported on IRd since
go (x
400 s, s, ()p( Odxx >0, Vos, Rdd
(40)
15

---

Continuous Semi-Implicit Models
Therefore, the optimal variational distribution (0(2g 8,() in (38) satisfies that
V logp( s) V (o s,t,, Vzz. R
It implies that (z(zii p(x,;s)) s). Furthermore, for the independent random variables as p(:it) and (0,,1). the characteristic function of (- ()p((:() satisfies
(G)()))pp(-s))) (w) Ee Go Be atoy w (a()))a((aa))) (Ee s(a) w e)--
Bo mb, "n an (Ee mto) "² 9--1 Bp(mm.s) "ato wn my (Ee (2()) w0 9)-1
V(ceo,) aly (a(s)xo (Be ccto in 9)² =
Pamp (oo (w).
Therefore, the characteristic function of ())((;t)) t) is the same the characteristic function of p(zv; 0). We can conclude that G=(-, establishes a mapping from distribution p(:() back to p):
B.3. Proof of Proposition 3.4
Proof of Proposition 3.4. As the approximated variational distribution 93 is obtained by
8,(18,1) E arg min E. (2 u (i(o) (05; (V p(as;)))))) Var, log (ar (215,,1 1)] a all (a 8, 1)]
(41)
where u (0) st) = v log logp(<<;s) - f fútt) (2x)) t).
We can bound the Fisher divergence between q3 and s): as follows. First we rewrite the first stage optimization objective as
Uo (1 s, t)² V p(pp((s))) Vv, log qol (1g xx s, t)] allu (z:; t)1
= F (essii))))) (M
1)² V log ((gs((>;;))
Va,
log do (0s; 8, 1)] - a all
1 21
Byg (20 (2.
1 X Fvole.:)1))
(1);;;; 2a(1 + A)
4a(1 + 2)2
8 (2.;
(42)
2a(1 + A)
where 60(1)18tt) V logp(zz.s)
V (w s,t). Then we have
Fl(g)( {))p((ss)) = 4a(1 + X)2 Bag :s Vlogp(zz;; V log g3 (x,;;,))) s,t)
4a(1 + x)2
2A
A
E 1m, (Z.i
Fl
4a(1 + 2)2
2a(1 + A)2
Bring (42) into L, we have
-gg Eggg (cn (E,is,) [ Vlog((ssss) Va, (oggg((sis,()) ( (2..i,
A
(x.:s,t) 8 (2(20ois (1s 2a(1 + A) 5 (s(0sis,))) + 2a + A) 52620, f)]3
1 1+1
< BBg (estai0) (mssoo (mai [V logp(w.;)) - v log go- (#,;8,1) a (10))) (#s; 1) B (0) (2) + az
16

---

Continuous Semi-Implicit Models
For the term (e. (2), have
a)
-LE (1)))0,)) (3) -
B aggim Sagtm 160000 +
E (a (10011) 2a(1 2) 83(25 s, f)112.
20x(1 + 2)
2
2a(1 2)2 Fl
2227 =7.
ad
For the term 3, by (42)we have
1 2X
1 2a(1 + 2)
Bys (aresa))) 1lo (x,; t)l2 -
6ør (2)s 1)
4a(1 + 1)2
(veris)) ( 6ar (2siis,), t). (( (2sis,)) 20(1 2)
80 00- (001181,)
+ X
+ 2A
4a + x)2 Fllqo- (1s,t)]p((ss) aB (m.:s.t) 1 ا (Esis,t)
2a(1 + A) 85. (xs; s, s,t]
X²
1
+ 4(1 + 2)2a
(x,; 2)12 + aB Ey (m, (
60- (z.; 8,1
2a(1 + x)
Fl(go- (5 ())
Bring the above results back to @ we have
A
Fl(gg(11s =4 + () 2a(1 + 1)2 (l(gg(:ss
<(1 + F) (: t)lp((;s))) (1)l)(())))))) 2o 200 A)e?
(43)
Introduce the approximation gap from Assumption 3.5, we have
Fl(ga(::, s, ())((( s)) s 2(1 + 2) s)) + 21 + 2) N)
(44)
The assumption 3.5 quantifies the capacity the consistency map G (,, 1) based on the initial gap. That is, the Loo (x;; 8, s)
[((]) (x; 0, s))x, t), = p(=;t when Goo (x,, Dt. Therefore, the initial approximation gap FI(qoo (- +
s)]pp(, 8)) = Fl(p(;t s))(p())).
B.4. Proof of Proposition 3.8
Proof of Proposition 3.8. First we bound the Wasserstein distance between (=(:: and p(: 0). Since p(-:0) is supported on the hypercube 1]4, its support is contained in the Euclidean ball B(0, vd). By the results of bounds the logarithmic Sobolev inequality (LSI) constants (Bardet et al., 2018; Chen et al., 2021; Cattiaux & Guillin, 2022), we can derive the following LSI inequality
Ep(ma 2 (xs) ( (xs) - log 6) of f(x)) & 20LsIE [f
Vf e c (R2))
(45)
(z(:5.))
where the LSI constant CLSI < 6(4d + (o)) exp( 66) Let f?(-)
then we have
P(-5)
4d
(L(gg(50,t)lpp(:))) S (4d + (6)) exp(
Fl(gs(=: ())
0
Moreover, by theorem (Otto& & Villani, 2000), the following Talagrand's inequality holds
S (4d ( exp( 8d ()),
W (:(0)(::.tt).p(o)) & (c)
0
Denote A = (4d+a(()) 2 exp( ( ), we have the bound of Wasserstein distance between (=(=;6, and p(: 6) as
W3 8,t). S Etg(t) = t +d))p(;;))) + Ae?
(46)
17

---

Continuous Semi-Implicit Models
Next, following the methodology proposed in (Lyu al., 2024), give the bound for the multistep sampling of CoSIM.
Given sequence of fixed time points T 2 2 2 x 2 tx 6, we denote 2g ~ p(: to) and
z a(t)) G (2 11) a(tx))
(47)
(8) 2 = ( 1 + a(8)eex-1) = , 9' (::8) Law Law
(8)
(48)
where 6g (O, I) independent Gaussian noises, then it can be shown that o3
(2) (::) 9g(:5 T). For fixed time
points f2. we consider the optimal transport coupling 7(zx; T.a)) E T Top (Law(zx), p(: 4x)) and let
A k+1
[a())G (* (k) o6
(k)
(::).
9
y ( 1x) + E0)
9g(15, tx)-
Then we have
W2(05 .p(:0)) SW2(1 03(:5, (x)) +
(k-1) 3
(k+1)
< )m 11
y11 + ( (x)
=a R
(2x. G
G2 4 4)12 + (c)(:;
(49)
Recall that the consistency function aa( is parameterized as
G (mr,))) () = (Casi(() ge + Comi F (cm((1) ())
(50)
where
ann a(t)
(skip(t) = o (1)2 + amn a(t
(51)
Cour (tu ot)2 (t)
data al a(t) 1/2
(52)
(in(t) = tdma)
1/2
(53)
Practically, adma == 1/2 is fixed constant. Given any 3i, 22 Rdd and by the Lipscchit continuity of Fs(: 1) with respect He, we have
([),( (z( (t)cskip(t x2) Cour (F (aa
S(Caip (t) Coul (1) Ce (1)) x3 x2l12,
a(t)
(t)a(t))
< att + alt)2 + 2L, (t) + a(t) )| x2112
(54)
In the case of variance preserving (VP) scheme for (1), we have
a(l) = = al) = V1- Vt [0, T).
(55)
This implies that (2 t) is -iipschtt continuous with respect to æ if a(t), (t)) is defined as (55).
In the case variance exploding (VE) scheme for (1), we have
a(t) 1. a(t)= =t, Vt E [0,
(56)
Then (z) is 2 + アनन L-iischitz continuous in the variance exploding scenario. Denote these two Lipschitz constant as Iyp = 3L, adn Iave = FFI
+ से respectively. To ensure the condition ( > ad, we have the early stopping time 6up = (l)) for VP scheme and d'e ((vd) for VE scheme.
2L,
18

---

Continuous Semi-Implicit Models
For the variance preserving scheme, substituting the above results into (49) yields
W2(13 (k+1) .((:))) ((..)),,, 21, + (
Wa
Next, given the optimal coupling x²') E Topi (8) (::)) 8)), we consider the following coupling (222
p(1
a(tı 6) (6) + ott 8) x = atts - 5)æz + ot4 - 8)
Then
W2(Law(z)) p(:ta))= S (1((z,))>> - a(t; - 6) |z' - =e'113 S a(tx - 3) (:0).((:))
Therefore, have
W2
(k+1) .p(55)) S a(lx)) W2 (:0).((()) (=)0 + 3 (4x)
Let f1 = tk-1 = timd > with a fixed Eiid and apply the discrete type Grönwall's inequality (Gronwall, 1919), we have
$ (K) (:0))((;) $ (::0) 5
5od + (a(Imia) Imp)) K (T) + (1 - -a(4mia) [xp) [xp) -18 (mmid),
W?
=82d + (K-1)) ((K- tmot))))
53
(T) + (1 tan (mmotlog((3,))) -1
113 (Imid)-
1
For the variance exploding scheme, we have
W2 (K) (::0). (x)0))(()))))))))))))))))) (::)).((;))) 5d mid +
2Lg
(T)
(1
2L, )-
(lmia)
W?
miid
Therefore, the first term right-hand side has exponential decay rate as fmid log(4L5)) for VP scheme and fmid 4hg for VE scheme.
C. Experimental Details
Our consistency method is to build on top of existing pre-trained diffusion models and distill from them. With numerous existing diffusion models, we choose our base models using the following criteria:
1. Completely open-source, including checkpoints, model architectures, and all training and inferencing details.
2. Results that are generally recognized to be reproducible.
3. State-of-the-art performance.
Fortunately, all these are satisfied with EDM (Karras et al., 2022) and EDM2 (Karras et al., 2024). They are set to be our base models.
Pre-processing and Evaluation followed EDM to process CIFARI 10 and Imagenet x 64, and followed EDM2 for Imagenet 512 512. Most of the consistency methods we compared with in Table 1,2,3 follow the same pre-processing protocol. However, CTM (Kim et al., 2023) followed different way to down-sample Imagenet to 64 x 64. More precisely, CTM has a different down-sampling kernel compared to EDM.
This caussed a significant disruption to the FD-DINOv2 value. Because FD-DINOv2 is computed at a fixed resolution 224 x 224 and all images at lower resolution have to be up-sampled before computing this value, the final result will be very sensitive to the lown-sampling kernel. For a fair comparison, we compute FD-DINOv2 between CTM generated 50K images and 1, 1271,167 Imagenet images down-sampled to 64 64 using the same kernel as CTM.
19

---

Continuous Semi-Implicit Models
Table 5. Hyperparameters for different experimental setup.
llperparaaeeters
CIFARIO 32
Imagenet 64 x 64
Imagenel 512 x 512 S
Imagenet 512 x 512 M
Imagenet 512 x 512 L
Batch Size
2048
2048
2048
2048
2048
Batch per GPU
64
16
32
32
32
Gradient accumulation round
16
8
8
# GPU (L40S 48G)
8
8
Learning rate G2 and f of EMA half-life images
le-5
4e
2e-5
2e 2--5
le 1e-
0.5M
2M
2M
Optimizer Adam eps
1e-5
le 12
1c 12
le 12
le-12 12
Optimizer Adam 31
0.0
0.0
0.0
0.0
Optimizer Adam 92
0.999
0.99
0.99
0.99
0.99
R of total training images of parameters dropout
8
8
4
200M
200M
200M
200M
20M
56M
296M
280M
498M
778M
0
0.1
0
0.1
0.1
augment
0
0
0
Table FD-DINOv2 results of CoSIM on class-conditional ImageNet (512 x 512) with different regularization strengths (coef) and total training images. All results evaluated with NFE= 2.
Regularization Strength
204k
1024k
2048k
4096k
coef 0.5
309.77
101.33
69.65
61.53
coef = 0.75
392.85
92.81
61.77
50.54
coef 1.0
121.90
95.23
58.63
49.28
Network Architecture and Initialization Following Algorithm and Section 3.3, there are three networks our training process, generator Ga teacher Sg- and auxiliary function fa. All are initialized from the pre-trained checkpoints of our chosen base model. During our training. Sg- frozen while generator G and auxiliary function f are iteratively refined. During inferencing, only generator G. is used generate new samples.
Since G F Sg- are initialized from the same checkpoint, their architectures are kept the same as the base model. But fo must accommodate the additional imput from Section 3, so we duplicate the time-embedding layer as described in Section 4. This only adds a very small number of extra parameters during training, and makes no change to the generator used during inferencing.
The hyperparameters for all of our experiments are presented in Table 5. For those parameters not mentioned in this table, they are kept the same as the base model.
Hyperparameters
Training Budget We conducted all of our experiments using 8x NVIDIA L40S GPU with 48GB video memory. For CIFARIO 32 32, train models for days. For Imagenet 64 64, need 7 days to reach our reported results. On Imagenet 512 x 512, though the final image size is significantly larger, the training of consistency model is still conducted on 64 x 64 internally. This is because EDM2 (Karras et al., 2024) applied a VAE to encode original input size into 64 x 64 latent size, and our consistency model only works on the latent feature. For S,M,L setups, the training takes 3. ~ and days accordingly.
Sampling Steps In Section 4.2, we show that 4-step sampling generates better quality than 2-step sampling. Here we provide more visual results Figure 8. Compared 4-step sampling results, some details deteriorate 2-step sampling, such the floating leaves on the spider web, and the shape of shoes are not ideal.
On the Role of We conducted an ablation study on À with the coefficient coef a(1 X) in (15) for ImageNet 512 x 512 generation of CoSIM using the L model size with fixed 1.2.
The results show that introducing regularization (i.e., increasing coef) within suitable range significantly enhances the learning process of fv, which in turn improves the training of gg.
Here we provide more visual results on our experiments.
20

---

Continuous Semi-Implicit Models
y
Figure Unconditionally generated 32 x 32 images on CIFARIO using 2-stop sampling.
21

---

Continuous Semi-Implicit Models
Figure Unconditionally generated 32 x 32 images on CIFARIO using 4-stop sampling.
22

---

Continuous Semi-Implicit Models
Figure 5. Conditionally generated 64 x 64 images on Imagenet using 2-step sampling.
23

---

Continuous Semi-Implicit Models
Figure 6. Conditionally generated 64 x 64 images on Imagenet using 4-step sampling.
24

---

Continuous Semi-Implicit Models
Sulbeno
(a) 2 steps
(b) steps
Figure 7. Class-conditioned 512 x 512 images generated by CoSIM with different steps on Imagenet using M model, starting from identical noise.
25

---

Continuous Semi-Implicit Models
(a) 2 steps
(b) steps
Figure Class-conditioned 512 x 512 images generated by CoSTM different steps on Imagenet using I. model, starting from identical noise.
26


Reviews:
[{"review_id": "m4Sdr73G4g", "reviewer": "Reviewer_vhoo", "summary": "This paper proposes CoSIM a continuous hierarchical variational inference framework for semi-implicit distribution aiming to accelerate the sampling speed of diffusion models. The main contribution of this paper includes:\n\n * The authors propose a score-based diffusion distillation approach demonstrating superior performance on DINOv2 metrics comparing to data-free diffusion distillation algorithms with 4 sampling steps across multiple datasets (CIFAR-10, ImageNet 64×64, and ImageNet 512×512).\n * The authors theoretically connects the Hierarchical Semi-Implicit Variational Inference (HSIVI) frame work with the Score-identity Distillation (SiD) and propose a regularization technique to address the bias of the fake score network in the score matching loss.\n *  The authors demonstrate that the generator network of SiD is intrinsically a consistency function and prove that a few-step sampler could potentially be a sweet spot in the tradeoff between iterative refinement sampling and error accumulation in multi-step sampling\n\n## update after rebuttal: See the comments below", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. We address your specific questions and comments below.\n\n`Q1`: It does not seem very clear to me what is the bias in the SiD loss ... on $\\lambda$.\n\n`A1`: Thank you for your valuable suggestion! The significance of our regularization comes in three aspects:\n- Regularizing the second-stage optimization of (10) (as discussed in Theorem 3.1) helps guide the optimal $f\\_{\\tilde{\\psi}^*(\\phi)}$ towards the pretrained score function $S\\_{\\theta^*}(\\cdot;s) \\\\approx p(\\cdot;s)$. And the Nash equilibrium of $\\\\phi^*$ remains consistent as in Theorem 3.1. \n- Our formulation strictly contains SiD loss as a special case, which is equivalent to $\\alpha = 1.2$ and $\\alpha(1+\\lambda) = 0.5$. This introduces a potential bias when $f_\\psi$ is perfectly trained and $\\alpha$ is too large. \nAs shown in (38), this causes $q_\\phi$ to optimize in a direction that deviates from the target distribution, corresponding to the failure case in Figure 7 of SiD.\nOur formulation allows for a larger $\\lambda$, but still ensuring positive coefficient $(1+2\\lambda)$ in (38), mitigating the potential bias in SiD loss.\n- The ablation study on $\\mathrm{coef} := \\alpha(1+\\lambda)$ in (15) with $\\alpha=1.2$ show that introducing regularization facilitates the learning of $f_\\psi$, improving the training of $q_\\phi$.\n\nTable1: Class-conditional ImageNet (512x512) for CoSIM-L given **$\\alpha=1.2$**.\n|$\\mathrm{coef}$ \\\\# of total training images  | 204k| 1024k | 2048k | 4096k|\n|-|-|-|-|-|\n|$\\mathrm{coef}=0.5$ with NFE=$2$| 309.7 | 101.3 | 69.6 | 61.5 |\n|$\\mathrm{coef}=1.5$ with NFE=$2$| 421.9 | 95.2 | 58.6 | 49.2 |\n\n\n`Q2`: The authors mentioned in line 167-169 that the fused loss function ... on $\\alpha$ if time permits.\n\n`A2`: The pathological behavior when $\\alpha > 1$ refers to the SiD loss which is equivalent to $\\mathrm{coef} := \\alpha(1+\\lambda) = 0.5$. We propose using a regularization term by increasing $\\lambda$ to mitigate this issue, as discussed in A1.\n\nThe ablation study below with a larger $\\alpha=2.0$ shows the effectiveness of our regularization and the pathological results of SiD loss is similar to the observations shown in Figure 7 in SiD's paper.\n\nTable2: Class-conditional ImageNet (512x512) for CoSIM-L given **$\\alpha=2.0$**.\n|$\\mathrm{coef}$\\Iteration | 204k| 1024k | 2048k | 4096k|\n|-|-|-|-|-| \n|$\\mathrm{coef}=0.5$ with NFE=$2$| 374.8 | 3960.0 | 4129.3 | 4039.9 |\n|$\\mathrm{coef}=2.5$ with NFE=$2$| 619.0 | 114.1  | 78.7   | 64.9   |\n\n`Q3`: My major concern lies in the statement for Proposition 3.4 and assumption 3.3 ... regularization. \n\n`A3`: $\\epsilon_f$ measures the gap between $f_\\psi$ and its optimal $f_{\\psi^*}$ in the second optimization problem described in (12). This optimization is a quadratic problem, and in the optimal case, $\\epsilon_f$ approaches 0. Therefore, the introduction of regularization does not affect $\\epsilon_f$'s convergence to 0. Moreover, when $f_\\psi$ is initialized as $S_\\theta$, regularization brings $f_\\psi$ closer to the optimal $f_{\\psi^*}$, as shown in (16).\n\n`Q4`: I noticed that CoSIM 4-step differs by only 0.06 ... observation.\n\n`A4`: FID and FD-DINOv2 are designed in different numerical scales. FD-DINOv2 was designed more suitably for diffusion model evaluation. Results below are some well-known diffusion models evaluated for conditional image synthesis on ImageNet $256\\times 256$.\n| Metric \\ Model|   ADM  |   LDM  | \n|-|-|-|\n| FID       |  4.59  |  3.60  |\n| FD-DINOv2 | 203.45 | 112.40 |\n\nFurthermore, FID was introduced in 2017 and does not have enough numerical range to differentiate strong modern diffusion models. FD-DINOv2 was introduced to provide a larger numerical gap to solve this issue.\n\n`Q5`: The contribution of this paper is solid yet the overall significance and novelty is moderate.\n\n`A5`: Thank you for acknowledging the novelty of theoretical derivation of the fused loss and insight of regularization. \nWe believe that CoSIM contributions go beyond SiD in the following aspects:\n\n- We theoretically introduce a regularization framework of the SiD loss to stabilize the training and solve the pathological behavior under large alpha values.  \n- We propose a continuous HSIVI training scheme. With a specific parametrization, we derive a novel training method for consistency models at a distributional level. \n- Experimentally, our method performs better than the state-of-the-art consistency model sCD [1], using much less training budget (ours 20M compared to sCD 819.2M). \n\nTable3: Sample quality measured by FID on class-conditional ImageNet (512x512). \n|Model|#params|NFE|FID|\n|-|-|-|-| \n|sCD-S| 280M | 2 |2.50|\n|sCD-M| 498M | 2 |2.26|\n|sCD-L| 778M | 2 |2.04|\n|CoSIM-S| 280M| 2 | 2.66|\n|CoSIM-S| 280M| 4 | 2.56|\n|CoSIM-M| 498M| 2 | **1.95**|\n|CoSIM-M| 498M| 4 | 1.93|\n|CoSIM-L| 778M| 2 | 1.84|\n|CoSIM-L| 778M| 4 | **1.83**|\n\n[1] Lu Cheng et al. \"Simplifying, stabilizing and scaling continuous-time consistency models.\"", "claims_and_evidence": "This paper overall supports the claims with clear evidence. However, my major concerns lies in the following two aspects:\n\n* The authors mentioned in line 199-202 that when $\\nabla \\log q_{\\phi} (x_s;s,t)$ deviates significantly from the target score model $S_{\\theta}(x_s;s)$, this initialization strategy becomes inefficient for the second-stage optimization in equation(10). It does not seem very clear to me what is the bias in the SiD loss suggest by the authors and how this is connected to the regularization, I would appreciate more elaboration from the authors on this, either theoretically or through ablation studies on $\\lambda$.\n\n* The authors mentioned in line 167-169 that the fused loss function exhibits pathological behavior when $\\alpha>1$, I guess this is a typo? Otherwise it would be interesting to ask why the authors follows $\\alpha = 1.2$ in the implementation of CoSIM. Given the combination of $\\lambda$ and $\\alpha$, it would also be interesting to see the ablation studies on $alpha$ if time permits.", "methods_and_evaluation_criteria": "The methods and evaluation make sense to me.", "theoretical_claims": "The proof for Theorem B1 Proposition 3.2 and 3.4 make sense to me. My major concern lies in the statement for Proposition 3.4 and assumption 3.3, $\\epsilon_f$ may be inflated here due to the regularization, so it would be good to show that how regularization benefits the training of CoSIM.", "experimental_designs_or_analyses": "I noticed that CoSIM 4-step differs by only 0.06 from SiD on ImageNet 64×64, yet shows a significant 20-point improvement in FID DINOv2. Interestingly, CoSIM 2-step differs by a larger margin of 1.83 from SiD on ImageNet 64×64, while maintaining the same 20-point improvement in FID DINOv2. I would appreciate intuition on this observation.", "relation_to_broader_scientific_literature": "The key contributions of this paper can be related to accelerating the inference of diffusion models and enriched the tools in hierarchical variational inference and provide a novel perspective on Score-identity distillation for diffusion models.", "essential_references_not_discussed": "Related works that are essential to understanding the key contributions of the paper are cited.", "other_strengths_and_weaknesses": "The paper provides solid theoretical derivation backed with extensive experiment. However, I think the Section 3.1 and 3.2 essentially re-derives the SiD loss for a few-step sampler, which is limited the novelty. Though the derivation through Legendre-Fenchel Dual provide insights on the loss function of SiD, the improvement comparing to the SiD bi-level algorithm optimization is limited. The derivation on the error bounds for multi-step sampling could provide insights on few-step diffusion acceleration. The contribution of this paper is solid yet the overall significance and novelty is moderate.", "other_comments_or_suggestions": "Please see the above sections.", "questions_for_authors": "Please see the Claims And Evidence section. It would be helpful if the authors could elaborate more on the $f_{\\psi}$ shifting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bcWhfU7NaV", "reviewer": "Reviewer_yUwN", "summary": "The paper proposes Continuous Semi-Implicit Models (CoSIM), extending hierarchical semi-implicit models into a continuous-time framework for diffusion model acceleration.\nCoSIM introduces a continuous transition kernel that allows the simulation-free training.\nIt uses semi-implicit variational inference (SIVI) as training criteria, providing a distributional-level consistency model without relying on moment-level or sample-level reverse processes. \nThe method yields comparable or superior results on FID and improves existing approaches on the FD-DINOv2 metric on CIFAR-10 and ImageNet datasets.\nThe authors further provide theoretical insights into their approach via error analysis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable feedback! Here are our responses.\n\n`Q1`: I personally like this paper. It extends HSIVI ... paper's contributions.\n\n`A1`: Thank you for your interest in our work! We acknowledge that CoSIM can be regarded as a continuous extension of HSIVI, but our contributions go beyond that and is different from traditional consistency models in the following aspects:\n- Unlike traditional ODE trajectory distillation applied by normal consistency distillation methods, we train the consistency model at a distributional level using Fisher divergence. This approach eliminates the need to constrain the generator $G_\\phi$ to sample along a reversed ODE path predefined by the score function, allowing the model to learn a more flexible consistency mapping on a distributional level.  \n- Experimentally, we find that this distinction enables CoSIM to train much faster outperform the state-of-the-art consistency model sCD [1], using only 20M total training images (compared to the 819.2M by sCD). We provide extended results of CoSIM compared with sCD on FID, demonstrating its superior performance across various model sizes.\n\nTable1: Sample quality measured by FID on class-conditional ImageNet (512x512).\n|Model|#params|\\NFE|FID|\n|----|----|----|----| \n|EDM2-S-FID| 280M| 63 | 2.56|\n|EDM2-M-FID| 498M| 63 | 2.25|\n|EDM2-L-FID| 778M| 63 | 1.96|\n|sCD-S| 280M | 2 |2.50|\n|sCD-M| 498M | 2 |2.26|\n|sCD-L| 778M | 2 |2.04|\n|CoSIM-S| 280M| 2 | 2.66|\n|CoSIM-S| 280M| 4 | 2.56|\n|CoSIM-M| 498M| 2 | **1.95**|\n|CoSIM-M| 498M| 4 | 1.93|\n|CoSIM-L| 778M| 2 | 1.84|\n|CoSIM-L| 778M| 4 | **1.83**|\n\n`Q2`: How sensitive is CoSIM to the choice of the regularization hyperparameter?\n\n`A2`: We conducted an ablation study on $\\lambda$ with the coefficient $\\mathrm{coef} := \\alpha(1+\\lambda)$ in (15) for conditional ImageNet $512 \\times 512$ generation of CoSIM using the $L$ model size, with fixed $\\alpha=1.2$. \nThe results demonstrate that introducing regularization (i.e., increasing $\\mathrm{coef}$) significantly enhances the learning process of $f_\\psi$ in a wide range.\n\nTable2: Sample quality measured by FD-dinov2 on class-conditional ImageNet (512x512) for CoSIM-L given $\\alpha=1.2$.\n|$\\mathrm{coef}$\\ \\# of total training images  | 204k| 1024k | 2048k | 4096k|\n|-|-|-|-|-|\n|$\\mathrm{coef}=0.5$ with NFE=$2$| 309.7 | 101.3 | 69.6 | 61.5 |\n|$\\mathrm{coef}=1.0$ with NFE=$2$| 392.8 | 92.8 | 61.7 | 50.5 |\n|$\\mathrm{coef}=1.5$ with NFE=$2$| 421.9 | 95.2 | 58.6 | 49.2 |\n\n`Q3`: Computational Overhead: Could you clarify or quantify the computational overhead ... SiD and Diff Instruct?\n\n`A3`: There are three networks in the training process, generator $G_\\phi$, teacher $S_{\\theta^*}$ and auxiliary function $f_{\\psi}$. During training, teacher $S_{\\theta^*}$ is frozen while generator $G_\\phi$ and auxiliary function $f_{\\psi}$ are simultaneously evolving.  For SiD and Diff Instruct, all three networks are kept the same as the baseline methods and initialized from the same checkpoint. For our method, the auxiliary function $f_{\\psi}$ is modified to incorporate the extra time parameter $s$ introduced in Section 3. In practice, we duplicate the time embedding layer, which only adds a very small number of extra parameters.\n\nSpecifically, on CIFAR, the total number of trainable parameters is 111,465,478 for SiD and Diff-Instruct. With our modification, it increases to 116,128,006, adding only 4% extra parameters. \nOn ImageNet $64\\times64$, the total number of trainable parameters is 59,1798,534 for SiD and Diff-Instruct. With our modification, it increases to 619,999,878, also adding only 4% extra parameters.\n\nAdditionally we time the computational overhead for SiD, Diff-Instruct and our method on unconditional CIFAR generation. We conduct all experiments on a single L40S GPU with all fixed hyperparameters such as batch size, learning rate, optimizer etc..\n\nTable 3: training time for 10496 images\n| Time \\ Model|   SiD   | Diff-Instruct |  CoSIM  |\n|------|:-------:|:-------------:|:-------:|\n| Time | 2min28s |    2min42s    | 3min08s |\n\nCompared to SiD and Diff-Instruct, our method requires 27% and 18% more training time respectively. Apart from the extra trainable parameters explained above, the regularization we introduced in Theorem 3.1 also contributes to the extra time. \n\nWe argue such overhead is worthy, as the overall training objective stabilizes and the total training budget shrinks. On conditional imagenet $64\\times64$ generation, our method only requires 200M training images to reach FID 1.46, surpassing SiD which needs 1000M training images to reach FID 1.52. On the same task, Diff-Instruct reaches FID 5.57 with 4.8M training images. Since we aim for a much lower FID, we did not frequently test our model in the early stages of training. The closest one we have is that our model reaches FID 4.15 with 5.12M training images.", "claims_and_evidence": "The paper makes the following claims:\n\n- Faster Convergence: The authors show that CoSIM reduced the number of iterations required for distillation. \n\n- Improved Generation Quality: CoSIM achieves comparable or better generation quality. Supported by Comparable FIDs and better FD-DINOv2.\n\n- Continuous-time Framework: The authors extend the previous SIVI to a continuous framework.", "methods_and_evaluation_criteria": "- This paper evaluates CoSIM using widely accepted metrics (FID, FD-DINOv2) and datasets (CIFAR-10, ImageNet) for generative modeling.\n- The FD-DINOv2 metric specifically helps evaluate perceptual alignment with human vision, a useful complement to standard FID metrics.\n\nThe chosen evaluation makes sense given the objective of achieving accelerated image generation.", "theoretical_claims": "The paper provides several theoretical claims:\n\n- Derivation of the continuous transition kernel.\n- Proofs of unbiasedness and error bounds for two-stage optimization (Theorem 3.1).\n- Error analysis (Propositions 3.3 and 3.4), providing theoretical guarantee about the quality of multi-step sampling.", "experimental_designs_or_analyses": "- Standard benchmarks on CIFAR-10 and ImageNet datasets for unconditional and conditional generation tasks, respectively.\n- Comparison against strong, established baselines such as SiD, iCT, ECT, CTM, and Moment Matching methods.\n- Reporting quantitative (FID, FD-DINOv2) and qualitative evaluations (sample images).\n\nExperiments look valid and sound. This paper also outlines experiment setups and hyperparameters.", "relation_to_broader_scientific_literature": "- Builds upon recent advances in diffusion model acceleration and semi-implicit variational inference (SIVI).\n- Identifies the advantages of continuous over discrete-time frameworks in hierarchical generative modeling.\n- Compares itself to recent notable works (e.g., SiD, HSIVI), effectively highlighting contributions and improvements.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "I personally like this paper. It extends HSIVI to continuous-time framework and provides a solid theorectical ground for prior works like SiD and validates the regularization techniques.\n\nHowever, I find the paper's emphasis on its connection to consistency models overstated. The current approach seems more naturally framed as an extension of HSIVI to continuous-time settings or as a multi-step generalization of SiD. The relation to consistency models feels unnecessary and shouldn't be demonstrated as the paper's contributions.", "questions_for_authors": "- Regularization Strength:  How sensitive is CoSIM to the choice of the regularization hyperparameter? \n\n- Computational Overhead: Could you clarify or quantify the computational overhead introduced compared to simpler baseline distillation methods, such as SiD and Diff Instruct?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "VplcvaLEHb", "reviewer": "Reviewer_hsDu", "summary": "CoSIM is a continuous extension of hierarchical semi-implicit models, designed to enhance expressiveness and accelerate diffusion models with pretrained score networks. Unlike traditional hierarchical semi-implicit models, which suffer from slow convergence due to sequential training, CoSIM introduces a continuous transition kernel, enabling efficient, simulation-free training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We will address your concerns below.\n\n`Q1`: It is recommended to provide empirical evidence supporting this claim, such as FID/FD vs. training iterations or runtime comparisons.\n\n`A1`: We give the details of our training iterations in Table 4 in Appendix C. For comparison with HSIVI, we give the following two tables. \n\nTable1: Training Efficiency and Performance on Unconditioanl CIFAR10\n|Method | \\# of total training images| FID |\n|-|-|-|\n|HSIVI| 27 M | 4.17 (NFE=15)|\n|CoSIM| 20M | 2.55 (NFE=4)  |\n|CoSIM| 200M | 1.97 (NFE=4) |\n\nTable2: Comparison of Training Efficiency and Performance on ImageNet $64\\times 64$\n|Method | \\# of total training images| FID |\n|-|-|-|\n|HSIVI| 27 M | 15.49 (NFE=15)|\n|CoSIM| 20M  | 2.51 (NFE=4)|\n|CoSIM| 200M | 1.46 (NFE=4)|\n\n`Q2`: Is the training stable given that multiple networks are trained simultaneously? How does the parameter lambda affect performance? Is there an ablation study?\n\n`A2`: Yes, all networks are trained simultaneously and stably, which has also been observed in previous works such as SiD[1] and Diff-Instruct[2]. We provide an ablation on $\\lambda$ in the form of coefficient $\\mathrm{coef} := \\alpha(1+\\lambda)$  below.\n\nTable 3: Sample quality measured by FD-dinov2 on class-conditional ImageNet (512x512) for CoSIM-L given $\\alpha=1.2$  and NFE=$2$.\n|$\\mathrm{coef}$ \\\\# of total training images | 204k| 1024k | 2048k | 4096k|\n|---|----|----|----| ----|\n|$\\mathrm{coef}=0.5$ | 309.7 | 101.3 | 69.6 | 61.5 |\n|$\\mathrm{coef}=1.0$ | 392.8 | 92.8 | 61.7 | 50.5 |\n|$\\mathrm{coef}=1.5$ | 421.9 | 95.2 | 58.6 | **49.2** |\n\nThe results show that introducing regularization (i.e., increasing $\\mathrm{coef}$) within a suitable range facilitates the learning of $f_\\psi$, which in turn improves the training of $q_\\phi$. Furthermore, since the reduction on FD-DINOv2 continues as training progresses, CoSIM demonstrates stable training across a relatively broader range of regularization coefficients, such as 1.0 and 1.5.\n\n[1] Zhou, et. al., \"Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation\", ICML 2024\n\n[2] Luo, et. al., \"Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models\", NeurIPS 2023\n\n`Q3`: The numerical performance with a one-step generation.\n\n`A3`:  CoSIM is inherantly a multi-step model, trained to model the continuous transition kernel rather than a one-step deterministic generation model like GANs. One significant advantage of multi-step generation is the ability to inject more randomness in each step, enhancing sampling diversity.\n\nFor ablation, we also evaluated one-step generation quality of CoSIM on ImageNet $512 \\times 512$ for M model.\n\nTable4: Sample quality on conditional ImageNet 512x512.\n|Model | \\# params|NFE|FD-dinov2| FID |\n|-|-|-|-|-| \n|EDM2-M-dino| 498M |63 |58.44| - |\n|Moment Matching|400M |1| - |$3.3^*$ |\n|CoSIM-M| 498M| 1 | 74.23| 3.52|\n\nNote that FID result of Moment Matching[1] is evaluated at a lower resolution 128x128. Consider the difficulty of generating in higher resolution, our method is comparable with Moment Matching[1] on one-step generation. \n\n[1]Salimans, et. al., \"Multistep distillation of diffusion models via moment matching.\" NeurIPS 2024.\n\n`Q4`: How does CoSIM integrate ... such as classifier-free guidance?\n\n`A4`: If an unconditional model $G_\\phi(x_t; t,\\emptyset)$ is used to modify the generation, the parameterization can be considered to integrate it into CoSIM's training.\n$$\nG_\\phi^{(w)}(x_t; t,c) = wG_\\phi(x_t; t,c) + (1-w)G_\\phi(x_t; t,\\emptyset).\n$$\n\nWe also noted that SiD-LSG [1] proposed a novel guidance method by using CFG in training and inferencing with different weights $w$. This method also adapts to our formulation and provides inspiration for future work.\n\n[1] Zhou, et. al., \"Long and short guidance in score identity distillation for one-step text-to-image generation.\"\n\n`Q5`: The numerical performance to the choice of time distribution? ... the weighting functions.\n\n`A5`: We give another ablation study on the hyperparameter $R$ in the time distribution $\\pi(\\gamma|r_s)$. \n\nTable5: Ablation on $R$ for CoSIM-M on class-conditional ImageNet ($512 \\times 512$) with NFE=4 by FD-DINOv2\n|$R$  \\\\# of total training images | 2M | 4M| 38M |\n|-|-|-|-| \n|$R=4$| 220.2 | 73.6 | 57.8 |\n|$R=6$| 110.4 | 55.6 | 54.7|\n|$R=8$| 95.6 | 54.7 | 55.8|\n\nDifferent $R$ gives slightly different convergence rate during early stages of training. \nBut after 38M training images, this difference is minimized.\n\nFor the weighting functions, we adopt the same weighting function as in EDM[1], this setting has been widely adopted in many works such as SiD and irrelevant to our algorithm design. Hence we keep it fixed. \n\n[1] Karras, et. al., \"Elucidating the Design Space of Diffusion-Based Generative Models\", NeurIPS 2022\n\n`Q6`: Typos\n\n`A6`: Thank you for catching these typos! We will correct them in our revision.", "claims_and_evidence": "The authors claim that HSIVI suffers from slow convergence, while CoSIM's continuous framework improves training efficiency. It is recommended to provide empirical evidence supporting this claim, such as FID/FD vs. training iterations or runtime comparisons.", "methods_and_evaluation_criteria": "The paper employs standard evaluation metrics, such as FID and FD-DINOv2, commonly used in image generation.", "theoretical_claims": "The theorems and proofs appear sound.", "experimental_designs_or_analyses": "The overall validity of the experimental design appears valid and reasonable.", "relation_to_broader_scientific_literature": "Extending the discrete-time framework (in HSIVI) to a continuous-time framework is natural, especially within the diffusion model community. However, the significant performance gain over HSIVI-SM suggests that this extension is both meaningful and interesting.", "essential_references_not_discussed": "The authors discussed essential references.", "other_strengths_and_weaknesses": "- Is the training stable given that multiple networks are trained simultaneously?\n\n- What is the numerical performance with a one-step generation?\n\n- How does the parameter $\\lambda$ affect performance? Is there an ablation study?\n\n- How does CoSIM integrate with flexible (un-)conditional generation techniques, such as classifier-free guidance?\n\n- How sensitive is the numerical performance to the choice of time distribution $\\pi(\\gamma | r_s)$? The same question applies to the weighting functions $w_i(s)$.", "other_comments_or_suggestions": "- Some notations are not defined or explained well:\n    1. What is $q_t(x_t)$ in Lines 154–155, in the right-hand column?\n     \n    2. What is $L_{SIVI-f}$? Is it a symmetric divergence: it seems like  $L_{SIVI-f}(p_\\text{parametrized}||p_\\text{target})$ and $L_{\\text{SIVI}-f}(p_\\text{parametrized}||p_\\text{target})$ are not consistent (Eq. (4), Eq. (9), and  [1])? \n\n    3. What is $p_s(x_s)$ in Eq. (9)?\n\n\n\n[1] Yu, et. al., Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration\n\n\n------\n\n**After rebuttal:** \nAlthough the extension to continuous time is relatively straightforward, I appreciate that the authors have included supplementary experiments to address my concerns regarding the empirical aspects. Thus, I raise my rating.", "questions_for_authors": "Please see the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaxKNaZGHR", "reviewer": "Reviewer_LH32", "summary": "The paper studies semi-implicit models, and proposes an extension from hierarchical semi-implicit models (hSIM) to continuous-time SIMs (cSIM). The continuous-time SIM has the advantage of being simulation-free, as opposite to hierarchical SIM; and enables multi-step sampling at the same time. This is achieved thanks to a formulation inspired from consistency models. Based on this framework, the authors propose several adaptations to the hSIM framework, e.g. a shifted regularization term. The authors analyse some theoretical properties of their method, particularly the benefits of multi-step sampling. Finally, they demonstrate high performance on Cifar10, ImageNet 64x64, and ImageNet 512x512.\n\n## update\n\nI decide to not change my rating for the unclear link between the proposed method and consistency models: \n\nThe authors claim in the rebuttal that \"our method surpasses ECT by a large margin on both FID and FD-DINOv2, details shown in Table 1 and Table 2 in paper.\". However, this is not true when looking at fixed NFE. It surpasses on FD-DINOv2, but not in FID.\nThe claim that the proposed method allows for a faster training than consistency models would require more empirical evidence, by comparing both methods in a similar setting, with same number of iterations/compute budget. I am unsure that we can draw such conclusions.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! Below are our responses to your concerns.\n\n`Q1`: I checked the experimental design ... Stein et al. “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models”, NeurIPS 2023.\n\n`A1`: Exactly! The paper “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models” from NeurIPS 2023 by Stein et al, directly proposed the metric FD-DINOv2. They pointed out that FID as a metric is unfairly favoring GAN-style models, and has limited numeric accuracy to handle modern advanced diffusion models, as they usually reach a very small FID value. This inspires us to use FD-DINOv2 in our experiments. \n\n`Q2`: As already explained in Relation to Broader Literature ... limits the technical novelty of the paper.\n\n`A2`: Here we first clarify the novelty of our method.  CoSIM is derived from the framework of continuous semi-implicit variational inference. Methods such as HSIVI [1] and SiD [2] are built upon this semi-implicit training objective to accelerate diffusion models. Theoretically, We introduce a regularization framework of the SiD loss to stabilize the training and address the pathological behavior under large alpha values as demonstrated through theorem 3.1 and the ablation study on the regularization term coefficient $\\mathrm{coef} = (1+\\lambda) \\alpha$.\n\nTable1: Class-conditional ImageNet (512x512) for CoSIM-L given **$\\alpha=2.0$**.\n|$\\mathrm{coef}$\\Iteration | 204k| 1024k | 2048k | 4096k|\n|----|----|----|----| ----| \n|$\\mathrm{coef}=0.5$ with NFE=$2$| 374.8 | 3960.0 | 4129.3 | 4039.9 |\n|$\\mathrm{coef}=2.5$ with NFE=$2$| 619.0 | 114.1  | 78.7   | 64.9   |\n\nFurthermore, by using a specific parametrization, we derive a novel training method for consistency models at a distributional level. \nUnlike traditional consistency model fixed on the reversed ODE path,\nCoSIM learns the consistency map directly, without being constrained by intermediate process.\nThrough our experiments, we show this achieves superior performance compared to the current state-of-the-art consistency models on ImageNet $512 \\times 512$, using **much less training budget** (ours 20M compared to sCD [3] 819.2M).\nTable3: Sample quality measured by FID on class-conditional ImageNet (512x512).\n|Model|#params|\\NFE|FID|\n|-|-|-|-|\n|sCD-S| 280M | 2 |2.50|\n|sCD-M| 498M | 2 |2.26|\n|sCD-L| 778M | 2 |2.04|\n|CoSIM-S| 280M| 2 | 2.66|\n|CoSIM-S| 280M| 4 | 2.56|\n|CoSIM-M| 498M|2| **1.95**|\n|CoSIM-M| 498M|4|1.93|\n|CoSIM-L| 778M|2|1.84|\n|CoSIM-L| 778M|4| **1.83**|\n\n[1] Zhou, et. al., \"Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation\", ICML 2024.\n\n[2] Yu, et. al., Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration, NeurIPS 2023\n\n[3] Lu, et al., \"Simplifying, stabilizing and scaling continuous-time consistency models.\", OpenAI\n\n`Q3`: Do you have an opinion ... Fisher divergence ... a better model.\n\n`A3`: As a novel training scheme derived from continuous semi-implicit variational inference, CoSIM differs fundamentally from traditional consistency models in the following aspects, offering distinct training advantages:\n\n- Fisher divergence does not require strictly following the reverse ODE trajectory but only learns a consistency mapping on a distributional level. This relaxes the choice on distillation path and enables various sampling strategies. It also reduces the training budget as the mapping can be more flexible. Conversely, ECT strictly follows the ODE trajectory, and its training budget is much higher than ours on comparable performance level.\n- Experimentally, our method surpasses ECT by a large margin on both FID and FD-DINOv2, details shown in Table 1 and Table 2 in paper.\n- Our approach primarily focuses on few-step sampling, which is also different from SiD, diff-instruct. We show that our approach shows significantly better performance using much less total training budget in our experiments, and scales to larger models better.\n\n`Q4`: Are semi-implicit models ... lacking in the paper.\n\n`A4`: To clarify, our contributions over semi-implicit models are:\n- We extend the framework of hierarchical semi-implicit variational inference models to the continuous settings. \n- We introduce a regularization term and give theoretical analysis on why this term could further stabilize the training. This addresses the pathological behavior of SiD loss when $\\alpha$ is large. \n- Our formulation, under a specific parameterization, reforms to a consistency training on a distributional level. This enables a more flexible mapping rather than traditional consistency training on a fixed reverse ODE trajectory. This reduces training difficulty and reduces training budget, already validated by our experiments. \n- Please refer to A2 to see that our method can surpass the performance of latest consistency models with much lower training budget.", "claims_and_evidence": "The claims made in the paper are supported by empirical and theoretical evidence. \nTheorems prove 1) the soundness of the shifted regularization term and of the training objective 2) the benefits of multi-step sampling. Empirical evidence show that the proposed method reaches SOTA-level performance on standard image benchmarks.", "methods_and_evaluation_criteria": "Method: The method is sound. The core idea is to use a consistency formulation in the setting of semi-implicit models. This resonates with ideas validated in recent works. \n\nEvaluation criteria: The method is tested on standard image datasets (CIFAR-10, ImageNet) and with standard metrics (FID, FD-DINOv2).", "theoretical_claims": "I did not verify in detail the theoretical proofs.", "experimental_designs_or_analyses": "I checked the experimental design and did detect any issue. The authors use standard benchmarks in the generative modelling literature (FID/FD on CIFAR-10 and ImageNet-64). An area of improvement would be to use more fine-grained evaluation metric, for example metrics that detect memorization and over-fitting in generative models (Stein et al. “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models”,  NeurIPS 2023).", "relation_to_broader_scientific_literature": "This paper is studying the setting of semi-implicit models. It consists in applying an idea from consistency models (Song et al., 2023) to this setting. This allows to go from hierarchical semi-implicit models to continuous-time and simulation-free semi-implicit models. However, continuous-time SIM are very close to consistency models. Most notably, the sampling algorithm is the same. However, the training differs, since continuous-SIM are based on a Fisher divergence minimization scheme.", "essential_references_not_discussed": "Not in my knowledge.", "other_strengths_and_weaknesses": "Weakness 1: As already explained in Relation to Broader Literature, the extension proposed in this framework relies on the use of a \"consistency\" formulation, which is largely adopted in the generative modelling community. This limits the technical novelty of the paper.\n\nWeakness 2:  To me, this paper asks the question: what is the best way to train a consistency model? Indeed, since your sampling is basically the same as the one of a standard consistency model, the main difference is in the training procedure. Do you have an opinion on why it would be better to rely on a Fisher divergence minimization rather than on a consistency loss? This is not clear since coSIM is not superior to ECT. There is no theoretical evidence, nor empirical evidence, that using a your training procedure leads to a better model.\n\nOverall, the contribution is not clear to me. The proposed method seems to improve semi-implicit models with the consistent formulation. But are semi-implicit models needed for high-quality few-step generation? Why would this framework be more interesting than standard consistency models? This discussion is currently lacking in the paper.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper unifies Hierarchical Semi-Implicit Variational Inference (H-SIVI) and Score identity Distillation (SiD) to construct a scaled Fisher divergence. Minimizing this divergence is achieved through an alternating optimization scheme that switches between optimizing a generator (given the score network) and optimizing the fake score network (given the generator).  While the generator loss resembles that of SiD, it diverges from SiD, which targets one-step generation, by optimizing for multi-step generation. Moreover, an additional regularization term, controlled by a tunable parameter $\lambda$, is introduced into the loss of the fake score network.

The paper refers to this continuous-time optimization framework as Continuous Semi-Implicit Models (CoSIM). CoSIM enables the same model to be flexibly applied across different numbers of sampling steps, as opposed to training a separate model for a fixed number of inference steps. Empirical results demonstrate improved performance when increasing the number of steps from 2 to 4, with the 4-step generation achieving competitive FID scores and SoTA results in FD-DINO_v2. These findings are supported by both detailed theoretical analysis and convincing experimental results on CIFAR10, ImageNet 64x64, and ImageNet 512x512.

The AC considers the paper above the acceptance threshold given the amount of original theoretical and methodological contributions. However, the AC recommends that the authors revise the paper to better position the paper in the literature:

A primary concern raised by reviewers is the paper's positioning of CoSIM relative to consistency models, and the perception that it attempts to present CoSIM as a superior alternative. While the concern of one reviewer was addressed by the authors’ promise to revise the description of consistency models, another reviewer remained cautious regarding this comparison. The final scores remained at 3-3-3-1. The AC found the authors’ rebuttal, particularly the comparison between sCD and CoSIM, sufficient to alleviate these concerns and opted to follow the majority recommendation to accept the paper.

The AC agrees with the authors’ plan to revise the paper to better position CoSIM in relation to consistency models. In addition, based on their own reading of the submission, the AC suggests the following additional revisions:



1. Report CoSIM's one-step generation results. While CoSIM is designed for arbitrary-step inference and may underperform in the one-step setting compared to methods tailored for it, including these results would provide a more comprehensive understanding of CoSIM’s strengths and limitations.

2. Report CoSIM's generation results beyond 4 steps. This will help understand how much room is left for improvement when the number of inference steps is further increased.

3. Add FID comparisons reported in the rebuttal, and add comparisons with SiD on ImageNet 512×512. While the original SiD paper did not include results on ImageNet 512x512, the FID and FD-DINO_v2 results of SiD were later reported in: "Zhou, M., Zheng, H., Gu, Y., Wang, Z., & Huang, H. (2025). Adversarial Score Identity Distillation: Rapidly Surpassing the Teacher in One Step. ICLR 2025." Although per ICML reviewing guidelines, this paper was not considered in the decision-making process, its reported results offer useful reference points for evaluating CoSIM against previous baselines on ImageNet 512×512.
Decision: Accept (poster)