A Likelihood Based Approach to Distribution Regression Using Conditional  Deep Generative Models
Authors: Shivam Kumar
Yun Yang
Lizhen Lin
Venue: ICML 2025 poster (2025)
Keywords: Distribution Regression; Conditional Deep Generative Models; Intrinsic Manifold Structure; Sieve MLE; Wasserstein Convergence.
Primary area: deep_learning->theory
TL;DR: We study the theoretical properties of conditional deep generative models under the framework of distribution regression where the response variable lies in a high-dimensional ambient space but concentrates around a lower-dimensional manifold.

Abstract: In this work, we explore the theoretical properties of conditional deep generative models under the statistical framework of distribution regression where the response variable lies in a high-dimensional ambient space but concentrates around a potentially lower-dimensional manifold. More specifically, we study the large-sample properties of a likelihood-based approach for estimating these models. Our results lead to the convergence rate of a sieve maximum likelihood estimator (MLE) for estimating the conditional distribution (and its devolved counterpart) of the response given predictors in the Hellinger (Wasserstein) metric. 
Our rates depend solely on the intrinsic dimension and smoothness of the true conditional distribution. These findings provide an explanation of why conditional deep generative models can circumvent the curse of dimensionality from the perspective of statistical foundations and demonstrate that they can learn a broader class of nearly singular conditional distributions. Our analysis also emphasizes the importance of introducing a small noise perturbation to the data when they are supported sufficiently close to a manifold. Finally, in our numerical studies, we demonstrate the effective implementation of the proposed approach using both synthetic and real-world datasets, which also provide complementary validation to our theoretical findings.

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Shivam Kumar'
Yun Yang² Lizhen Lin
Abstract
1. Introduction
Conditional distribution estimation provides principled framework for characterizing the dependence relationship between response variable Y and predictors X, with the primary goal of estimating the distribution of Y conditional on X through learning the (conditional) data-generating process. Conditional distribution estimation allows one to regress the entire distribution of Y on X, which provides much richer information than the traditional mean regres- sion and plays central role in various important areas ranging from causal inference (Pearl, 2009; Spirtes, 2010), graphical models (Jordan, 1999; Koller & Friedman, 2009), representation learning (Bengio et al., 2013), dimension re- duction Gaareira-Perpinaae 1997; Van Der Maaten et al., 2009), to model selection (Claeskens Hjort, 2008; Ando, 2010). Their applications span across diverse domains such forecasting (Gneiting & Katzfuss, 2014), biology (Kr- ishnaswamy et al., 2014), energy (Jeon & Taylor, 2012), astronomy (Zhao et al., 2021), and industrial engineering (Simar & Wilson, 2015), among others.
In this work, we explore the theoretical proper- ties of conditional deep generative models un- der the statistical framework of distribution re- gression where the response variable lies in high-dimensional ambient space but concentrates around potentially power-dimensional manifold. More specifically, we study the large-sample prop- erties of a likelihood- brelihood-bbsed approach for estimat- ing these models. Our results lead the conver- gence rate of a sieve maximum likelihood estima- tor (MLE) for estimating the conditional distribu- tion (and its devolved counterpart) of the response given predictors in the Hellinger (Wasserstein) metric. Our rates depend solely on the intrin- sic dimension and smoothness of the true con- ditional distribution. These findings provide an explanation of why conditional deep generative models can circumvent the curse of dimensional- ity from the perspective of statistical foundations and demonstrate that they can learn broader class of nearly singular conditional distributions. Our analysis also emphasizes the importance of introducing a small noise perturbation to the data when they are supported sufficiently close to a manifold. Finally, in our numerical studies, we demonstrate the effective implementation of the proposed approach using both synthetic and real- world datasets, which also provide complemen- tary validation to our theoretical findings.
There is a rich literature in statistics and machine learning on conditional distribution estimation including both fre- quentist and Bayesian methods (Hall & Yao, 2005; Norets & Pati, 2017). Traditional methods, however, suffer from the curse of dimensionality and often struggle to adapt to the intricacies of modern data types such as the ones with lower-dimensional manifold structures.
Recent methodologies that leverage deep generative mod- els have demonstrated significant advancements complex data generation. Instead of explicitly modeling the data distribution, these approaches implicitly estimate it through learning the corresponding data sampling scheme. Com- monly, these implicit distribution estimation approaches can be broadly categorized into three types. The first one is likelihood-basee with notable examples including Kingma & Welling (2013), Rezende et al. (2014), Burda et al. (2015), and Song et al. (2021). The second approach, based on ad- versarial learning, matches the empirical distribution of the data with distribution estimator using an adversarial loss. Representative examples include Goodfellow et al. (2014), Arjovsky et al. (2017), and Mrouch et al. (2017), among others. The third approach, which is more recent, reduces
'Department of Applied and Computational Mathematics and Statistics, University of Notre Dame. Notre Dame, USA
Deppatmennt of Mathematics, University of Maryland, Col- lege Park, USA. Correspondence to: Shivam Kumar <sku- mar9@ edu..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
the problem of distribution estimation score estimation through certain time-discrete or continuous dynamical sys- tems. The idea of score matching was first proposed in Hyvärinen & Dayan (2005) and Vincent (2011). More re- cently, score-based diffusion models have achieved state-of- the-art performance in many applications
model, considering full-dimensional noise and the poten- tial presence of singular underlying support. We provide solid statistical foundation for the approach by prov- ing the near-optimal convergence rates for this proposed estimator.
We derive the convergence rates for the conditional den- sity estimator of the corrupted data Y with respect to the Hellinger distance and specialize the obtained rate for two popular deep neural network classes: the sparse and fully connected network classes. Furthermore, we characterize the Wasserstein convergence rates for the induced intrinsic conditional distribution estimator on the manifold (i.e., a deconvolution problem). Both rates turn out to depend only on the intrinsic dimension and smoothness of the true conditional distribution.
et al., 2015; Nichol & Dhariwal, 2021; Song et al., 2020; Lipman al., 2022; Brehmer & Cranmer, 2020; Han al., 2022).
On the theoretical front, recent works such as Liu et al. (2021), Chae et al. (2023), Altekrüger et al. (2023), Stanczuk et al. (2024), Pidstrigach (2022), and Tang & Yang (2023) demonstrate that distribution estimation based on deep gen- erative models can adapt to the intrinsic geometry of the data, with convergence rates dependent on the intrinsic dimen- sion of the data, thus potentially circumventing the curse of dimensionality. Such advancement has naturally motivated us to employ and investigate conditional deep generative model for conditional distribution estimation. Specifically, we explore and study the theoretical properties of a new likelihood-basee approach to conditional sampling using deep generative models for data potentially residing on low-dimensional manifold corrupted by full-dimensional noise. More concretely, we consider the following condi- tional distributional regression problem:
Our analysis in Corollary 2 suggests the need to inject small amount of noise into the data when they are suffi- ciently close to the manifold. Intuitively, this observation validates the underlying structural challenges in related manifold estimation problems with noisy data, as outlined by Genovese et al. (2012).
We show that the class of learnable (conditional) distribu- tions of our method is broad. It encompasses not only the smooth distributions class, but also extends to the general (nearly) singular distributions with manifold structures, with minimal assumptions.
YJX = VIX + E,
where X serves as a predictor in IP,, VIX represents the (uncorrupted) underlying response supported a mani- fold of dimension d < D, Y|X represents the observed response, and E ~ (O,, TD) denotes the noise residing in the ambient space P Our deep generative model focuses on the conditional distribution V|X by using a (conditional) generator of the form (Z, X), where G. a function of a random seed Z and the covariate information X. This approach termed 'conditional deep generative' because the conditional generator is modeled using deep neural net- works (DNNs). Observe that, when d < D, the distribution of G. (2, is supported on lower-dimensional manifold, making it singular with respect to the Lebesgue measure in the D-dimensional ambient space. We study the statistical convergence rate of sieve MLEs in the conditional deep general model setup and investigate its dependence on the intrinsic dimension, structure properties of the model as well as the noise level of the data.
1.2. Other relevant literature
The problem of non- non-parametric conditional density estima- tion has been extensively explored in statistical literature. Hall & Yao (2005), Bott & Kohler (2017), and Bilodeau et al. (2023) directly tackle this problem with smoothing and local polynomial-basee methods. Fan & Yim (2004) and Efromovich (2007) explore suitably transformed regres- sion problems to address this challenge. Other notable ap- proaches include the nearest neighbor method (Izbicki et al., 2020; Bhattacharya & Gangopadhyay, 1990), basis function expansion (Sugiyama et al., 2010; Izbicki & Lee, 2016), tree-based boosting (Pospisil & Lee, 2018; Gao & Hastie, 2022), and Bayesian optimal transport flow Chemseddine et al. (2024) among others.
In the context of conditional generation, we highlight re- cent work by Zhou et al. (2022) and Liu et al. (2021). In Zhou et al. (2022), GANs were employed to investigate conditional density estimation. While this work offers a consistent estimator, lacks statistical rates or convergence analysis, and its focus is on a low-dimensional setup. In Liu et al. (2021), conditional density estimation supported on a manifold using Wasserstein GANs was examined. How- ever, their setup does not account for smoothness across either covariates or responses, nor do they address how deep generative models specifically tackle the challenges of high-
1.1. List of contributions
We briefly summarise the main contributions made in this paper.
To the best of our knowledge, our study is the first at- tempt to explore the likelihood- basedd approach for distri- butional regression using a conditional deep generative
2

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
dimensionality. Moreover, their assumption that the data lies exactly on the manifold can be restrictive. Our study shares some commonalities with the work of Chae et al. (2023), as both investigate sieve maximum likelihood estimators (MLEs). However, the fundamental problems addressed and the methodologies employed differ significantly, and our work involves technical challenges that span multiple scales. While Chae et al. (2023) concentrates exclusively on unconditional distribution estimation, our theoretical analy- sis necessitates much more nuanced techniques due to the conditional nature of our setup. This shift is noteworthy be- cause it demands more refined analysis of entropy bounds, considering two potential sources of smoothness across the regressor and the response variables. Furthermore, our setting accommodates the possibility of an infinite number of x values, which gives rise to a dynamic manifold struc- ture, further compounding the intricacy of the problem at hand.
Pa =x) can thus be expressed as:
ps p=(()z)= f G. ( dP2.
where do. is the density of N(O, N(0, We define the class of conditional distributions P as
p = Pg.o g(; E F, [Tmins Fmaxx] }
(3)
where Pa represents the distribution with density Pg. f O (y - (2, (222(()2 xd)2 In this notation, P. = Pc.... and P. = o The elements of P comprise two components: 9 originating from the underlying function class F, and o, which characterizes the noise component. This class enables us to obtain separate estimates for G, and o+, furnishing us with both the canonical estimator for the distribution of Y|X and enhancing comprehension of the singular distribution of G. .), supported on a low-dimensional manifold.
2. Conditional deep generative models for distribution regression
Given a data set {(x,, Y,) the log-likelihodd function is defined as (g 0) = n Σ log Pg.o (Y.) For a sequence The $ as n + oc, a sieve maximum likelihood estimator (MLE) (Geman & Hwang, 1982) is any estimator (g. σ) E F x [oin maxx] that satisfies
We consider the following probabilistic conditional genera- tive model, where for given predictor value x, the response Y is generated by
en (T,5) N
sup (n(g, - 17n-
(4)
Y +, ZEXCR, R
(2)
eElemin Smaxl gEF
Here 9 E F and ē E [Tminn max) are the estimators, and s) represents the optimization error. The dependence of Ò and ā on illustrates the sieve's role in approximating the true distribution when optimization is performed over the class P. The estimated density y = Pğ.a provides an estimator for p. (-I),, and Q3(-IX = x) serve the estimator for Q. = z).
Here, 2.(,)): : 3 M2 is the unknown generator func- tion, Z a latent variable with known distribution Pz and support Z c Rº independent of the predictor X. The ex- istence of the generator G. directly follows from Noise Outsourcing Lemma 3. This lemma enables the transfer of randomness into the covariate and an orthogonal (inde- pendent) component through generating function for any regression response. We denote M Mz C R as the support of the image of G2, X) such as a (union of) d-dimensional manifold. We model G, (.,-) Z x X IR x IRP + y C TR." using a deep neural network, leading to conditional deep generative model for (2).
In this section, we formulate the main results, which provide convergence rates in the Hellinger distance for our sieve MLE estimator. The convergence rate was derived for any search functional class F, with a brief emphasis on their entropy and approximation capabilities.
Assumption (True distribution). Denote fx((x) as the distribution of X. We denote the true conditional densities as p. {p- (-x),, E [P]. It is natural to assume that the data is generated from p- from model (2) with some true generator G. and o.. We denote Qx = x) (or QG. as the distribution of G. (7, x) for some distribution Pz-
In the next section, we present a more general result in terms of the entropy bound (variance) for the true function class of G. and the approximability (bias) of the search class. We then proceed to simplified understanding in the context of conditional deep generative models in subsequent sections.
2.1. Convergence rates of the Sieve MLE
A function g is said to have a composite structure (Schmidt- Hieber, 2020; Kohler & Langer, 2021) if it takes the form as
In light of equation (2), it is evident that the distribution of Y|X = x results from the convolution of two distinct dis- tributions: the pushforward of Z through G. with X = x, and e following an independent D D-dimensional normal dis- tribution. The density corresponding to the true distribution
9= Sq fq o.....
(5)
where fs : (aj, bz)d + (as+1, (1))) do = + d
and dg41 = D. Denote f, (1)) ...., {(d (dd11)) as
3

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
the components of Jj. let ig be the maximal number of variables on which each of the f depends and let ? E 74 b;),, K) (see Section 2.4.1 for the def- inition of the Hölder class ).)) A composite structure is very general which includes smooth functions and ad- ditive structure as special cases. In addition, in the next section, we show the class of conditional distributions (Qa. (-IX- = x)= x E RP, G. E induced by the com- posite structure is broad.
Given two conditional densities pı Pi(- P2((2x) and pi denoting the density of X, we use integrated distances for measure of evaluation. With slight abuse of no- tation, we denote d1 (P1. = Ex [dl (p1 P2(-[=))] (e(( where di
and di = Ex (pr (( (=)x)) and dh represent the L1 and the Hellinger distance as di (pi ())))) = I (v(()))) p2(y(=>) dy and d (PI, P2) = Pl tively. Denote (6, F, d) and M (6, F. d) as covering and bracketing numbers of the function class F with respect to the (pseudo) -etric d.
p2(y]=]]] ( respc-
Assumption 2 (composite structure). Denote G =
G (q. d, K)) as a collection of functions of form (5), where = (do...., ,.,,,))),, t = (to...., 14+1), and B = (Ba..... .........997 ..411). We regard (q, d, L. B, K) as constants in our setup, and assume that the true generator Gx(,z) as
We first present Lemma 1, which establishes the bracketing entropy of the functional class P with respect to Hellinger distance in terms of the covering entropy of the search class F. This enables us to transfer the entropy control of the individual components F and a to the entire P.
in (2) belongs to for all X. Additionally, we assume oo S K.
Let F be class of functions from Z x X to Lemma 1.
RD such that |elg|solle I K for every g E F. Let P {Pa. e FF o E moin Tmax]] with min I 1. Then, there exist constants
3, Bj II (3 J. argmax 4 1=j+1
(E(O.....) Bj
c(Omax: K, D) and C
B. By, to =j.
C(omax: K, D) and 6, (D) such that for every d (0.6.).
The quantities t. and Ba are called intrinsic dimension and smoothness of G. (or of g).
ddd)) & log.N(oo ((o0 F.,,8
F.II bo)l + log
(6)
Remark 1 (Strength of the Composite Structure). The ex- pression (ay, by) C [-K, K] can be intuitively visualized by setting aj = - K and by- = K. To illustrate the impuct of intrinsic dimensionality and smoothness, consider a func- tion f: .R. + R.. defined as fxx fi f((x)) +.... + fa(za), where (#1,..., Id) and f, E (M((K, K), for j = 1....,d d. While f E ((-K, KY², K), its intrin- sic dimension is to = 1 with intrinsic smoothness B. This mitigates the curse of dimensionality.
The proof of Lemma is provided in the Appendix E. The- orem I presents the convergence rate of the sieve-MLE to the true distribution (see Appendix F for the proof).
K,
Theorem 1. Let F,P, min and 8_
(D) be given as in Lemma 1, and n > 1. Suppose that log N (5, F. lcolloo) S & ([++1V log 8-1) for every 8 E (0,5.) and some A,E > O. Suppose that there exists a G E F and some δapprox E =(0,6, 60] such that ||| - bollo S Sapprox. Furthermore, suppose that Z 1, A Jmin S dapprox S and a. E [o maax].
Example Oine-dimensoonal 3-Hölder Generator). Let U ~ Unif(O, 1) and define G(u)) = 1/, u E [0, 1]. Then X = G(U) has density
Then
P. (dH > En, < 50 -insn + +C2n
d dr
(P(Usx))=
x E [0, 1],
provided that nn S ne?? /6 and en S 28.,, where log (n/ (nn/omia) (V 5(A
which belongs to the B-Hölder class on [0, 11. In our nota- tion one checks to = 3, = B, and setting 3 = 1 recovers (n, 1) case and thus provides fully explicit illustration of Assumption 2.
V Sapprox o.
6f Cs
(8)
n
C is an absolute constant, C2 = C2(D) and Cs Ca(D, K, mmax).
=
Assumption 3. Let M, be the closure of G2, We assume that M. does not have an interior point, and reach(M.)) r. with r. >0. O.
The outlined rate has two components: the statistical com- ponent, expressed as an upper bound to the metric entropy of F. and the approximation component, denoted as Sapprox. The statistical error is quantified by measuring the com- plexity of the class P, as formulated in Lemma 1. The approximation error is assessed through the ability of the provided function class to approximate the true distribution.
Assumption 2 permits low intrinsic dimensionality within the learnable function class. Assumption 3 imposes the strong identifiability condition necessary for efficient esti- mation, as seen in manifold literature (Aamari & Levrard, 2019; Tang& & Yang, 2023).
4

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
2.2. Neural network class
The approximation properties of the sparse and fully con- nected network are provided in Lemma 4.1 and Lemma 4.2 of the Appendix K, respectively.
We model (: using a deep neural network. More specif- ically, we parameterize the true generator G. with deep neural neural architecture (L,r) of the form
Having established the essential components for Fe in (11) and Lemma 4.2, and for Fs in (10) and Lemma 4.1, re- spectively, we can simplify Theorem and state Corollary 1.
f:R" R 2 f(z) WEPUL S.P.,WW-10 W2-1 WiP,, Wos, (9)
where W E R Uj E RR'S Poy(() (-) ReL.U(- M3) and r = (ro....., r2+11 E N The constant L is the number of hidden layers and r- = .o....., represents
Corollary 1. Suppose that Assumptions and 2 hold, and a. E [Amin; m with m S and max < cc. Moreover, assume that the noise a. decays at rate a, i.e., G, n nn " and m =2 for some y a 0. Then, for every ōapprox [0, 1}, the following holds:
the number of nodes in each layer.
We define the sparse neural architecture class F. s,, as set of functions of form (9) sat- isfying
Let F, = FF, (L, B, K) with 6_ = =.( be as given in Lemma 1, and I. x log 8 d approxx v = a-t///
maxessst
S B. Ejs [W 19ylo s., III sls & K.
s x Supprox log aapprox B x Sapprox. Then the sieve MLE Do satisfies (7) with En as in (8) with & = óapprox log" 2og'(5 (Sapprox) and A ( provided that 1)n I ne
with ro- = d p and rI++1 D. where |. and bmo stand for the L and - vector norms, and lo = SuPSERro max;=1...... [f(()) is sparsity parameter and K is functional bound.
8-t-/8.
2 /6 and E, S 25...
2. Let Fo - Fe B,L, K) with 8. 8D( be as given in Lemma I, and T. x log 8-1 approx: r = 2,23.
The fully connected neural architecture class Fe Fe (L,r, B, K) is set of functions of form (9) satisfying
B x 8xpprox- Then the sieve MLE Pe satisfies (7) with e as in (8) with E ōuppeux log" (Sxpprox) and A = log 2 (5 (8approx) provided that Ta S ne /6 and = S 28..
max WWo V 10ylo0 B, 0595
ll s K.
Both classes F, and Fe for the deep generator will be con- sidered in our analysis of the resulting sieve maximum like- lihood estimator. We denote the corresponding sieve-MLE as Ps and Pe, respectively. When use instead of r, it refers to TL+1 = D.
In particular, choosing dapprox == (o2/n) (02//))..10001 3.1(20.4..) mini- mizes S E log (n//min)] /n 8approx and gives
TI. = r along with ro
and
de 23. log" (n).
(12)
We can simplify and visualize the result stated in Theorem in both cases: when the sieve-MLE obtained with opti- mization performed over the class F, and F... To fulfill the conditions stated in the Theorem 1, we need to establish entropy bounds for these function classes, F. and Fe, and gain insight into their approximation capabilities for the composite structure class described in Assumption 2.
Remark 2. The convergence rate in (12) illustrates the influ- ence of intrinsic dimensionalliyy smoothness, and noise level on the estimation process. Note that a upper bounded as e S V28 (D). For large values of a, estimation of G. is inherent difficult the data is very close on the sin- gular support. To address this, a small noise injection, as described in Corollary 2, can smooth the estimation and ensure consistency.
For the sparse neural architecture class F, r, s, K), the entropy, formally stated as Proposition in Ohn & Kim (2019), bounded as follows.
The proof of Corollary 1 is provided in Appendix G. For the composite structural class G, the effective smoothness is denoted by B., and the dimension is l,, This effectively mitigates the curse of dimensionality. The convergence rate at (12) also recovers the optimal rate when q and a 0. and there is small lag of polynomial factor t,, a/ + t..) when a > O (Norets & Pati, 2017). This lag arises due to the presence of full-dimensional noise in the response observation Y. Note that when the noise is small, that is a is large, achieving a sharp estimation of p. requires an equally accurate estimate of G.. This can be quite challenging.
log (8, III- lool S sL (log( + log 8 }
(10) From an entropy perspective, the fully connected neural architecture class F,,,, B, K) can be viewed as F, with- out any sparsity constraint, meaning s x 2T,, Therefore, we have
log.N (6, Fer fll 160 $ (log((L) + log 8 811..
(11)
5

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Our practically tractable approach attempts to address this without initially estimating the singular support.
parameters specified in Corollary 1, the sieve MLE Pper and Qper based on the perturbed data ([y,xj)]3 satisfies
2.3. Wasserstein convergence of the intrinsic (conditional) distributions
P. [W: (Qper. Q.) 2 (ea voottoolg
x 5e Seneh
where en can be chosen such that
Using Wasserstein distance metric for distributions Qg is meaningful due their singularity in ambient space: when d- < D, the conditional distribution is singular with respect to the Lebesgue measure on R.P.
[n Mo (n), ( (n-méee log (og).
fa < 9.//2 (11/220,+000 +c,), otherwise.
c t ++o./oo((ee))-)) x
(13)
The integrated Wasserstein distance, for r 2 1, between P.(-IX) and (2)) is defined as
2.4. Characterization of the learnable distribution class
Section 2.2 focuses on the true generator Gs within the class of functions with composite structures. In this subsection, we show that such a conditional distribution class achieved by the push-forward map G. is broad and includes many existing distribution classes for Q, as special cases.
W, (P,, Pa) = Bx [if (Bug, [IU, - "1]
where 12) is the set of all couplings between Pi and Po that preserves the two marginals. The (dual) representation of this norm, (,(Pi, P2) =
Ex suPpn/lu,,, [E] Epgl/1]] (Villani et al.,
2.4.1. 1. SMOOTH CONDITIONAL DENSITY
2009) with ||- I|Lip, denoting the r-Lipschitz norm, is par- ticularly useful in our proofs.
For 3 > 0, let 11 (D, be the class of all 3-Hölder func- tions f D C R + IR with 3-Hölder norm bounded by M O. Let HD) = UM>o0H (D, M). See Appendix B for their formal definitions.
Theorem 2. Suppose that Assumption 3 holds. If
dh (Pg.o. Pa) S E holds for some E E [0, and some Pg.a E Pp then we have
Lemma 2. Suppose that (i) Z X and y are uniformly
convex and (ii) pz E (2), 1'x E HH (X) and q, E H8a ())) for some 32, 3x, BQ > 0 and are bounded above and below. Then, there exists a map g(..)) 2 x X + y
Wi(Og, Q+) Q+) SO ( Vooge-)))
where C. = C(D, K,,,) depends only on (D, r..).
such that (-+-) = Qg and g E H8mm + 8 x X), where iiin = min ( 8x. (.).
The proof of Theorem 2 is provided in Appendix H. Theo- rem 2 guarantees that Wi (Qg. Q.) Stog d (P.p.) + d.s where Stog represents less than or equal up a logarith- mic factor of n. Following from Corollary 1, the Wasser- log?(() (n) V o- log" (o)) comprises two components: the convergence rate in the Hellinger distance and the standard deviation of the true noise sequence. It is noteworthy that the first ex- pression is influenced by the variance of noise by the factor a. When a is very small, indicating that the data Y, lies very close to the manifold, the second expression n in the overall rate dominates. Intuitively, this phenomenon arises from the underlying structural challenges in related manifold estimation problems with noisy data, as discussed by Genovese et al. (2012). To address this issue, we pro- pose a data perturbation strategy by transforming the data {(Y,, x, into ((Y,, X,) X,)) where Y, = +,+ and Cj ~ N (0D, n -.. (._ ID). The resulting estimation error bound is summarized below, whose proof is provided in Appendix I.
Lemma 2 establishes that the learnable distribution class in- cludes Hölder-smooth functions with smoothness parameter 9min and intrinsic dimension d. As a result, following Corol- lary 1, the convergence rate for density estimation is given by S x n is a transport map between two distributions. The well- established regularity theory of transport map in optimal transport is directly applicable here [see Villani al. (2009) and Villani (2021)). The proof of Lemma 2 is based on The- orem 12.50 of (Villani et al., 2009) and Caffarelli (1996), which establishes the regularity of this transport map and its existence follows from Brenier (1991). When pz is selected as a well-behaved parametric distribution, the regularity of the transport map is determined by the smoothness of both n and Q.. For more detailed discussion on this, please refer to Appendix C.
stein convergence rate, n t.-
+2+0), A push-forward map
2.4.2. A BROADER CONDITIONAL DISTRIBUTION CLASS WITH SMOOTIIEES DISPARITY
Corollary 2. Suppose that Assumption 1, 2, and 3 hold, and o. E [amin mmax] with o. = n and m = n for some 0 S a S. Then for each of the network architec- ture classes (sparse and fully connected) with the network
In Appendix L, we present a novel approximation result for the function class exhibiting smoothness disparity in Theorem 5. This new result facilitates the study of theo- retical properties of estimators when the generator G.
6

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Hopp (2,X, K). Note that such a function class defined in (16) in Appendix I. is much broader compared the smoothness class in Section 2.4.1 as Z and X do not have to be jointly smooth and it allows for smoothness dispar- ity among them. The subsequent Theorem 3 combines our approximation result with (1I) and enables us to specialize Theorem to this class (see Appendix J for the proof).
number of charts K is finite. Analogous to the single chart scenario, we first construct Ok o Sf transport the measure each chart. We then patch these local transport maps together to construct a global transport map; see Appendix D for full details. As a result, following Corollary 1, the convergence rate for density estimation shall be given by Ex x n (9min (00/iiuDaa//C0 (20mitm)))))
Theorem 3. Let G. E H8 K2) Suppose that Assumption 1 holds and 0x E [m maax] with miin S and max < xc. Moreover, we assume 0. n and miin n for some 0 S a S y S (3z p)-p)-1 Then, for every Ōapprox E [0.1], we have: Let F. Fe ()) with L x log d apppeox T X p) 0-87'0007
3. Numerical Results
In this section, we present numerical experiments to vali- date and complement our theoretical findings using two syn- thetic dataset examples. These experiments cover a range of scenarios, including full-dimensional cases as well as benchmark examples involving manifold-based data, Ad- ditionally, we provide real data example to further enrich our experimentation and validation process. It is worth not- ing that, although not significant, the computational cost of fitting a conditional generative model is higher compared to fitting an unconditional one, as the input dimension of the deep neural network (DNN) is p + d rather than just d.
log 5 8mpprox. Then the sieve MIFE Pr satisfies (7) with the rate outlined in (8) with E = =Spppe 10+3x'0 p) log" 8- Sapprox and A = log" 2 8-1 apprrox provided that Th I nett /6. In particular, choosing b+9x'p) 1/(-02204052') S 1 minimizes en" x Japprox == (02//) 1/(2+
187''+07'1 p) Sapprox
s s= Ōapprox
VE(A+ + log (n//mmi)) /n Sapprox/ / and gives
1-0(87] b+og""n ')
n 2407 0+0x'' p log (n).
(14)
Learning algorithm to compute sieve MLE. For the computational algorithm, we adopt a common conditional variational auto-encoder (VAE) archi- tecture to maximize the following log- ligeliielihood term: term> Ej- LVAB(O, 0, db; Yg, Xy), where
The proof of Theorem 3 is provided in Appendix J. In the special case when = 0 and d D, our convergence rate in (14) recovers the minimax optimal rate for conditional density estimation based on kernel smoothing, as established in (Li et al., 2022).
(Pg.o (Pg.o y. 9s 2 y, x)
(, log
2.4.3. CONDITIONAL DISTRIBUTION ON MANIFOLDS
The variational distribution g0 (Zly, is chosen as the stan- dard normal family N( x), Ey,y,
In this part, we extend Lemma 2 and provide the existence of the generator when the conditional distribution is supported on compact manifold with dimension d., <. D. Due to space constraints, we provide only a sketched proof here; the detailed proof can be found in Appendix D. Specifically, we first present arguments for the existence of the generator when y is covered by a single chart. We then extend this to the multiple chart case using the technique of partition of unity.
We examine two classes of datasets: (i) full-dimensional response and (ii) response residing on a low-dimensional manifold. The first highlights the generality of our proposed approach, while the second underscores its efficiency in terms of the Wasserstein metric and validates the small noise perturbation strategy outlined in Corollary 2.
Simulation from full dimension distribution. We use the following models for data generation.
In the simpler case when there exists single ()) cover-
ing V/ where yp : (0..) + ) is a homeomorphism. we
assume & E 71 In this case, we use the change of variable formula to transfer the measure on B₁ (Oa.) (unit
FD1 : Y =
((U N (-X,0.22)) + [uSo..; N (X, 0.25 ~ Unif(O, 1), X ~ N(3, 1).
ball in Rd.) from V/. Following Lemma 2, we can find a transport map g E Hisi mapping from Z x X' to B1 (0..). The map g o to then serves our generator.
FD2 : Y = X? + (Xx+Xs/33) + sin(X4 + Xs) +
(x,)3-1 sidd N(O, 1), -~ 1).
FD3: (X +Xa- X5 +0. (1 (1+ N(O, 1).
In the general case where the compact manifold y needs to be covered by multiple charts, demonstrating the existence of a transport or push-forward map is challenging because y is not uniformly convex. Suppose that {Ue, { forms a cover of y. Due to the compactness of V, the
x3) x (X,} 15 i.dd
These are examples of a mixture model, an additive noise model, and a multiplicative noise model, respectively. The
7

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
neural architecture for both the encoder and decoder consists of two deep layers, i.e., L = 2. The hyperparameters are as follows: Tenc (p 1, 10, 10) for μs and Eg, and Tdee (10 + p. 10, 1) for The sample size used for
100, 2) for g. We utilized sample size ddec (2 p, 100,100, of 5000 for simulation, with training-to-testing ratio of
4 1. A batch size of 100 was employed, with a learning rate of 10-3
simulation is 5000, with a training-to-testingg ratio of 4 : We employ batch size of 64 with a learning rate of 10
1.0
375
We compare the sieve MLE with CKDE (Hall et al., 2004) and FlexCode proposed by Izbicki & Lee (2017). To evalu- ate their performance, we compute the mean squared error (MSE) for both the mean and the standard deviation. We use Monte Carlo approximation to compute the mean and standard deviation for the sieve MLE, and numerical inte- gration for CKDE and Flexcode. This evaluation strategy resembles that implemented by Zhou et al. (2022). Table 2 summarizes the findings.
Plae Doal
44
209
43-
Figure 1. Generated samples from manifold M1 and M2 are dis- played.
Table MSE for the estimated conditional mean and the standard deviation.
Thorox
Bipeo
Sieve MLE
CKDE
FlexCode
FDI
MEAN
0.0379 0.0170
1.0053 0.1004
1.1660 :: 0.1076
SD
0,0280 = 0.0045
0.9887 0.0347
1,2000 = 0.0126
Wasseraent
FD2
MEAN
0.1943 = 0.0427
0,2640 4: 0.0515
0,3954 + 0.0571
SD
0.2843 0.0093
0,2853 0.0213
5.8278 = 0.1607
FD3
MEAN
0.2337 = 0.0453
0.2967 + 0.0537
1.3419 = 0. 1087
SD
0.0861
0.0460
11.4898
0.1559
Note that the sieve MILE outperforms all other methods in all scenarios except for the MSE(SD) for the FD3 dataset. How- ever, for the FD3 dataset, we found that as the training ple size increases further, the MSE(SD) of the sieve MIE achieves performance increasingly comparable to CKDE.
di 38
1 No deldtct
Figure 2. Box plots for the empirical Wasserstein distance at dif- ferent noise levels a...
We computed the empirical W, distance using the algorithm proposed by Cuturi (2013) to evaluate the performance, Figure 2 presents the boxplots of W1 between the true and learned distribution for M1 and M2 across 20 repetitions. The left panel highlights the following general behaviors:
Simulation from distributions on manifolds. We consider two examples of manifolds with an intrinsic dimension d 1, while the ambient dimension D 2.
MI Y G.(ZUU) + E, G. -GG." .!!)).
a( Iuvooss (1- c2s( + 4u>0.s) cos (Z),
When a is small and close to zero, the noise variance is large, making estimation challenging due to the singular- ity of the true data distribution.
d! Tiucas) (0.5 sin 7 [(U>0.5) sin ());
Unif(0, Uni
M2 Y G.(Z,U) e, G. (Gl)) _.. (c!))...)))
When a is large, the noise variance is small, and the perturbed data facilitates efficient estimation.
cm = Louc (2) + [ coss 2). c(2)
[or5 0.5 0.5 sin (2) +) (U00.)) sin Z); Z ~ Unif (0, (0,2m) 2T),
This observed pattern, as emphasized in Corollary 2, closely aligns with the results achieved in (13). An additional nu- merical experiment on real data has been performed and can be found in Appendix A.1.
U ~ (0,
The manifold Mj consists of two moons. The mani- fold M2 comprises ellipses, with conditions distinguish- ing the inner and outer confocal ellipses. The noise se- quence follows a two-dimensional centered Gaussian distri- bution, & ~ N(02,0??2)) I2) We investigated this setup across various noise variances o2. Our neural architecture em- ployed Fenc = (p + 2, 100, 100, 2) for Ps and Eg, and
4. Discussion
We investigated statistical properties of a likelihood- conditional deep generative model for distribution regres- sion in a scenario where the response variable situated in
8

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
High-dimensional ambient space but is centered around potentially lower-dimensional intrinsic structure. Our anal- ysis established favorable rates in both the Hellinger and Wasserstein metrics which are dependent on only the intrin- sic dimension of the data. Our theoretical findings show that the conditional deep generative models can circumvent the curse of dimensionality for high-dimensional distribution regression. To the best of our knowledge, our work is the first of its kind.
Bhattacharya, P. K. and Gangopadhyay, A. K. Kernel and nearest-neighbor estimation of conditional quantile. The Annals of Statistics, pp. 1400-1415, 1990.
Bilodeau, B., Foster, D. J., and Roy, D. M. Minimax rates for conditional density estimation via empirical entropy. The Annals of Statistics, 51(2):62--70. 2023.
Bott. A.-K. and Kohler, M. Nonparametric estimation of conditional density. Annals of the Institute of Statistical Mathematics, 6991):19-2114, 2017.
Given the novelty of emerging statistical methodologies with intricate structural considerations in the study of deep generative models, there exist numerous paths for future exploration. Among these potential directions, we are partic- ularly interested in investigating controllable generation via penalized optimization methods, studying statistical proper- ties of deep generative models trained via matching flows, as well as delving into the hypothesis testing problem within the framework of deep generative models, among others. Another interesting direction is to explore residual neural network structure for modeling time series of distributions with interesting temporal dependence structures.
Brehmer, and Cranmer, K. Flows for simultaneous mani- fold learning and density estimation. Advances in neural information processing systems, 33:442 453, 2020.
Brenier, Y. Polar factorization and monotone rearrangement vector-valued functions. Communications on pure and applied mathematics, 4A(4):375-417, 1991.
Burda, Y., Grosse, R., and Salakhutdinov, R. Importance weighted autoencoders. arXiv preprint arXiv: 1509.005 2015.
Caffarelli, L. A. Boundary regularity of maps with convex potentials ii. Annals of mathematics, 144(3):453-44 496, 1996.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning theory by understanding the statistical foundations deep neural network models. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Carreira- Ppreiia-Perpinnn, M. A. A review of dimension reduction techniques. Department of Computer Science. University of Sheffield. Tech. Rep. CS-96-09, 9:1-69, 1997.
Chae, M., Kim, D., Kim, Y., and Lin, L. A likelihood approach to nonparametric estimation of a singular distri- bution using deep generative models. Journal of Machine Learning Research, 24(77):1 2023. URL http:
References
Aamari, E. and Levrard, C. Nonasymptotic rates for man- ifold, tangent space and curvature estimation. The An- nals of Statistics, 47(1):177 204, 2019. doi: 10.1214/ 18- A0-ASS16 1685. URL https: /doi org/10..21// 18-A0S1585.
/ l/jmlr.org/ wttpg/aaper//22//1110000 ///ph//..rrg//ppprr////////11--0 html.
Chemseddine, J., Hagemann. P., Steidl, G., and Wald, C. Conditional wasserstein distances with applica- tions in bayesian ot flow matching. arXiv preprint arXiv:2403.18700 2024.
Altekrüger, F. Hagemann, P., and Steidl, G. Conditional generative models are provably robust: Pointwise guar- antees for bayesian inverse problems. arXiv preprint 17X55: 2023.
Claeskens, G. and Hjort, N. L. Model selection and model averaging. Cambridge books, 2008
Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Ando, T. Bayesian model selection and statistical modeling. CRC Press, 2010.
Arjovsky, M., Chintala, S., and Bottou, I.. Wasserstein gen- erative adversarial networks. International conference machine learning, pp. 213. PMLR, 2017.
Efromovich, S. Conditional density estimation regres- sion setting. The Annals of Statistics,, 35(6):2504 2535, 2007. doi: 10. 10.1114/009055477000000000000000
URL https: 10. 121 1tt.::/r.0....222000000000000000
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transac- tions on pattern analysis and machine intelligence, 35(8):
Fan, J. and Yim, T. H. A crossvalidation method for esti- mating conditional densities. Biometrika, 91(4):819-834 2004.
1798-1828, 2013.
9

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Gao, Z. and Hastie, T. Linede: conditional density estima- tion via lindsey's method. Journal of machine learning research, 23(52):1-55, 2022.
2831, 2017. doi: 10.1214/1--ES11302. URL https: / //doi. 1tt.:/or/r...111111-----3-3
Izbicki. R., Lee. A. B., and Pospisil. T. Nnkede: Nearest neighhor kernel conditional density estimation. Astro- physics Source Code Library, pp. ascl-2005, 2020.
Geman, S. and Hwang, C.-R. Nonparametric maximum likelihood estimation the method of sieves. The annals of Statistics, pp. 401-414, 1982.
Jeon, J. and Taylor, J. W. Using conditional kernel density estimation for wind power density forecasting. Journal of the American Statistical Association, 107(497):66-79, 2012.
Genovese, C. R.,
M., Verdinelli, I., and Wasserman, L. Manifold estimation and singular decon- volution under Hausdorff loss. The Annals of Statistics, 40(2):941 - 963, 2012. doi: 10. 10.1214/12-A09994 URL https: /ttpp::://do.iee org/ 10. 1214 -
Jordan, M. I. Learning in graphical models. MIT press. 1999.
Ghosal, S. and van der Vaart, A. Fundamentals of Nonpara- metric Bayesian Inference. Cambridge Series in Statisti- cal and Probabilistic Mathematics. Cambridge University Press, 2017. doi: 10. 101111997111902999900
Kallenberg, O. Foundations of modern probability, vol- ume 2. Springer. 1997.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Gibbs, A. I.. and Su, F. On choosing and bounding probability metrics. International statistical review, (3):419-435, 2002.
Kohler, M. and Langer, S. Discussion of: "Nonparamet- ric regression using deep neural networks with ReLU activation function". The Anmals of Statistics, 48(4);
Gneiting. T. and Katzfuss, M. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1:125- 151, 2014.
1906 - 1910, 2020. doi: 10.1214/1-A0S1911 URL https: /twii/diii.//////// org/ 121 / 1
Kohler, M. and Langer. S. On the rate of convergence of fully connected deep neural network regression estimates.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances neural informa- lion processing systems, 27, 2014.
The Annals of Statistics, 49(4):2231 2249, 2021. doi: 10. 10.1214/2040022044 URL https: /doi //tp:////.........././// 1214 1214/20-A0S2032
Hall, P. and Yao, Q. Approximating conditional dis- tribution functions using dimension reduction. The Annals Statistics, 33(3):1404 1421, 2005. doi:
Kohler, M., Langer, S., and Reif, U. Estimation of re- gression function manifold by fully connected deep neural networks. Journal of Statistical Planning and Inference, 222:160- 102:160-11,, 2023.
10. 10.214/0990333440000002222 URL https: //ooi. http11...224/0000536688202
Koller, D. and Friedman, N. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Hall, P., Racine, J., and Li, Q Cross-validation and the estimation of conditional probability densities. Journal of the American Statistical Association, 1014-1): 2004.
Krishnaswamy. S., Spitzer, M. H., Mingueneau, M., Bendall, S. C., Litvin, O., Stone, E., Pe'er, D., and Nolan, G. P. Conditional density-based based analysis of cell signaling in single-cell data. Science,
Han, X., Zheng, H., and Zhou, M. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 35:18100-18115, 2022.
2014.
Lee, M. Smooth manifolds. Springer, 2012.
Hyvärinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Li, M., Neykov, M., and Balakrishnan, S. Minimax opti- mal conditional density estimation under total variation smoothness. Electronic Journal of Statistics, 16(2):3937- 3972, 2022.
Izbicki, R. and Lee, A. B. Nonparametric conditional den- sity estimation in a high-dimensional regression setting. Journal of Computational and Graphical Statisties, 25 (4):1297-1316, 2016.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint 2rxiv:2210.027447 2022.
Izbicki, R. and Lee, A. B. Converting high-dimensional regression to nigh-dimensional conditional density esti- mation. Electronic Journal of Statistics, 11(2):2800
Liu, S., Zhou, X., Jiao, Y., and Huang, J. Wasserstein generative learning of conditional distribution. arXiv preprint arXiv:2 /2. 10039, 2021.
10

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Mroueh, Y., Li, C.-L., Sercu, T., Raj. A., and Cheng, Y. Sobolev gan. arXiv preprint arXiv: / 171 1.04894, 2017.
Song, Y., Durkan, C., Murray, and Ermon, S. Maxi- mum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34: 1415-1428, 2021.
Nichol, Q. and Dhariwal, Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-88 8171. PMLR, 2021.
Spirtes, P. Introduction causal inference. Journal of Machine Learning Research, 11(5), 2010.
Norets, and Pati, D. Adaptive bayesian estimation of conditional densities. Econometric Theory, 33(4):980- 1012, 2017.
Stanczuk, J. P., Batzolis, G., Deveney, T., and Schönlieb, C.-B. Diffusion models encode the intrinsic dimension of data manifolds. International conference on machine learning. PMLR, 2024.
Ohn, I. and Kim, Y. Smooth function approximation by deep neural networks with general activation functions. Entropy, 21(7):627, 2019.
Sugiyama, M., Takeuchi, I., Suzuki, T., Kanamori, T., Hachiya, H., and Okanohara, D. Least-squares condi- tional density estimation. IEICE Transactions on Infor- mation and Systems, 933)):833-594, 2010.
Je J. Causal inference in statistics: An overview. Statis- tics Surveys, 3(none):96 146, 2009, doi: 10.1214/ 09-SS057. URL https:: /tpp:://doi..// wttpii.orr/1..... /t.gg11..1111 1214/ 09-55057.
Tang, R. and Yang. Y. Minimax rate of distribution estima- tion on unknown submanifolds under adversarial losses.
The Annals of Statistics, 51(3):1282 1308, 2023. doi: 10.1214/23-A052291 URL https: /doi org/10. 1214
Pidstrigach, J. Score-based generative models detect mani- folds. Advances in Neural Information Processing Sys- tems, 35:35852-35865, 2022.
Van Der Maaten, L., Postma, E., Van den Herik, J., et al. Dimensionality reduction: comparative. J Mach Learn Res, 10(66-71), 2009.
Pospisil, T. and Lee, A. B. Rfcde: Random forests for conditional density estimation. arXiv preprint arXiv:1804.05753. 2018.
Villani, C. Topics in optimal Iransportation, volume 58.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
American Mathematical Soc., 2021.
Villani, C. et al. Optimal transport: old and new, volume 338. Springer, 2009.
Vincent, P. connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661- 1674, 2011.
Schmidt-Hieber, Deep relu network approxima- tion of functions on manifold. arXiv preprint arXiv:1908.00m 00695, 2019.
Wong, H. and Shen, X. Probability Inequalities for Likelihood Ratios and Convergence Rates Sieve MLES.
Schmidt-Hieber, J. Nonparametrie regression using deep neural networks with ReLU activation function. The Annals of Statistics, 48(4):1875 1897, 2020. doi: 10.1214/19-A0S1875. URL https: //t///i.oo/// odoi.org/10. 1214 2214/19-A0S1875,
The Annals of Statistics, 23(2):339 362, 1995. doi:
10. 10.1214/aose ///11////// URL
10. 21
Yarotsky, Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103-114, 2017.
Simar, L. and Wilson, P. W. Statistical approaches for non- parametric frontier models: guided tour. International Statistical Review, 83(1):77-110, 2015.
Zhao, D., Dalmasso, N., Izbicki, R., and Lee, A. B. Diagnos- tics for conditional density models and bayesian inference a algorithms. Uncertainty Artificial Intelligence, pp.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265 PMLR, 2015.
1830-1840. PMLR, 2021.
Zhou, X., Jiao, Y., Liu, J., and Huang. J. A deep gener- ative approach to conditional sampling. Journal of the American Statistical Association, pp. 1-12, 2022.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- based generative modeling through stochastic differential equations. arXiv preprint arXiv:20/ 134456 2020.

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Supplementary Materials for "A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models"
A. A Additional numerical results
A.I. Numerical result for real data
We utilized the widely used MNIST dataset for two purposes: to demonstrate the generalizability of our approach to benchmark image dataset where the intrinsic dimension d is much lesser than the ambient dimension D 784 and to underscore the effectiveness of sparse networks as outlined in Lemma 4.1 and Corollary 1.1.
For the fully connected architecture, we set T Tenc (10+ 784, 512, 2) for 1e and Eg. and rdee = (10-2, 512, 784) for g. For the sparse architecture, we use Tenc = (10 + 784, 608, 432, 256, 2) for Pa and Σg, and Idec = (10 + 2, 256, 432, 608, 784) for g. The input dimension of for both the encoder and decoder corresponds to the one-hot encoding of the labels. We employ batch size of 64 with learning rate of 10-3
Figure 3 presents visual comparison between real and generated images, organized according to their respective labels. The real images were randomly sampled from the training set along with their corresponding labels, while the generated images were produced using these labels (conditions) and random seeds.
0
O
O
O
O
O
O
O
0
0
O
0
0
0
O
/
I
/
/
/
/
I
/
/
/
1
L
I
2
2
2
2
Z
2
2
]
2
2
2
2
2
2
2
2
2
2
3
3
3
3 3 3
3
3
3
3
3
3
3
3
3 3
3 3
3 3
3
3
3
4
4
4
1 4
4/
4
<
4
4
4
/
4 4 4 4 4 4 4 4
4 4
4
4 4
4
5
5
5
S 5
5
5
5
5
5
5
5
5
5 5 5 5 5 5 5
5 5
5
5
5
5
Le
6
6
6 6
6
4
6
6
6
6
6
6
6 6 6 6 6 6
6
6
6
6
6
6
6 6
6
6
7
7
7
7
7
2
7
7
7
7
7
7
7 7 7 7 7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8
e
8
8
8
8
8
8
8
8
8
6
8
8
8
8
6
8 8
8
4
9
9
9
9
9
9
9
7
9
7
9
9
7
9
9
9
9
7
9
7
9
9
7
Figure 3. MNIST images: real images (left panel), generated images with sparse architecture (central panel), and generated images with fully connected architecture (right panel)
This MNIST example highlights where the intrinsic dimension is significantly smaller than the ambient data dimension. This example serves validate the proposed methodology in high-dimensional settings.
To quantify sample quality, we computed the Wasserstein-1 distance (W1) between generated and test images. For each digit, we averaged W, distances over 50 samples, reporting results as mean + standard deviation. For reference, the baseline W1 distance between two test images is 2.0219 + 0.7450. Table summarizes these distances across different levels of Gaussian noise added during training.
12

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Table 2. Mean W1 distance (+ SD) between generated and test MNIST images under varying training-data noise.
Noise
Sparsely connected
Fully connected
0
1.9555 + 0.7182
1.8859 = 0.7355
0.005
1.9478 + 0.7329
1.8663 = 0.6251
0.01
1.9503 + 0.7291
1.9598 = 0.6867
0.02
2.0699 + 0.6937
2.0616 0.7410
0.04
2.2199 + 0.6735
2.2117 = 0.6627
0.06
2.3487 = 0.6576
2.3172 = 0.6267
0.08
2.4623 + 0.6245
2.4076 = 0.6308
0.1
2.5734 + 0.6492
2.5002 = 0.6337
0.3
3.4931 + 0.7012
3.4943 = 0.7164
0.5
4.0880 + 0.7518
4.0995 = 0.7633
As shown Table 2, at zero noise both architectures achieve WI distances slightly below the baseline, indicating high-fidelity sample generation. As noise increases, W1 distances grow steadily, reflecting degradation in sample quality. Both network types follow similar trends, underlining robustness architectural choice; minor deviations suggest subtle differences in sensitivity to noise. These empirical observations accord with theoretical predictions the large-sample properties of manifold-supported data.
A.2. Additional numerical results for distributions manifold
We extended our analysis to examine how the empirical W1 distance varies with sample size, while keeping the noise level fixed at 0.01. Below is summary table showing the median empirical Wasserstein distances for different sample sizes. The experimental setup remains consistent with the manifold case described in the Section 3.
Table 3.
Empirical Wasserstein distance W (median) for different sample sizes
Sample Size
Two Moon (a, 0.01)
Ellipse (o. 0.01)
4000
0.251
0.295
6000
0.232
0.285
7000
0,216
0.271
8000
0.214
0,253
9000
0,212
0,259
10000
0.196
0.251
While extracting exact rates through simulation can be challenging, the results in the table validate the large-sample properties for manifolds. These empirical findings align well with the theoretical expectations, further confirming the consistency and convergence trends our framework.
B. Notation
We denote V b and A b as the maximum and minimum of two real numbers and b, respectively. The notation [a] represents the smallest integer greater than or equal to a. The inequality S b indicates that a is less than or equal to b up to multiplicative constant. When we write Stog 6, it means that a is less than or equal to to a logarithmic factor, specifically log(n). We denote a x b when both a S and S hold. For vector norms, lp represents the e norm, while || p denotes the LP norm of function for I < oc. Lastly, (u) signifies the Euclidean open ball with radius e centered u.
We use the multi-index notation through the main paper and the appendix. Denote N the set of natural numbers and No as N U {0}. For vector x E R², denote the components f: D C I" + R. the operator is defined as ~ == S ...ga(t) with a E Nj. where yمار) f == gost) f(x)/0x0) For a E Ng. the expression aa| Eラ=1 |ao Given a function f(,)): D x D, R. x R -> R. we denote the operator
(c()),..,,,17)) Given function
13

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
gata, = galh) ...0 go)" ....aa)" with a E Ng and a E No' where Or f(x,y) = )r f(x, y)/aal) x and go! f(x,y) g ()xy))0)(0) with x E D and y E D,. This notation allows us represent the derivative with variable and y separately through the vector and a,, which is required to tackle the smoothness disparity along and y variable. The H-Hölder class functions are defined as
H (D, {s: D
Σ 100 fl
E sup U u2
0 f(u)
0 f(u)) U2
M
(15)
[
[1
<8
a:|a|=10
[o]
We extend this definition to include the Hölder class of functions with differences smoothness (smoothness disparity) along two variables. This class is defined as
H.S.A. D,D,,M) = (f(:): DX R' X RR + R
1 S(V1, U1) gata, (eattaa((V... U2)
Σ
| +
sup
112 18-10] V 11 - V2 1.
S M).
(16)
a:|a/<8 ar: 0r/3B
or ax 1aal=187 au:/a [
In
Vi.vgE Dy
U1
Vi
denote = MD) and H.S.8 (D, D,) = UNSoHH (D,
C. More on Smooth conditional density
Theorem 4 ((Villani et al., 2009) Theorem 12.50). Suppose that
(i) Ai and A2 are uniformly convex, bounded, open subsets of R with CL3J+2 (contimuously differentiable up to order [3] + boundaries,
(ii) h E (Ai) and liz E (A2) for some 3 > 0, are probability densities bounded above and below.
Then, there exists unique map (up to additive constant) g AA A2 with g HS+1(Ai) such that U h2 then g(U) h2-
Proof of Lemma 2. Given that Z and X is independent, the product measure Z X' pzp Following the smoothness
from Pz and uxx, the map Pz (+)x' (.) E Hminn (82 xx) x X).. This implies that Pz(-) (-) E (3in 12,88,.99 .903 (2 x X).
Again 4, E 0 ()) implies E Hmiul 8x Bol ()). The result now follows directly from Theorem 4.
Many of the problems in the conditional setting have an analog in the joint setup. Our proposed approach has a direct statistical extension to this setup. The sufficiency such extension follows from the observation in the subsequent Lemma which is based on Lemma 2.1 and Lemma 2.2 of Zhou et al. (2022) (see also Theorem
of Kallenherg (1997)).
Lemma 3 (Noise Outsourcing Lemma). Let (Y,X) E y X with joint distribution Py.x. Suppose Y standard Borel space, then there exists Z N(O, Im for any given 2 1, independent of X, and Borel measurable function G: Rm X + ) such that
(X, G,G(Z,X)) ~ XY.X).
(17)
Moreover. the condition (17) is equivalent of
G(Z,z) Pr|x=r
D. More on Conditional distribution on manifolds
Suppose ()r, sp) is the single chart covering V/, where y : Bi (0a_) + y is a homeomorphism. assume that y E HBmm and that inf,e (0o.) [Jpp (u)) is bounded below by a positive constant, where
Op 0e Tu du
[bo( v det
14

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
the Jacobian determinant of sp
Note that when d,, < D. the distribution Q. cannot possess Lebesgue density because of the singularity of V. We, therefore consider a density with respect to the d, dimensioaa Hausdorff measure in Rd, denoted by Ha.- Suppose that Q allows the Radon-Nikodyl derivative q with respect to Ha.- We further assume that q is bounded from above and below and that q o 4 E H Then by change of variable formula, the Lebesgue density of Q, the push-forward measure on Bi (0a.) ) through the map sp- is given as
7(u) = ((((())))
Following the assumptions the Jacobian determinant and y E A it follows that [Mp(u)] is bounded from above and below, and the map u + p (u)) belongs to H Therefore, ② is bounded above and below, belongs to S (Bi (0a. )). By Lemma 2, assuming Smin S 3z A 8x. there exists g E Huuntt such that Q = Qg. Thus, we have Q = Qoog where o Z X + V. Following Lemma 4, is possible find the appropriate neural network approximating them.
Suppose )' is covered by the charts (k)) with 1 < K < co, where PP (0a.) - Us is homeomorphism. As before, we assume 9% E H0 ps (u) is bounded below by positive constant, Q possesses density g with respect to Ha. that is bounded above and below, and that g Φx E Mmin. Let Qx(+) = (U)) be the normalized measure of Q over Uk.
We denote qk as the corresponding density with respect to Ha. For u E Uk n Ue, qu (u)Q(Uk)) = (e())((U)) qu)) holds due the measure Q(-) being compatible with the charts. This is ensured because the densities Q(Ux) (+) and ((U)))e(-) are consistent and align with the measure Q over the overlapping regions of the charts. This compatibility is essential for constructing coherent global measure from local chart densities.
A compact manifold y can be covered by a finite partition of unity (Tk, k = .,, K, each sufficiently smooth (Lee, 2012). By definition, each function in this partition satisfies (u) = 0 for & Us and Ek=i Tx(u) = 11 for all E y. Given that (uu) = QUv))q (u) for each k and u E Uk, we can express (u) as:
q(u) = Σ QUU))T((
To normalize, let = f Tx (u) and define of Tx (u) (1))))((//ee (u)/ck- Thus, we can rewrite q(u) as:
K
q(u)= Σ (u),
where = c& QUUk). This formulation reveals that is a mixture of the component densities & (u), weighted by mr. This mixture approach ensures compatibility across different charts, providing a unified density representation over the entire manifold y
Since & is sufficiently smooth, we can construct a mapping 9k 1i y such that Q3 is the distribution of gı (V). supported on U2, where Y is a uniformly convex set in Rdd, and V follows a uniform distribution on V. Next, construct a disjoint partition of the interval (0. 1) into K intervals I,.... IK with lengths ......, TK. where I = |E n. Σ T.). Define hy as the indicator function on the interval TE. i.e., hy = 1 if u E T and 0 otherwise. For random variable U following Uniform(O, 1). it follows that (U) = 1) = Te. and (h (U) = 0) = 1 - тk- Now, define v = (u, 7), where ~ Uniform(0. 1) and v ~ Uniform(V). Using this, construct g(v) = k= ha (u)gx (v). It is straightforward to observe that Q = Qg- as the partitioning through hg ensures that the measure is correctly matched to each gk. and gk ensures that the restricted distributions Q¹ are appropriately supported on Us.
From an approximation perspective, the indicator functions hy and the localized generators can be effectively approximated using ReLU neural networks. This also holds for their products and further linear combinations. For details on such constructions, one may refer to Schmidt- Hieber (2019) for sparse neural networks and Kohler et al. (2023) for dense neural networks.
It is important to note that we do not guarantee the regularity of the 9a maps, as they are not necessarily lower bounded. However, the partition of unity maps T. vanish only at the boundary of Uz. This property may allows for the construction of sufficiently smooth maps. For the multiple-chart case, we rely on more stringent results, such as Brenier's Theorem (see, for example, Villani et al. (2009)) or the Noise Outsoureing Lemma (Lemma 3), to ensure the existence of the transport maps.
15

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
E. Proof of Lemma 1
Proof. For gi (-1x), E F with Il|9: - 92loollo0 S mn. Then
pon (y))) Pozia (z/lz
=/ -f Lo(u 91 (x, 2))
Lv 22(
d. dPP((2)
Φ (y
91 (I, 2)
= / Ooly 91 (2, z))
exp {
02 2)13
- 91, 2)12 } dP2(2))
20
$ f 0u - 91 (x)
1 92 (2 2) ly 91
(z, 2)13
dPz(2)
(18)
202
= - 2o 9 (1x2))))
(92(x,2) - 91 (x, 2)
2(y 91 (x, 2)) 8) -2) 91 2]))
dP>(z)
20²
/ 00(v (2 2))
(02(1,2) 91 (16
2/y - 9 (2, x) (02 2) - (1; x)
dP2(22)
202
20
- / 0o(U - (1(x,2)) 2))
2KDm dP2(2)) +
2111
/ |V (((2,))200(0 - (z 2))dP((2)
(19)
202
202
2T
1
111
/V
D
(V2mo2)0-1 dP2(2)
(20)
202
02
2he
2TT2
[K D)omla
(D+22)
711
(21)
For the last line, we use the fact that min S The inequality at (18) follows from e- (1- - -x). The ones at (19) follows using
192(2,2) 2) 91 2x, 2K (2(g)(1,2) g1 (xx,,)) - 2K D/gsr, 2) - 91 (z, 2)
- 2K |l|g] - 92 S 2K Dm
and (92(x1, 2) - 91 (21, S wh. The change (20) follows from c. 91 (Ix S (3V [V2T0² and the bound
|0| 10]100(1) S V/ 2Te (V2To2)0-1
Now for σ1, 02 E [omiu mmax] with |o1 02| 122 It holds that |0 [01-
20?00000 (ar 02) and log
02
a
0i.000 We have
Pg.or (y/z)) o2 (3/zz
/ Do (y- - =g( 2)
D
exp
ly
g(x, 2)12
//2
|}
dPz(2)
02
2
o
/ do (v - g(0, 2)
ly g0z, 2)
1
01 dPz(2))
D Dog
(22)
2
03
/ On (v g(2, 2)
1y - g(x, 2)1
01 + 02
D1p
1/2 +
dPz(2)
2
oia
min(a1,02)
σ + 02
Dij
1/2 +
(23)
(V 2
co?
(2
D
min(or1.0
(D2 (1)0))))))
-(D+1)
72.
(24)
16

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
The (22) follows from e 2 a. The change at (23) follows from Oo (v - ((x,))) - (V2mo²)
and
2
(v) - (v (V2TO22)
Let 0. Let (.,,..., (N:) be 1n1 -covernngg of F and far ,.,,.... NN2 be 7/2 coverinnin of [Tmin Tmax] with respect n+1
IlI looloo and loo. By (21) and (24), 1n Ci 2 2/4 and 1/2 C2 min E/ implies
(Ps.os (+1-) :
Ni.j
N2}
forms an e/2-covering for P with respect to || |o.. Denote the envelope function of J
1y12 4K
Hyy = sup p(y(2)) -
exp
PEp
(2TO min
D/2
40
=ex
k² D 12220 max D/2
max mmi
① /20 (1).
Following from M do(y)dy S -1/20 we have
[
Hly x))(ylz)dy) ux(e)do <e,
where
K2D
1/2
T max
B = max
log
+ D log
+ 20 max
+ log 21)
For each (i,j) define
(i,(1. x) = max (pm.os (v, -
and
u1 min {Ponos (v,x) + Hy,
It follows that
/ [ (Uiy(w - 165 x)) pix (x)dyda
/ / ENx (x)dydx + J Hu,, x) (x)dydr
(25)
$ { (2B) + 1}c.
Denote 82 = ((2B)D + 1]. With dy (uij 4i)) S di (uij, 4z), we have
(n P. dH) & 18 P. P.di) - N1 Tmax Tmin ((,, F,lll loolloo).
(26)
is possible to write
maxx D/2
8² (i D)
+
log
Tinn
where (o mmaae D) and (2K) is constant. There exists small enough e, (D) such that for all E (0,
D/2
8 < Ca(@mmax D. KV
log
mmax Tmin
Consequently, there exists & = (D), such that for all 8 < 8., we have
(o K, D)8*
log
Tmax
I c.
Juin
17

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
lead us to, for all 6x
Cooni 8
M1 Jmin (oagg/max//mmiii
D+3 00 min 30+
(27)
where max K, D) constant. We use the fact that min (log(Tmax minn]] is bounded above by some constant depending only upon maxx muin I 1. Similar to (27), it is possible to write for all 5 > 6.
122 do D+28 min
for all 6 < 8,,
(28)
where d' (m K, D) is some constant.
The result now follows direetly (28) and (27) with (26).
F. Proof of Theorem
Proof. Choose four absolute constants Ci....,Cc C1 as in Theorem Wong & Shen (1995). Define c and in the statement of Lemma 1, The proof closely follows Chae al. (2023). We have therein the proof of Theorem that
V22
Jesp2 /og Nog.N/(8/cs, Pdd
(29)
S /25 / (D+)) log min + csf + /25 4(E 1) log((2/E2))
for every 8 S /2 S c38./V2, where cs = (c, C, eg). Observe that 64 ne, upper bound to (29) and Eq. (3.1) of Wong Shen (1995) satisfied.
Using 1.12 of Ghosal & van der Vaart (2017), we have
K(pa.... Pa.o.) $ / K(N (G.(2,)),...) N (s(2,z), 1 (x) de dPz(2)
/ [G.(2, - x) dP
D82
203
approx
5n.
20?
20
One may easily see that
[los (10g 2x(2) 2x(x - y) (103300t)))
)² do (x)dx=
1u12 + 4|2TT/
y/
/
|x|3
do
(x)
do
(x)dx.
4g2
4
Combining this with Example B.12, (B.17) and Exercise B.8 of Ghosal & der Vaart (2017), have
J log Pa..o. (ulx))
2
d.. dP. ux (x)dr
Pa.a. (v/x)
log Lo 2 - G(2,z)
(2
2
2o dy 1x (x)dx
D2 54
a dr (v)dy + o²
2D82
82
4
approx
D82
approxx
o²
approx
< C7
approx
o²
T
where = (D). We are using 5, and Tn although they are independent of n, for notational consistency with Theorem 4 of Wong & Shen (1995). Let e = en V v128.. Then, using Theorem 4 of Wong & Shen (1995), we have
P. (dn en) & 5e c2n + Ta
5e - +
20
nd,
Dn.
The proof is complete after redefining constants.
18

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
G. Proofs of Corollary 1
Proof. For the sparse case 1.1, utilizing the entropy bound from (10), we observe that
5{A + log(n//minn) x Sappiex log" (6 (Sxpprox);
which naturally leads to the required convergence rate.
Similarly for the fully connected case 1.2, utilizing the entropy bound from (11), observe that
{ + log(n/O Suppte log"(6 (Sapprox),
which naturally leads to the required convergence rate.
H. Proof of Theorem 2
Proof. It suffice assume that and o, Vloge-I are sufficiently small. not, let e + as / /llgg-1 2 co. where Co(K, K, r.r.). Then Theorem holds trivially by taking a large enough constant depending just on D, K, and r..
Let V ~ Q(-x z). V. Q(-IX z). N(On, o2a) and c. N(OD, be independent with underlying probability density We truncate the random variable and e. componentwise as ( max{ KK, min ey}) and (c.k), m K, min [K (c.), respectively. We denote Pg.or as P. Qg as Q. P as distribution of V + cσ and P. as the distribution of V, E.K. One may note that (P.,.. S W2(P,. Q.) S / /Blle.K 13] S /E[le, S S..D. Similarly, Wi(., Q) ovD. The & diameter of [-2K, 2K]P, where the support of P and P., is 4KD. Observe that
Wi (P..P) S4KDA, (P., P) < <AKD& P) (P
where the first inequality follows from Theorem 4 of Gibbs & Su (2002), the second inequality follows from the fact the distance between two truncated distributions is always lesser than the original distributions and the last inequality follows from di < 2dH. Hence,
W1 (Q-, Q) S W3 (c., Wi (P.. + W2 (P.Q) S <ovD+sKDe+ovV
Now it is suffice show that S co. Vloge-, where c,D, K, r*) is constant, because we have assumed that small enough. We establish this in the rest of the proof. Let o D log (2))
1/2 Observe that
J Pa,
60. (z)dx S
/ d. (x)dr < 221 De 6//2002 Se.
Let M = M. + (0p). We may write
e. 1 1- P. (M..) = uY, t $ M s s (lea)2 > t.)
(30)
- P( 2 -
the implication in the last line follows from sup, PP(B) - (.()) S (P.(P.P.) & e. For the sake of contradiction, let o E [2t,,r*/22 /2] U (rw/2, (t, sufficiently small, from the assumption we made at the beginning of this proof).
rr/2,, then
28 > 1 P (M:)) 2 1- 1-111 P(-K((()))))) 2 C2 D,r*)
where C2 is some positive constant. It is a contradiction following from the smallness of E. Lets make claim that if
E [2t,, r2/ then for every E D, there is some E RD such that |2 v/2 < and B,/2(z) n = 0.
Following from the claim, we have
V VY M v) v (EE Bo/2(2 y)
19

---

A L Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Since |= y/2 v a, the right hand side is bounded below by a positive constant depending just D which is again contradiction (30). This proves the assertion made in the theorem.
The proof of the claim is divided into three cases. Let p M.) inf - 3/ E M, be the Eg set distance.
Case 1.
2 a We may choose: = =
Case 2. ply. M.) E (0, Let yo be the unique Euclidean projection of y onto M,,. Such unique projection exists because < ru within the reach and y .M., since M,, is closed. Suppose yt yo f yo). We shall define two continuous functions do
1e/- Yol2 and d(() p(ye, MM.). It is obvious that d(t) Σ (t). Fort [0, 1 + a /ly Yo la
(L) d(() because y/ is the unique projection for all the points that lie on the line segment including the farthest point with = 1 + a/ll - 3012. Otherwise, say d(t) ((v,, z) and
vol2 y val2
yo 1y
vel + 1ite
=1 yy 2|2
which contradicts yo being a unique projection. The claim holds for the point 2 = /1++//y-voll To see this, observe [z - y = and B, (2 a because 1, S a and the ball Ba)2(z) C M is within the reach of the manifold. Case 3. p(y, M..) Because M. has empty interior, for all 7 > away from M,. For small enough y, reduce to case 2 by taking y - O, the limit point of B, has the required behavior.
always find point y/, which in (y) which
I. Proof of Corollary 2
Proof. The effective noise variance after the perturbation would be
o. = = + +n 9./2(8.-1.)
n
a << <. + (.))
9. 12)
otherwise.
Following this and the Theorem 2, for the rate we have
8e-tom 2o + -a) log
5 + [og((e)))
x
ممttي Bottl log log²(n); (n),
a <a<3B/(2(0 + +(.),
मलग log (n),
otherwise.
J. Proof of Theorem
Proof. With m [n)] and N (2'0"d000" d+07 ((02'a+02/p[(aa(0pp (1[1+0605' (2400 d+02'v) in Theorem 5, we can find (1/0
network G with the mentioned architecture such that
||| G. Sapprox-
Following the entropy bound from (10), we have log Nog, Ill loa olo) & sL (log(rL) + log 8-13
-37'' d+#x'n p)) log2 Smprrox { logg (5mp)rox log (6xp)rx)) ) + log (Smpprox))
< Sappeex
The rest directly follows from the Theorem
K. Approximation properties of the sparse and fully connected DNNs
The approximability of the sparse network is detailed in Lemma 4.1, which restates Lemma from Chae et al. (2023). For the fully connected network, Lemma 4.2 demonstrates approximation capabilities, derived directly from Theorem 2 and
20

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
the proof of Theorem in Kohler & Langer (2021). Additionally, the inclusion of the class in the fully connected setup supported by the discussion in Section of Kohler & Langer (2020).
Lemma 4, Suppose that G. E G. Then, for every small enough E (0,
1. there exists sparse network G E F. = (T,r,, K 1V) with L log , r 8-1.10, & 8-1.13. log 5-1 satisfying ||| - G+loolloo ó.
2. there exists a fully connected network G E Fe with L S . ---1, r
B B 8- satisfying ||| 1o
8.
L. A new approximation result for functions with smoothness disparity
In this section, prove the approximability of the sparse neural network for the Hölder class of function f H.S.S. (DD,,K)
Theorem 5. Let f E H.p and N 2 (Bsum + 1)rmm V (K + 1)e there exists network
3. ((0, 1]", 10, K). Denote sum = and Baam B + Br. Then for any integers m 2
F. (rsum: 6(rsum + [ 1)N, 6(rsum + [Bam 1)AN, 1), 00)
with depth
L + (m++)) (1+ [log2 (reum V (aum))
and the number of parameters
S (0 + Baum + 1) 3+ N(m + 6),
such that
1 [1l.~((0. S (2K + 1) + + PPum) 6romm N2 K 3rom (8- N 1/8-111118,77
We denote 3 -
and 7 = (3 + (rB
+ B.). Before presenting the proof of Theorem 5, we formulate
some required results.
We follow the classical idea of function approximation by local Taylor approximations that have previously been used for network approximations in (Yarotsky, 2017) and Sechmidt-Hieber 2020). For vector [0, 1]" define
(ga+a () (ataa)(aab)) (u - av b)a
p a.b f(u, v) Σ 0sla|<3 <8 0 a. <9
(31)
a.!
We use the notation the (ub)) to represent the component of the vector when the index well understood. Accordingly have (b(1)) (ali)), and b (b()), By Taylor's theorem for multivariate functions, have for suitable & E [0,1]
( f) (u-a) a) b)
f(u.v)=
Σ a:
a!a!
b E(v - b)) (u (u- - b)a
Σ
(8a+a () b)
2d !!
8.
21

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
We have I(u- | = [[ luj - a,l 0 $ a lg and - b) = IT 1, - -,110 - l b loll Consequently,
for f E H4,.0 (I0, 11","",,,," [0,1]",KK)
v((uvv) v) - P su, v)
(a+ (aatar a), b. <(v -b) a:-a, ala bl) (u (u-a)
b).. b
Σ
(gata,
(32)
al a/
8-13|a|<8
< (Iu
a 2l V | - b b
We may also write (31) linear combination of monomials
v) = Σ G,y, u pal a.b
(33)
05/1< 0s/
for suitable coefficients Sy For convenience, we omit the dependency on a and b in Cym. Since Ir pe. a.b f(u, v) 0,v=0) y cyno must have
(-a)a-
(8 b) 7 p/ (a -) (a., 71)!
C
<sa&la<< 7Sa.aa
Notice that since a E [0, b E [0, 11 and f E H.S (10, 1]", , [0, KK,,
lcv1l S k/(?!m!! and Σ
lyy < 1 2020 y 20
Σ
إ(3! (6) = Ketty,
(34)
720 7.20
where 7 = (,."),...,77)) and 7, = (7 , (m)).
Consider the of grid points
D(M)= = {uga =(1,0) / 1MM)) and Vgai = ((/2)/ /M
e2)) = (1()) (2) E {0, 1, 10.1,..,,Mi, M,,",
((2) = (1 )()), E (0, 1,...,My) ,rY",M, = M313 M = -M818..
The cardinality of this set is (M, + 1)" (M2 + 1)" We write 4ga = (u (m))))=i.... and Ve:2 = (1 (020))j=1.....
to denote the components of Ugon and Va respectively, With slight abuse of notation we denote w (u, (n(1), ...... (r), (1) (())) l = ((),(()) (6(2) ...., (()) ((2) (22) and we
(w=²))==1..... )==1....
(ugo), V ) = (u gcs...... 2m 2.)..... (1)) Define
p f(u,v)
= - ((w)
Σ
p s(w)
II
(1 M5 - -w0 D+
W2ED(M)
Σ
Lnn 83) v( II
P01-M|lube Mi(u() () l)+
I[
n
Mgle(1 - 162 -020 1+)
=
ED(M)
where M, = Mo j = I,..., and M, = M2 for j = + 1,... + Tr.
M
22

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Lemma 5. Ifʃ H, (0, 1]", [0, 11 K), then (..... --ll--------pp [0..)+t S KM- Proof. Since for all w (wl)), ..... (rtr)) E 10,1
744,, Mr
Σ weED(M)
II Mylw(() w/ w
II Σ
M2lw( e/M,])+
(35)
1=1 10
we have
f(w) f(u
5 [a M2 - -00 1)+
Σ
S(u,v)
II 1=1
Mi
Milu(00
-
4y(1) (2 EDMM) (DM)
I4-4,(1 Sw
||v-v S S1/M2
and with (32),
pp.9 f(u, - -[l,vv)]
P&...
Us(t) 200 ((u [(u
max
"(t) (D(M)
- 10 S1/M,
((3) S1/M2
< K M1 M55 K M-
In the next few steps, we describe how to build a network that approximates p8,8 f.
Lemma 6. Let M, m, be any positive integer. Denote M1 M310, M₂ M313, M (M + 1) 1)" and summ + Then there exists network
Hat foum E FF (m + 5) [o]]ggrrmm) )1, Tsums 21sum M, Fuu M, 6r M.... Gr M, M), s, 1)
with 37 MMm + [[[loz((rumm) such that Hat" E [0, 11M and for any u = (1(1),.... 00) E [0, 1|" for any (o(1) ..... (1)) [0, 1)
{ (T - -lu(t) - -2(0)
Hat (u, v)
[II (1/M2 - 1000 - -ug 1+) j=1
< Tsum 22-m m
M201) cD(M)
For any Ugn, Vaa E D(M), the support of the function (u. v) + (Hat v)) V))u,ayss is moreover contained in the support of the function
(u, v)
( (1/M - (n()) - ( D
Fr (II))(iAAAA (iaa/M-b0))
1100) -1000
-100 -00 D+ )
Proof. Step 1: (For r +r,= 1) Without loss of generality we consider the case when r = 1 and r, = 0. We compute the 17,
functions ((w6) 1/M and {(e/M] - 2r(M, (M, + 1) units (nodes) and 2rMM, 1 + 1) non-zero parameters.
for the first hidden layer of the network. This requires
23

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
For the second hidden layer we compute the functions (1/M, (u( - e/M1)++ = (1/M1 - (u() - e/M1)++++---/ (e/M,
)())+)++ using the output (ul)) f/M1)+ and (e/M1 =(())+ from the output of the first hidden layer. This requires
r(Mi + 1) + (M2 + 1) units (nodes) and 2r + 1) non-zero parameters. This proves the result for the base case when
r ++r,=1 1.
Step 2: For r + ry > 1, we compose the obtained network with networks that approximately compute the following
-uecs 1+ -u (2) II (1/M -110) -
[1 (1/M2
Yr II
-1/622 - -Egaa 1+
-10(0)
G)
D+)
}}
For fixed ugay and Vela,, and from the use of Lemma 8 there exist Mult, networks in the class
FF (2 + (m + 5) (l 7,)1, (r + 2(r +1), 6(r + +r), 6(r + +r),, ................................ 6(r + ry), 1))
computing =I (4/M (/ (1 Observe that we have two extra hidden layers to compute (L/M, - |u( -g( and (1/M₂ |u( Egrrr)) for fixed Ug(y and Vyee respectively, before we enter into the multinomial computation by regime invoking Lemma 8. Observe that the number of parameters in this network is upper bounded by 37 + +, - (r + +,)l-
up to an error that is bounded by (r 1r+r,)2g 2-m.
Now we use the parallelization technique to have (M1 + 1)" (M + 1)" parallel architecture for all elements of D(M). This provides the existence of the network with the number of non-zero parameters bounded by 47) - ( + 1)"(mm + 5) [Dog2 (r + r,)1
By Lemma 8, for any x E RR, Mult,, (x) = 0 if one of the components of x zero. This shows that for any Ugan, Veca E ((M), the support of the function (u, v) + (Hat (u, v))u, is contained in the support of the function (u, v) +
(IT) - (0(6) - 22 D+ IT (1/M - (0(0) - (22)++)
Proof of Theorem 5. All the constructed networks in this proof are of the form =(L.,..) oo) with F = oc. Denote M1 Mă/8, M2 = 33/8, Beum 3 + B, and rsum = r + ry. Let M be the largest integer such that M = (M, + (M, + 1)" I N and define L = (m + 5) () V (ram)) Thanks to (34), (33) and Lemma 9, we
can add one hidden layer to the network Mon to obtain a network
Q F I=, (r, (r, C B :.. 6[,3]. C faum Soum s G S M)),
such that Qi( E [0, 1jM and for any u E 1) and for any v E 1)
@1(u.v) ( (p s(u.
1) + 2 2222) ED(M) S f82 fmmm2-
(36)
B
with B := [2Ke mmm The total number of non-zero parameters in the Q1 network is 6raum (Beum + 1) Baum + 42 (Beem + C)2 Frum, Baum (I." + 1) + Craum Baaon M.
Recall that the network Ham computes the products of hat functions (splines) TI/M - l2() Ugas 1) )TTT - |000 1)+= up to an error that bounded by r2 r 2-mm It requires at most 37 NL* active parameters. Observe that Cr.mm. S (Baum + 1) < N by the definition of Cr,s and the assumptions on N. By Lemma 6, the networks Q1 and Hat can be embedded into a joint parallel network (Q1, Hat fumm) with + L hidden layers of size (Taum, 6( + [Baum of non-zero parameters in the combined network (Q1, Hat") ) is bounded by
T)., (
6(raum + [8 1)N, 2M). Using Cr,9 V (M + 1)" < N again, the number
6r (Bsum +
Boam + 42(3sum + - Famm Brum (I+ + 1) + C 1rum M + 37 sum
NL
< 4remm + Baum + Cr Bman (1)
(37)
< 84(reum + Beum + 1) 1)3troum N(m + 5),
24

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
where for the last inequality, we used &au S (Beum + 1)mm, the definition of L and that for any x 2 1, 1 ++ [zgg(2))] < 2 (og)(x)) 2(1 + log(x)) S 2x.
Next, we pair the ( Vggnn)-th entry of the output of Qi and Hat" and apply to each of the M pairs the Mult, network described in Lemma 7. In the last layer, we add all entries. By Lemma 7 this requires at most 24(m++5)M+ & 25(m++5) active parameters for the M multiplications and the sum. Using Lemma 7, Lemma 6, (36) and triangle inequality, there exists a network Q2 E F(2 + L + m + 6, (Feum, Graum + [,aum]] N).......................... 6(raum + [Baum]] 1).V, 1)) such that for any u E [0, 1]" and for any v E [0, 1
Q2(u,v) - E
Pš. f(u. v)
1
(1/M1 112(0 - -ug
+
B
2
(e2)) (D(M)
1=1
(1/M2 - 1060 - -2 D+)
Σ
(1+ sum + +33 12 )
4 (e)) eDM)
| Dc S1/M,
[V-V b0 S1/M62
$ (1 2 + (3umm) )2--m
(38)
Here, the first inequality follows from the fact that the support of (Hat** (u, v))agan is contained in the support of
[I] - 1x(0) 2g D+ (1/--1//M - lu(t) - D=) (see Lemma 6). Because of (37), the network Q2 has at
most
109 + Boum + 1) 1)3 N(m + 5)
(39)
non-zero parameters.
To obtain network reconstruction of the function f. remains to scale and shift the output entries. This is not entirely trivial because of the bounded parameter weights in the network. Recall that [2Ke"]. The network x -> BM M x the class F33, (1, Mí MI' 1, [2Ke"], with shift vectors v, are all equal to and weight matrices W, with all entries equal to one. Because of N (K 1)) the number of parameters of this network is bounded by 2.M M + +[22K"
6.N, c) with = M3'2 This network computes in the first hidden layer (a c)+ and (c - a)+ and then applies the network I + BMT M, x to both units. In the output layer, the second value is subtracted from the first one. This requires at most + 12N active parameters.
This shows existence of a network in the class F(4, (1, 2M M 2, 2[2Ke"], 1)) computing a + BM
Because of (38) and (35). there exists a network Q3 in
F(mm + 13) + L", (rsum: 6(rsum + [ 1).N.... 6(rsum + [,mum 1).N 1))
such that
F ([/ - (4 -lu0) 1.f) 1++)
Qs(u,v)-
Σ
p&... flu,v) v)
(R) EDMM)
1=1
[[ - 1000 -222h 12-)
(2K- 11M MF (1+T) sum 32 Punm (2mm m for all (u, v) [0,1]amm 5mm
Feum
25

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
With (39), the number of non-zero parameters of Q3 is bounded by
(ram + Saum + 1)3+roum N(m + 6).
Observe that by construction M (Mi + 1) + 1)"h S N < (3M2) 3 M and hence M M-
N N-8/F3rmmm81 Together with Lemma 5, the result follows.
L.1. Embedding properties of neural network function classes
We denote p) the class of neural networks with L hidden layers and p E N4+2 nodes per layer. The class F(L, is subset of F) with the sparsity parameter s.
function by a network, we first construct smaller networks computing simpler objects. Let - (po...... ,P....)) and p' = -V+1-...PL+11 To combine networks, we make frequent use of the following rules.
For the approximation
Enlarging: F(L.p, S F(L. s') whenever p S componentwise and s'.
Composition: Suppose that f E F(L, p) and g E p(',,p') with PL+1 Pb- For vector E R we define the composed network y ( which is in the space F(I++ I/' + 1, (P, ,',....''' In most of the cases that we consider, the output of the first network is non-negative and the shift vector will be taken to be zero.
Additional layers/depth synchronization: To synchronize the number hidden layers for two networks, we can add additional layers with an identity weight matrix, such that
F(L.P, F(L + t. (Po.....PPoPPlll
Po. Po. P), s PP))
(40)
times
Parallelization: Suppose that f.g are two networks with the same number of hidden layers and the same input dimension, that is, E F(,, p) and g E F(L,, p') with pa = 1-- The parallelized network (S.g) computes and g simultaneously in joint network in the class F(L, (po: Pi + pi..... .h....................... PL P + 1))+
L.2. Technical lemmas for the proof Theorem
use F(L,r) r) to denote fully connected network with deep layers and r No+2 representing the nodes in each layer. The following technical lemmas required for the proof of Theorem 5. Lemma 7, Lemma 8, and Lemma 9 restate Lemma A.2, Lemma A.3, and Lemma A.4 from (Schmidt-Hieber, 2020), respectively.
... 6, 1)), such that Lemma 7. For any positive integer m, there exists network Multm F(m + 4, (2, 6, ...
Multm (z. v) e [0, 1].
Mult, (x, - zzl < 2m for all x, E [0,1],
and Multm (0. (0,y)
Multm 0.
Lemma 8. For any positive integer m. there exists a network
Mult", F(mm + (log r1, (r, 6r, 6r,.... ...,. 6r, 1))
such that Mult E [0, 1] and
Mult", (x) - II 24 <P22
for all x = (I1,..., xr) [0,
Moreover, Mult,, (x) 0 one of the components of x is zero.
The number of monomials with degree |ax] < y is denoted by Cr,y. Obviously, Cr S (y + 1)" since each a; has to take values (0, 1,..., [r]].
26

---

A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models
Lemma 9. For y > O and uny positive integer m, there exists a network
Mon'nn F (0..
Cr,y)),
such that Mon E 10,1 and
, - (x |ale)
5222 for all [0.
27


Reviews:
[{"review_id": "N2FT6vLy0a", "reviewer": "Reviewer_p9g3", "summary": "This submitted manuscript works on the statistical aspects of conditional generative models, which is an important topic in the broad area of deep generative models. The analysis focuses on the likelihood-based models and, under the assumption that the high dimensional response variables have low-dimensional data manifold, studies the convergence rate of a sieve MLE. The finding, as stated in the paper, tries to explain why deep generative models can overcome the curse of dimensionality. \n\nIn the experimental section, the authors demonstrate the performance in terms of relevant theorems presented in the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and constructive feedback, and we appreciate the positive remarks on the main results and the comprehensibility of our proofs.\n\n**Our derivations beyond Theorems 1 and 2** follow a unified strategy controlling two sources of error. First, we bound the statistical (estimation) error using entropy bounds (Eq. 10 and 11), which essentially capture our function class’s complexity via covering or bracketing numbers. Second, we quantify the approximation error—the bias incurred when approximating the true generator with our network class. Theorem 1 establishes a general convergence rate by balancing these errors, while Corollary 1 specializes the result to sparse and fully connected network architectures. With appropriate tuning parameters such as depth, width, and sparsity, the estimator adapts to the intrinsic dimension and smoothness of the true model, crucial for mitigating the curse of dimensionality.\n\nFor nearly singular distributions, where data concentrate near a low-dimensional manifold, Corollary 2 demonstrates that a small noise perturbation smooths distribution gaps, achieving a favorable Wasserstein convergence rate and extending Theorem 2. Theorem 3 further expands our analysis to generators with varying smoothness across dimensions. By combining our new approximation result (Theorem 5) with adjusted entropy bounds, we established that the convergence rate in these settings depends on a composite measure of smoothness and intrinsic dimension.\n\n**Methods And Evaluation Criteria and Experimental Designs Or Analyses**: We would like to emphasize that our work is primarily theoretical. The simulations and MNIST experiments serve to validate our statistical insights rather than introduce a new algorithm. While some experiments use low-dimensional data in the experiments, MNIST (D = 784) is high-dimensional yet exhibits low intrinsic dimensionality (around 10). Additionally, our framework is broad and applies equally to other likelihood-based methods—such as Normalizing Flows—as demonstrated in related empirical works [A] and [B].\n\n**Other Comments**: We appreciate the suggestion. In the revised manuscript, we will add a toy example—using a one-dimensional generator map map that transforms a uniform variable into one with a β-Hölder smooth density: $u\\mapsto u^{1/(\\beta+1)}$. Setting $\\beta^*$ = 1 and $t^*$ = 1 in our framework reinterprets our results in this concrete setting, clearly illustrating the theoretical ideas.\n\n**Other Strengths And Weaknesses**: Assumption 3 holds in practice, ensuring the manifold’s closure is well defined—a common requirement in manifold learning [E]—and is crucial for maintaining low-dimensional structure. If the manifold were space-filling in $\\mathbb{R}^D$, the data would lose their low-dimensional properties. The concept of reach, related to the manifold’s condition number, is well established in the literature [C, D, E] and ensures our analysis remains robust even when the support is singular.\n\nWe hope these clarifications, along with the additional analyses (see below) and proposed examples, address your concerns and strengthen our manuscript. We welcome any further questions or discussion.\n\n**Additional Experiment Addressing All Rebuttal Feedback**\n\nTo address concerns about the empirical quality of our MNIST images, we computed the W1 distance metric (W1 ± SD). For each digit, we averaged the W1 distances over 50 images and aggregated the results. For context, the average distance between two test images—reflecting baseline variability—is 2.0219 ± 0.745. Below is a table summarizing the empirical results (noise denotes the $\\sigma$ of $N(0,\\sigma^2)$ added to the training data):\n|Noise|Sparsely Connected|Fully Connected|\n|-|-|-|\n|0|1.9555 ± 0.7182|1.9555 ± 0.7182|\n|0.005|1.9478 ± 0.7329|1.8663 ± 0.6251|\n|0.01|1.9503 ± 0.7291|1.9598 ± 0.6867|\n|0.02|2.0699 ± 0.6937|2.0616 ± 0.7410|\n|0.04|2.2199 ± 0.6735|2.2117 ± 0.6627|\n|0.06|2.3487 ± 0.6576|2.3172 ± 0.6267|\n|0.08| 2.4623 ± 0.6245| 2.4076 ± 0.6308|\n|0.1|2.5734 ± 0.6492| 2.5002 ± 0.6337|\n|0.3|3.4931 ± 0.7012| 3.4943 ± 0.7164|\n|0.5|4.0880 ± 0.7518| 4.0995 ± 0.7633|\n\nAt zero noise, both architectures yield W1 distances slightly lower than the baseline, indicating that the generated samples align well with the test images. As noise increases, the W1 distance grows steadily, showing that higher noise degrades image quality by pushing samples further from the true distribution. Both network types exhibit similar performance across noise levels, suggesting robustness to architecture choice, although subtle differences hint at variations in noise sensitivity.\nWhile extracting precise convergence rates from these experiments is challenging, the observed trends validate the large-sample properties for manifold-supported data and align with our theoretical expectations. \n\nWe propose including this analysis in the appendix alongside our MNIST experiments.", "claims_and_evidence": "There are three major assumptions that have been made. The first one is about the generative process from the underlying true distribution following the canonical form in Equation 2. The second one is about the composite function class. The last one builds the identifiability condition. \n\nThe Thm1 is followable, and the proof for Thm2 in the appendix is also understandable. But I am having trouble fully understanding the rest of the proofs and Corollaries.", "methods_and_evaluation_criteria": "The authors demonstrate their theoretical analysis through simulation data and real-world datasets. There is a quantitative and qualitative illustration that shows the performance in relevance to the statistical bounds. As I am not from the background of statistical theory, I cannot fully verify the appropriateness of the methods used to analyze the conditional deep generative models.", "theoretical_claims": "As I point out previously, my background is limited to judging the novelty and rigorous of the the theoretical claims presented in the paper. But by following the proof in the appendix, I can understand most of the proofs.", "experimental_designs_or_analyses": "There is one concern that I have about the experiments. As the paper mainly focuses on the statistical theory part of deep conditional generative models, the experimental data seems to be trivial. It is either from simulation or low-dimensional real data. And the authors claim that the statistical analysis reveals why the conditional deep generative models can avoid the curse of dimensionality. But in the experimental section, there is no dataset that has high dimensional data. Overall, I am a bit concerned how the statistical analysis helps in reality.", "relation_to_broader_scientific_literature": "I am fully capable of judging the novelty in terms of statistical theory aspects. From my reading through the paper, it does broaden my view of deep generative models from the statistical point of view. However, the concern arises from the experimental section where the scale of the dataset is limited.", "essential_references_not_discussed": "Overall, the paper is well written and cited all the key papers in the related areas.", "other_strengths_and_weaknesses": "The paper is quite theoretical and well-written in terms of problem setup and proofs. It links several key papers in the field of generative models and the story naturally follows. However, the weakness is that the experimental sections lack of convincing evidence to support the theoretical findings. Another minor weakness is that the assumption 3 excluding the interior point seems a bit too strong. I am not sure if it holds in reality.", "other_comments_or_suggestions": "Although it is a well-written paper, I find it would be better to include a starting toy example to demonstrate the idea.", "questions_for_authors": "1) Do you observe a similar pattern if you experiment on large-resolution image datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "kLFDCJdo5P", "reviewer": "Reviewer_8PjD", "summary": "In this paper, the authors explore the theoretical properties of deep neural network-based generative models for conditional density estimation. They tackle the specific problem of estimating a conditional distribution on a lower dimensional manifold, and find the convergence rate to be dependent on the intrinsic dimension and smoothness of the true conditional distribution. These results provide theoretical support for why deep generative models are effective at learning densities on a manifold in high dimensional space, and why noising data on the manifold may be beneficial.\n\n## update after rebuttal\nI appreciate the authors' further clarifications and simplifications. Taking into consideration the authors' response to my and the other reviews, and that this is primarily a theoretical work, I have updated my score accordingly, recommending weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. While our theoretical contributions are inherently technical, we have made every effort to present them in a manner accessible to a broader audience. Below, we address each of the reviewer’s points in detail:\n\n**Claims And Evidence**: Our work establishes nearly tight convergence rates for conditional density estimation on low-dimensional manifolds by analyzing deep generative models. The proofs throughout the paper follow a unified strategy that addresses two primary sources of error. First, we bound the statistical (estimation) error using entropy bounds (Eq. 10 and 11). Second, we quantify the approximation error—the bias incurred when approximating the true generator with our network class. Theorem 1 provides a general convergence rate, while Corollary 1 specializes the result to sparse and fully connected architectures, directly linking the rate to network parameters (depth, width, and sparsity). This adaptive behavior lets the estimator depend primarily on the intrinsic dimension and smoothness of the true model, mitigating the curse of dimensionality.\n\nFor nearly singular distributions, where data lie extremely close to a low-dimensional manifold—Corollary 2 demonstrates that adding a small noise perturbation smoothens the gaps. This noise injection smooths the data distribution and enables a favorable Wasserstein convergence rate, thus extending the ideas from Theorem 2. In essence, the noise helps overcome challenges inherent in estimating distributions that do not have full-dimensional support.\n\n**Extension to More General Settings**: Theorem 3 broadens our analysis to generators with differing smoothness across dimensions. By combining our new approximation result (Theorem 5) with modified entropy bounds, we show that the overall convergence rate depends on a combined measure of smoothness and intrinsic dimension, reinforcing the robustness and adaptability of our framework\n\n**Methods And Evaluation Criteria**: We acknowledge the reviewer’s concerns regarding the empirical evaluation. Our focus is primarily theoretical; the experiments were chosen to validate the statistical insights of our approach. Although MNIST is a high-dimensional dataset (D = 784), it naturally exhibits a low intrinsic dimension [C]. The presented simulations and MNIST experiments illustrate key trends predicted by our theory. Moreover, the framework we develop is general and applies equally to other likelihood-based methods, such as Normalizing Flows, as highlighted in related works [A] and [B].\n\n**Questions For Authors**: \n- While our theoretical analysis is carried out for sparse and fully connected feedforward networks, our tools are sufficiently general to extend to more advanced architectures—such as convolutional networks, GANs, and diffusion models. Our approximation-theoretic tools (e.g., Theorem 5) are potentially applicable to these architectures [F, G]. Future work will aim to incorporate these more complex models, further bridging the gap between theory and state-of-the-art empirical performance.\n- Additional quantitative MNIST experiment results have been added (please see end of Rebuttal 1).\n- For more complex image datasets, we expect these results to generalize as empirical evidence suggests that benchmark datasets typically exhibit intrinsic dimensionality on the order of $10^1$ [C].\n\n**Essential References and Other Weaknesses**: We appreciate the reviewer’s suggestions. In the revised manuscript, we will include the recommended reference (arxiv:2003.13913). Regarding notations, most are standard in the literature (e.g., [F, G, H]).\n\nWe hope that these clarifications, together with the additional analyses, address your concerns and further strengthen our manuscript. Should you have any further questions or require additional details, we are happy to discuss them further.\n\n**All Rebuttal References**\n\n- [A] Conditional variational autoencoder for neural machine translation. arXiv:1812.04405. \n- [B] Learning likelihoods with conditional normalizing flows. arXiv:1912.00042. \n- [C] The intrinsic dimension of images and its impact on learning. arXiv:2104.08894. \n- [D] Vector diffusion maps and the connection Laplacian. arXiv:1102.0075.\n- [E] Estimating the reach of a manifold. arXiv:1705.04565.\n- [F] Diffusion models are minimax optimal distribution estimators. arXiv:2303.01861.\n- [G] Adaptivity of diffusion models to manifold structures. PMLR:238. \n- [H] Smooth function approximation by deep neural networks. arXiv:1906.06903.\n- [I] A likelihood approach to nonparametric estimation of a singular distribution. arXiv:2105.04046.\n- [J] Statistical theory for image classification using deep convolutional neural networks. arXiv:2011.13602\n- [K] Nonparametric regression using deep neural networks. arXiv:1708.06633.\n- [L] Generalized Additive Models: Some Applications (JASA, 1987)", "claims_and_evidence": "The paper claims 1) to be the first to explore the convergence properties of deep generative models for conditional density estimation in a low-d manifold, 2) as a result, provide theoretical justification for why deep generative models have been empirically successful in modeling real world data in such scenarios (e.g., images), and 3) numerically demonstrate these claims in synthetic datasets and MNIST.\n\nI do not know the literature well enough to evaluate 1) or 2). In terms of 3, quantitative results on the synthetic task seem satisfactory (Table 1), but evaluation on the “real world” dataset of MNIST is rather qualitative. In particular, samples look quite blurry and lack diversity. It’s not that the qualitative performance is underwhelming, it’s more that I don’t quite understand what these results show—that the VAE setup with sparse / fully connected networks *can* model such low-d embedded distributions?\n\nAside from that, I was not able to understand the paper well enough, specifically Section 2, to evaluate the theoretical results fully.", "methods_and_evaluation_criteria": "I’m not sure whether the classes of neural networks considered here (sparse and FC feedforward) are complex enough to empirically support the theoretical points, since most modern DGMs come with more inductive biases (when working with real data) that are explicitly designed to first reduce the data dimensionality, e.g., from pixel space to an embedding via convolution. Additionally, as mentioned above, I did not find the visual evaluation on MNIST convincing.", "theoretical_claims": "Was not able to evaluate.", "experimental_designs_or_analyses": "The synthetic and MNIST datasets seem suitable to demonstrate the points. On the other hand, it would have been nice to see slightly more complex datasets and architectures.", "relation_to_broader_scientific_literature": "A theoretical understanding of how and why modern deep generative models work is an important research direction, given its prevalence in modern machine learning. That being said, I was not able to evaluate how meaningful of a contribution the current work is.", "essential_references_not_discussed": "This work seems quite relevant: Normalizing flows for estimating conditional densities on a manifold: https://arxiv.org/abs/2003.13913\n\nNot necessarily a problem for the current contributions, but definitely in the same problem space, and therefore should be discussed.", "other_strengths_and_weaknesses": "Not necessarily the authors’ fault, and potentially more so my lack of expertise, but I found the theoretical results, and especially the notations, to be very difficult to follow along.", "other_comments_or_suggestions": "Perhaps a high-level outline of what Section 2 aims to achieve, step by step, would be informative.", "questions_for_authors": "- how would the results generalize to (or are related to) deep architectures closer to those used in the last few years, e.g., flows, GANs, diffusion models, as the authors initially cite?\n- can the results on MNIST be quantified?\n- do the results hold for datasets more complex and/or relevant than MNIST?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R73KwMFp1q", "reviewer": "Reviewer_vpHs", "summary": "The paper investigates a likelihood-based framework for regression in a view of conditional distribution estimation. The authors propose a Sieve Maximum Likelihood Estimator (MLE) to estimate the conditional density of the response given predictors. Importantly, they derive convergence rates (under Hellinger and Wasserstein metrics) that depend solely on the intrinsic dimension and smoothness of the true conditional distribution. Specifically, by incorporating a small noise perturbation into the data distribution, it shows to improve the estimation. This insight is validated with numerical experiments on both synthetic datasets and MNIST data, illustrating that the proposed likelihood-based approach outperforms several existing methods", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and constructive feedback, and we sincerely appreciate their recognition of the strengths in our likelihood‐based framework and its theoretical underpinnings.\n\n**Claims and Evidence:**  We thank the Reviewer for highlighting these key aspects of our theoretical analysis, the primary focus of our work. The simulations and MNIST experiments are designed to validate our statistical insights rather than introduce a new algorithm. Moreover, our framework is broad and equally applicable to other likelihood‐based methods—such as Normalizing Flows—as demonstrated in related empirical works [A] and [B].\n\nOur theory indicates that a small noise perturbation is needed to smooth nearly singular data distributions. This is demonstrated with our synthetic manifold example and further supported by additional simulations (please see end of Rebuttal 1) which show how noise levels affect the MNIST W1 distance. In theory, this noise level is guided by the data distribution properties, and practically often chosen via cross validation as explored in [I], balancing minimal distortion with stable optimization\n\n**Theoretical Claims:**  While it is possible to derive the exact constants in our proofs, doing so can be cumbersome and distract from the main insights. Nevertheless, we will make every effort to track the constants that are tied to key problem characteristics. Our focus remains on clearly illustrating how these constants depend on critical variables—such as the intrinsic dimension and smoothness parameters—while remaining independent of the sample size n. As is common in the literature (e.g., [J, K]), tight upper bounds suffice to capture convergence behavior without specifying exact numerical values.\n\nWe consider our assumptions to be reasonably mild. In particular:\n- Assumption 1 guarantees a density exists for both noisy and noiseless data on the manifold—which is widely acceptable and foundational in statistics and machine learning.\n- Assumptions 2 and 3 are practically motivated. For example, [C] provides empirical evidence for lower-dimensional data structures, and Assumption 2 generalizes common structures like multiplicative and additive models [M].\n\n**Experimental Designs Or Analyses:** \n- Our experiments with synthetic data and MNIST illustrate and validate the key theoretical insights of our approach. While further empirical validation on diverse real-world datasets and architectures would better demonstrate robustness and practical utility (see [A, B]), a comprehensive evaluation is beyond our current scope. We expect these results to generalize to more complex image datasets, as benchmark data typically have intrinsic dimensionality on the order of 10¹ [L].\n- We have added quantitative analysis on MNIST to assess how the W1 distance varies with different noise levels (please see at the end of Rebuttal 1). Additionally, we evaluated sample size effects by comparing performance at 25%, 50%, and 75% of the data. In full-dimensional cases (FD1, FD2, FD3) where smoothness is critical, performance improves significantly with increased sample size. Although our current experiments show minimal variations with network depth and sparsity, we acknowledge their importance and plan to explore these further.\n- Since our framework is built on Wasserstein and Hellinger distances, our analysis focuses on these metrics. Nonetheless, exploring additional evaluations—such as uncertainty quantification and further baseline comparisons—would provide valuable insights. We view this as a promising direction for future work aimed at supporting optimal theoretical rates for these additional approaches.\n\n**Essential References Not Discussed**: We shall include Han, Zheng, and Zhou (2022) in our references. While CARD employs a score‐based, empirically focused approach, our work establishes nearly tight convergence rates for likelihood‐based methods (including VAEs and normalizing flows) in conditional density estimation, offering new insights into overcoming the curse of dimensionality.\n\n**Other Strengths And Weaknesses:** We have improved the resolution and clarity of Figures 2 and 3, ensuring clear fonts and shapes. These enhancements will be in the final manuscript, subject to permission.\n\nWe hope these explanations, along with the additional quantitative analyses, address your concerns and further strengthen our manuscript. We welcome any additional questions or further discussions.\n\n**References:** Please see at the end of Rebuttal 2.", "claims_and_evidence": "The paper clearly establishes that the convergence rates for the proposed sieve MLE depend only on the intrinsic dimension and smoothness of the true conditional distribution. The derivations that motivate adding a small noise perturbation to handle nearly singular distributions are mathematically sound. The empirical explorations are limited mostly to synthetic examples and MNIST data. More diverse real-world experiments would help convince the reader that the approach reliably circumvents high-dimensional challenges. Moreover, the theoretical analysis suggests that injecting a small noise is crucial when the data are very close to a lower-dimensional manifold. Although some experiments illustrate performance improvements with noise perturbation, it is unclear how to choose the magnitude of the injected noise, which is not fully explored.", "methods_and_evaluation_criteria": "The use of sieve MLE and the derivation of convergence rates in both the Hellinger and Wasserstein metrics directly address the statistical challenges of estimating nearly singular conditional distributions. This design effectively tackles the curse of dimensionality by focusing on intrinsic dimensions for conditional generative models. \nThe convergence rates in Hellinger and Wasserstein distances are used and are appropriate for assessing the quality of conditional density estimation. These metrics capture both the pointwise differences and the overall transport cost between the estimated and true distributions, which is critical when dealing with singular supports and avoid the potential issues posed in KL divergence. To this end, both the methods and evaluation criteria are thoughtfully chosen to match the problem’s demands.", "theoretical_claims": "The proofs for the key theoretical claims are largely correct and built on well-established methods, though they rely on strong assumptions. \nMoreover, the smoothness theorem is based on earlier work in the literature. While the authors’ application of these results appears appropriate, a more detailed verification could reveal sensitivities regarding constant factors or the exact rate dependencies.", "experimental_designs_or_analyses": "I reviewed the experimental designs and analyses described in the paper. Here are the key points along with some issues:\n\n- Most experiments use synthetic data. While these controlled experiments support the theoretical claims, more extensive testing on diverse real-world datasets would help demonstrate the method’s robustness and practical utility.\n\n- The experimental setup does not include an in-depth analysis of how variations in key hyperparameters (network depth, sparsity, noise injection magnitude) affect performance. Understanding these sensitivities would be beneficial, especially since the theoretical convergence rates depend on noise and smoothness parameters.\n\n- The evaluation is mainly based on MSE and Wasserstein distance. Additional evaluations, such as those used in uncertainty estimation, and additional baseline methods to compare with, could provide further insights into the performance of the conditional density estimates.", "relation_to_broader_scientific_literature": "The work builds on deep generative models and related likelihood-based approaches to estimate complex conditional distributions. This approach leverages a conditional generator modeled by deep neural networks to capture the data’s structure.\n\nThe derivation of convergence rates in both the Hellinger and Wasserstein metrics connects with a rich body of work in optimal transport. This naturually formalizes the benefits of small noise injection in learning nearly singular distributions, which has been practically used in generative modeling, such as training GANs with additive noise (e.g., with diffusion schedule), training autoregressive models with noisy distribution.\n\nBy analyzing deep generative models with composite structures and considering both sparse and fully connected network classes, the paper generalizes earlier findings on deep network approximation capabilities. This demonstrates that the framework can adapt to a broad class of conditional distributions, including those that are nearly singular, which is a significant extension over standard density estimation methods.", "essential_references_not_discussed": "CARD is a method motivated to address regression problems in a view of conditional generative models. It also provides several synthetic and real-world empirical justifications, which should be considered to compare to enhance the soundness of this paper. \n\nHan, X., Zheng, H., & Zhou, M. (2022). Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 35, 18100-18115.", "other_strengths_and_weaknesses": "The font of text and the plotted shapes in Figure 2 and 3 seem to be distorted, making the figure hard to read.", "other_comments_or_suggestions": "Please see the weakness above.", "questions_for_authors": "Please see the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper considers conditional generative likelihood models in high dimensional problems where the data lives on a low dimensional manifold. It presents several theoretical results to analyze the proposed method and some basic experiments to validate their model. Three reviewers all leaned towards accepting the paper, but were not strong in their confidence. While they expressed some lack of confidence in their ability to evaluate the theoretical contributions, these seem correct and well-developed from the standards of ICML. The main concern was with the empirical evaluation, which focused more on demonstrating the proposed method than how it compares with others and are not stress tested, for example the data considered is essentially toy data and not the realistic high dimensional data that the paper claims to care about. However, considering the impressive theoretical content, the paper can contribute to the conference by providing insight on the analytical properties of conditional generative models, which would make the paper worth accepting.
Decision: Accept (poster)