A Theory for Conditional Generative Modeling on Multiple Data Sources
Authors: Rongzhen Wang
Yan Zhang
Chenyu Zheng
Chongxuan Li
Guoqiang Wu
Venue: ICML 2025 poster (2025)
Keywords: multiple data sources
generative model
distribution estimation
MLE
Primary area: theory->learning_theory
TL;DR: We analyze distribution estimation of conditional generative modeling on multiple data sources from the perspective of statistical learning theory.

Abstract: The success of large generative models has driven a paradigm shift, leveraging massive multi-source data to enhance model capabilities. However, the interaction among these sources remains theoretically underexplored. This paper provides a first attempt to fill this gap by rigorously analyzing multi-source training in conditional generative modeling, where each condition represents a distinct data source. 
Specifically, we establish a general distribution estimation error bound in average total variation distance for conditional maximum likelihood estimation based on the bracketing number.
Our result shows that when source distributions share certain similarities and the model is expressive enough, multi-source training guarantees a sharper bound than single-source training.
We further instantiate the general theory on conditional Gaussian estimation and deep generative models including autoregressive and flexible energy-based models, by characterizing their bracketing numbers. 
The results highlight that the number of sources and similarity among source distributions improve the advantage of multi-source training. 
Simulations and real-world experiments validate our theory.

A Theory for Conditional Generative Modeling on Multiple Data Sources
Rongzhen Wang" 123
Yan Zhang*
Chenyu Zheng
123 Chongxuan L23 Guoqiang Wu
Abstract
key factor behind their strong performance is the diverse and rich training data. For instance, large language models are trained on heterogeneous datasets comprising web content, books, and code (Brown et al., 2020: Hu et al. 2024b). while image generation models benefit from vast datasets spanning various categories and aesthetic qualities (Peebles & Xiel 2023 Chen et al.] [022]] [Esse et al., (022)) Empirical evidence suggests that, under certain conditions, training multiple data sources can enhance performance across
The success of large generative models has driven paradigm shift, leveraging massive multi-source data to enhance model capabilities. However, the interaction among these sources remains theoreti- cally underexplored. This paper takes the first step toward a rigorous analysis of multi-source train- ing in conditional generative modeling, where each condition represents a distinct data source. Specifically, we establish a general distribution es- timation error bound in average total variation dis- tance for conditional maximum likelihood estima- tion based on the bracketing number. Our result shows that when source distributions share certain similarities and the model is expressive enough, multi-source training guarantees a sharper bound than single-source training. We further instanti- ate the general theory on conditional Gaussian estimation and deep generative models including autoregressive and flexible energy-based models, by characterizing their bracketing numbers. The results highlight that the number of sources and similarity among source distributions improve the advantage of multi-source training. Simulations and real-world experiments validate our theory. Code is available at: https: /ttp://///thtuutt/////p com/ MT. GSAT Mu t. Source- GM
all sources (Pires et al. 2019] Chen et al. [024 Allen-Zhu & Li 2024a). Consequently, data mixture strategies have become an essential research topic (Nguyen et al. 2022
Chidambaram et al. 2022 Hu et al. 2024b).
However, the theoretical underpinnings of this multi-source training paradigm remain poorly understood. This raises a fundamental question: is it more effective train separate models on individual data sources, or to train a single model using data from multiple sources? In this paper, we take the first step toward a rigorous analysis of multi-source training, focusing on its impact on conditional generative models, where each condition represents a distinct data source.
Our first contribution is establishing general upper bound distribution estimation error for conditional generative modeling via maximum likelihood estimation (MLE) in Section B Specifically, we measure the error using average total variation (TV) distance between the true and estimated conditional distributions across all sources, which scales as Ö O(y /lg.Noo /n), where is the training set size and Nxx is the bracketing number of the conditional distribu- tion space Pxxx Further, when source distributions exhibit parametric similarity, multi-source training effectively re- duces the complexity of the distribution space, leading to a provably sharper bound than single-source training.
1. Introduction
Large generative models have achieved remarkable success in generating realistic and complex outputs across natural language (Brown et al. 2020 Touvron et al., 2023) and
computer vision (Rombach et al. 2022 OpenAl 2024). A
Technically, our analysis extends classical MLE estimation error bounds from empirical process theory (Wong & Shen 1995 Geer 2000; Ge et al., 2024) to the conditional set- ting by adapting the complexity of the distribution space and measuring the estimation error in terms of average TV distance. Further discussions are provided in Section
'Gaoling School of Artificial Intelligence, Renmin Univer- sity of China, Beijing, China "Beijing Key Laboratory of Re- search on Large Models and Intelligent Governance 'Engineering Research Center of Next-Generalie ommendation, MOE 'School of Software, Shandong University, Shandong. China. *Conrespondence to: Chongxuan <chongxu- anli @ ruc. edu.cc>>. Guoqiang Wu <guoqiangwu @ sdu.edu.en>.
Intelligent Search and Rec-
As the second contribution, instantiate our general the- ory in three specific cases: (1) parametric estimation of conditional Gaussian distributions, a simple example clearly
Proceedings of the 4 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
illustrating how source distribution properties influence the benefits of multi-source training, (2) autoregressive models
use a[m] denote the m-th entry of vector a, and A Alm, :). A [ m). and A [m, n| to denote the m-th row, the n-th col- umn, and the entry at the m-th row and the n-th column of
(ARMs), the foundation of large language models (Brown
et al.. 2020 Touvron et al. 2023 Liu et al. 2024 Bai et al.
A. (a. b) denotes the concatenation of a and b as column vector. We denote [n] = (1,...,,) n for any n. E N and a V b as max(a, 6). For any measurable scalar function f(x) on domain X and real number S p S 00, its - is defined as [=(( == (011/l(((e (a:) dz) When p= 00, [z) = sup=ex (f))) I(-) denotes the indicator function. Notation a, = (b,,. indicates a, is asymptoti- cally bounded above by b, up logarithmic factors.
2023 Zheng et a 2024). and (3) energy-based models
(EBMs), a general class of generative models for continu-
ous data (LeCun et al. 2006 Du & Mordatch 2019 Song & Ermon 2019 Zhao et al. 2022). For each model, we derive explicit estimation error bounds for both multi-source and single-source training by measuring the bracketing number of the corresponding conditional distribution space. Based on the theoretical results in these instantiations, we observe a common pattern: across all cases, the ratio of multi-source to single-source estimation error bounds takes the form VI- (K- 1/K))sam where K is the number of sources and Baim E [0, 1] is an inductively derived quantity that can be interpreted as similarity among source distributions, with model-specific definitions detailed in Section J This ratio decreases with both K and Saim. indicating that the number of sources and their similarity improve the benefits of multi-source training.
2.1. Data from multiple sources
Let X denote the random variable for data (e.g., a natural im- age) in data space X, and Y denote the random variable for the source label in a label space y. Suppose there are K data sources (e.g.. K categories of images), each corresponding to an unknown conditional distribution Px for & E [K]. We assume that P is parameterized by a source-specific feature of in a parameter space & and a shared feature v6 in parameter space V. such that P' (zrkk = P (x)k). The conditional distribution of X given Y = y is conse- quently expressed as
A core technical contribution is establishing novel bracket- ing number bounds for ARMs and EBMs. This is challeng- ing since the one hand, the bracketing number provides a refined measure of function spaces, requiring carefully designed one-sided bounds over the entire domain. On the other hand, the conditional distribution space of deep gener- ative models is inherently complex, involving both neural network architectures and specific probabilistic characteris- ties for different generative modeling methods. Please refer to Appendixes C and D for detailed proofs and discussions.
K II Poi - (x1k)) T()RR
(xlv)
[I
This compact representation provides convenience for sub- sequent discussions.
We further assume the distribution of is known since the proportion of data from different sources is often manually designed in practice (Deng et al. 2009] Krizhevsky et al., 2009 Brown et al.) 2020: Chen et al. 2024). The joint distribution of X and Y is then given by Px,y(2x, v) Px|y (xly)pp
Finally, we validate our theoretical findings through simu- lations and real-world experiments in Section In simula- tions, we perform conditional Gaussian estimation, where the MLE solutions can be analytically derived, enabling a rigorous assessment of the tightness of our bounds. The close match between the empirical and theoretical error orders supports the validity of our results. Beyond simu- lations, we train class-condition diffusion models (Karras let al. 2024) on ILSVRC2012 (Russakovsky et al. 2015 where its semantic hierarchy (Deng et al., 2010)) provides a natural way to define inter-source distribution similarity. Empirical results confirm that multi-source training outper- forms single-source training by achieving lower FID scores, consistent with our theoretical guarantee in Section 3 and this advantage depends on both the number of sources and their similarity, supporting our insights in Section 4
2.2. Conditional generative modeling
Consider a dataset S = {(x;, Hi consisting of n inde- pendent and identically distributed (i.i.d) data-label pairs sampled from the joint distribution p'.y- In the learning phase, a conditional generative model uses maximum like- lihood estimation (MLE) to estimate pXyy based on the dataset S, where the conditional likelihood is defined
n
Ls(exi)) II PXY (#+ ly:).
2. Problem formulation
Multi-source training. Under multi-source training, the conditional distribution space is given by Pxpp
Elementary notations. Scalars, vectors, and matrices are denoted by lowercase letters (e.g., a), lowercase boldface letters (c.g.. a), and uppercase boldface letters (e.g., A). We
(xly) = { (xly))
=Ik K (po (x) ((==k) :ou E D }
2

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
and the corresponding estimator of Px|y is
3. Provable advantage of multi-source training
gmalti PX|Y
arg max Plunt Y
(
(2)
In this section, we establish a general upper bound on the average TV error for conditional MLE and provide a statis- tical guarantee for the benefits of multi-source training. Our analysis extends the classical MLE guarantees (Geer) 2000 Ge et al., 2024), which leverage the bracketing number and the uniform law of large numbers.
Here, we adopt the realizable assumption that true parame- ters satisfy o% E ① and v/* E V following (Ge et al. 2024). which allows the estimation error analysis to focus on the generalization property of the distribution space.
Single-souree training. Under single-source training, we train K conditional generative models for each source using data exclusively from the corresponding source. For any par- ticular source kt denoting S₁ == {( 1s) E Sly = k) { k) -- the corresponding generative model estimate P by maximizing the conditional likelihood on S as
3.1. Complexity of the conditional distribution space
We begin by introducing an extended notion of the bracket- ing number as follows.
Definition 3.1 (e-upper bracketing number for conditional distribution space). Let be real number that > 0 and be an integer that 1 S S co. An e-upper bracket of a conditional distribution space Pxly with respect to LP(X)) is a finite function set B such that for any Pxjy E PxxY. there exists some E B such that given any y y. it holds
single Px
single). arg max Cs. (PX(R single Epingg PxIE
where Ls, (PxIM) = I (x(x(a))))) and xik (Pow.e, (arkk) : Φb E $ the v.
Væ E X : (1x v) 2 PXIY (xlv), and Il'' PXIY (-1v) S..
Separately maximizing these K objectives is equivalent to finding the maximizer of Ls in conditional distribution space P
The c-upper bracketing number Ni (c:Pxiv, IP(2)) is the cardinality of the smallest euupper bracket.
This notion quantifies the minimal set of functions needed to upper bound every conditional distribution within a small margin, reducing eтror analysis from an infinite to a finite function class. Unlike traditional bracketing numbers for un- conditional distributions px using two-sided brackets (Well- ner 2002), this extension employs one-sided upper brack- ets GG et al. (024) and requires uniform coverage across y for all conditional distributions tailored for our setting. We provide a concrete example and corresponding visualization in Appendix G to make this notion more accessible.
single (a =I[ - (Pon.en (a:lk:) [(y=k) th E )
Therefore, the estimator of pxlx under single-source train- ing is
PXlY
arg max Cs(pinn)e)
(3)
PXIY XXX
The introduced multi-source setting abstracts practical sce- narios where different sources share certain underlying data structures (via v) while retaining unique characteristics (via σx). At the same time, the single-source setting provides controlled comparison to rigorously assess whether incorpo- rating other sources improves the model's learning.
3.2. Guarantee for conditional MLE
We now present general ertor bound which applies to both training strategies.
Theorem 3.2 (Average TV error bound for conditional MLE, proof in Appendix A.Ipp Given a dataset S of size n that i.i.d. sampled from p'x.y- let px be the maximizer of Ls(xx(t) defined in Equation (T) in conditional distribu- tion space Pxly. Suppose the real conditional distribution Pxjy is contained in PxlY. Then, for any 0 < 6 S 1/2, it holds with probability at least 1 - that
Evaluation for conditional distribution estimation. We measure the accuracy of conditional distribution estimation by average TV distance between the estimated and true conditional distributions, referred to as average TV error:
RTv(xxIx) = By PXII)
(4)
where TV xx -xxxx (xxl v)ly)dd
1
log log.Ni - PxiY, L (12) 1
RTv((xxt) S3
is the total variation distance between pxly and P
In the following sections, we investigate the effective- ness of multi-source training by measuring and comparing (ppplit) (amulti) and (P
As formulated in Equation 2 and Equation 3 multi- source and single-source training apply conditional MLE on
Rap
3

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
S within different conditional distribution spaces, The fol- lowing proposition shows that multi-source training reduces the bracketing number of its distribution space through source similarity.
Theorem 4.1 (Average TV error bound for conditional Gaussian estimation under multi-source training, proof in Appendix B.2). Let Y be the likelihood maximizer de- fined in Equation 2 given Pxyv with conditional dis- tributions as in Equation 5 Suppose & B)4, T -B 1- B]d-d with constant B > 0, and 0% E + v E V. Then, for any 0 < d S 1/2, it holds with probubil-
Proposition 3.3 (Multi-source training reducing complex- ity. proof in Appendix A.2). ). Let P and PxIyy be as 2
defined in Section we have
Then, for any E > 0 and S S co,
ity at least 1 8 that
(i(PPMi PXpi IP(X)) s SN (Ni(s; (e x IP(Z)).
RT (pmult)) = =0(1
(K- 1)d1 +
n
Combining Theorem 3.2 and Proposition 3.3 we conclude that when source distributions have parametric similarity and the model satisfies the realizable assumption, multi- source training can enjoy a sharper estimation guarantee than single-source training. Simulations and real-world experiments in Section support our result.
In contrast, single-source training results in an error of R (P (Pipp) = (Kdjn). with a formal result pro- vided in Theorem [B.2] The advantage of multi-source learning can be quantified by the ratio of error bounds: [K-1)ditd Kd -V -
K-1d-d.
K
4. Instantiations
d-d. Letting Saim = d-d1, where d represents the propor- tion of the shared mean dimensions relative to the total dimensionality, this quantity Asim can thus be interpreted as the similarity among source distributions. As we will see in subsequent instantiations, this general form of ratio
We now apply our general analysis to conditional Gaus- sian estimation and two deep generative models to obtain concrete error bounds.
4.1. Parametric estimation on Gaussian distributions
VI V1-Ax K-1 Batm applies across Section 4.2 and Section 4.3 with Paim instantiated in a case-specific manner. Further dis- cussion the notion of Baim and the measure of distribution similarity in practice can be found in Appendix
K
As employed in extensive work (Montanari & Saeed 2022
Wang & Thrampoulidis 2022 He et al. 2022 Zheng et al. 2023b Dandi al., 2024; Zheng et al. 2023a), Gaussian models provide a simple yet insightful case for illustrating the benefits of multi-source training and enable analytically tractable simulations under our theoretical assumptions.
Notably, this ratio decreases with both the number of sources and source similarity Asim- As K increases from 1 to oo, the ratio decreases from 1 to /1 - -ssim. and as Baim increases from 0 (completely dissimilar distributions) to 1 (completely identical distributions). it decreases from 1 to V1/K, reflecting a transition from no asymptotic gain to constant improvement. This highlights that the number of sources and distribution similarity enhance the benefits of multi-source training. Empirical results in Section 5.2 confirm this trend.
Parametric distribution family, Suppose each of the K conditional distributions is a d-dimensional standard Gaus- sian distribution, i.e.,
Vk E [K], Xik- ~ (( Ia) = (2m)- 2 1-=1-==-ill
with a mean vector 1e and an identity covariance matrix Ia E Rdxdd We assume each mean vector has two parts: the first di entries 141(1 di] represent the source-specific feature which is potentially different for each source, and the remaining entries م (d: + 1 : d] represent the shared feature which is identical across all sources. Corresponding to the general formulation in Section 2 we denote
4.2. Conditional ARMs on discrete distributions
For deep generative models, our formulations are based on multilayer perceptrons (MLPs), a fundamental network component, with potential extensions to Transformers and convolution networks with existing literature (Lin & Zhang
dx = म [1 : di), a = pild, +1:d+ =...:........ -...--kkle (d.+1:d],
2019 Ledent et al.) Shen et al.
2021 Hu et al. 2024a Trauger & Tewari 2024 Jiao et al.
2024). We formally de-
and the conditional distribution is parameterized as (alk) = (2r) E à (ol
fine ML.s mainly following notations in Oko et al. (0023
(5)
Definition 4.2 (Class of MLPs). A class of ML.s
F(L, W, S, B) with depth L, width W, sparsity S, norm B, and element-wise ReLU activation that RcLU(x) = 0 x
Statistical guarantee of the average TV error. In this formulation, the conditional MLE in Pxl under multi- source training leads to the following result.
is defined as F(L, W. S, B) == =f(()= = (A (eLU)))(() +
b(L)) ... (A (1) + b()) : {(A()) (()) )
4

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
W(L, W, S, B)}, where parameter space WLL, S, B) is
To ensure no components related azd is seen when esti- mating the conditional probability for wd, mask v bg using (D dd-dimensiooaa zero vector Op-d as
defined by W(L, [T,,, W, S, B) = {{(A(), b(9) ) 1=1
A
E
R 60) E R Wi, max; Wi 2 W, E - A +
(o)) S S, max: A | xo V || xc I B}.
10p-d Ao
Ao bo d] 0b-4
We now present the formulation for ARMs, which can be viewed as extension of Uria et al. (0166).
Then we calculate the distribution parameter vector by an MLP fw E F(L, S, B) with Wo = D and WL = M, followed by a softmax layer as
Probabilistic modeling with autoregression. Consider a common data scenario for the natural language where X represents a D-length text [M].. Each dimension of X an integer token following an --categorical distribution with MM being the vocabulary size. Adopting the autore- gressive approach of probabilistic modeling, conditional distribution Pxjy (mly)) is factorized using the chain rule as:
pa ( = softmax fw (u (Evs, vx(ax
This leads to conditional distribution as
(x = p( ... po( (6)
p(æly) = p(n (v) ... p(pn)eenn))
When training such an ARM, each row of Vy is only op- timized data with the corresponding condition, while parameters in Vx, Ao, boo and w are optimized on data with all conditions. That means [y|k, :] serves as the source- specific parameter, while other parameters are shared across all sources. Corresponding to the general formulation in Section 2 we denote
= =p(x) ... p(ED: p(x=D.)))
We omit the subscripts for notation simplicity. Here, for any E [D]. p(x=d: y) is the distribution parameter for Xd given Xea, Y <d that
p(za= = ma 8) = p( p(xm))
Φ = [k and t = (Vx, Ao, w].
satisfying p(x y) E RMM and EM pla<a fm(m] =
Statistical guarantee of the average TV error. In this formulation, the conditional MLE in Pruki under multi- source training leads to the following result.
Distribution estimation via neural network. Aligning with common practices, we suppose the distribution parame- ter vector is estimated with pa (I y) using a shared neural network parameterized by 0 across all dimensions. The net- work comprises an embedding layer, an encoding layer, an ML. block, and a softmax output layer.
Theorem 4.3 (Average TV error bound for ARMs under
multi-source training, proof in Appendix C..4p. Let pamlti be the likelihood maximizer defined in Equation 2 given Pmmulti with conditional distributions in Equation 6
Specifically, we first look up æ and y in two embedding matrices Vx [0, 1] Mxd and Vy e [0, 1] Kxd, then stack the embeddings to get
Suppose + = =00, 1]d, = [0, 1] xd x [-B, B) Dxde X
[-B, BJ xW(L, W, S, B) with constants L, W, S, B > 0,
and o% E b, 1p" E V. Then, for any 0 8 < 1/2, holds with probability least 1 8 that
Vrlu,: Vxlai,
Ev
E [0, Dxd.
(D+ (K) K)d.)
R RTv Gamultii Ô
Vx [x[
n
where the last dimension of is excluded since it is not used when estimating the distribution.
In contrast, single-source training results in an enror of ( osinglr) = D(VKT(++++D++ +11)d.)/n
Subsequently, we encode each embedding by linear trans- formation with parameters Ao I Dxde, by E IR' 2 and normalize the output with an element-wise sigmoid function (x) To as
with formal result provided in Theorem C.8 The ad- vantage of multi-source learning is quantified by the ratio of error bounds: L(S+D+(DD M K)d.)
K-1 Aaim.
KL(S-D+(D- M+11d.)
K
where the term Baim = 34D+(D+M+K)d E [0,
(DD+(D+MMd
1 bu EV,, vx (x. 2))
quantifies source distribution similarity based on the pro- portion of shared parameters. This ratio follows the same pattern discussed in [..tioon44]]hher the number of sources K and the distribution similarity Saim are two key factors improving the advantage of multi-source training.
(AolL, : +bn
E [0, 1)
a VX bbb
5

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
4.3. Conditional EBMs on continuous distributions
W,W, B 0 and assume &% E 4, v0 E V. Then, for any 0< it holds with probability least 1 1-6 that
In this section, we study distribution estimation for condi- tional EBMs, a flexible probabilisticc modeling approach continuous data. Our formulation follows Du & Mordatch (2019) with simplified neural network architecture.
LIS + Kdd))
RTV(PXIF (
π
Probabilistic modeling with energy function. Consider common scenario with natural image X flattened and nor- malized in [0, 1] The conditional distribution (x (y) is factorized with an energy function u(<ly) as:
In contrast, single-source training results in an error of R TV ( Ö (VIK(S+dd))) with formal proof provided in Theorem D.4 The advantage of multi- source learning is quantified by the ratio of error bounds:
[[S+K4.) V1- K-1 Asim. where Saim = $
e (((lp)
LK(S+d.)
S+d.
p(arl))
[0, 1] quantiffses source distribution similarity based on the proportion of shared parameters. Similar to the former two cases, the number of sources K and the distribution similar- ity 8aim improve the advantage of multi-source training.
Ix e e
Distribution estimation via neural network. We sup- pose the energy function is estimated with (xyy) using a neural network parameterized by 0, which comprises a condition embedding layer and an energy eetegaiist MLP.
5. Experiments
In this section, simulations and real-world experiments are conducted to verify our theoretical results in Section 3 and
Specifically, first look up y in condition embedding matrix V E [0, xj and concat the embedding with æ
5.1. Simulations on conditional Gaussian estimation
ev ev v)
10
V..
In this part, we aim to examine the tightness of the derived upper bound that R ( = Ö(V (K--)),+ in The-
Then we use an MLP fo F,L,W,,, with Wo D + de and Wr to estimate the energy as
orem 4.1 and R ( ssingle Ö(v Kd in Theorem B.2
up xx]x) = fw (ev(x,)))
The number of sources K, sample size n, and the similarity factor Bsim E [0, 1] key parameters. In all of simu- lations, we fix data dimension d 10 and py pk) all k E [K] The dissimilar dimension di = d - [Baimd]. We set the source-specific feature as Φ* k1 e Rd and the shared feature as g) = 0 E Rd--,, Under the setting of Section 4.1 conditional MLE has analytical solutions: under multi-source training, we have
where 0 = (V,W). This leads to conditional distribution
1/K
as
(o(x)))= = e (al(())
(7)
When training such EBM, each of V is only op- timized on data with the corresponding condition, while is optimized on data with all conditions. That means [lk, serves as the source-specific parameter and w is shared across all sources. Corresponding to the general formulation in Section 2 we denote
tu = Ey, 21 dil/nn & E ai + 1: d]/n,
and under single-source training, we have
dx - #:(1:di//nn hz= -y De (di + 1 1:dl/me
For evaluation, we randomly sample n test = 500 data points according to the true joint distribution Px,y- Empirically, we approximate the true TV distance by using the Monte Carlo method based on the test set, which can be written formally as
Φ :], and v = w.
Statistical guarantee of the average TV error. In this formulation, the conditional MLE in Pmmutii under multi- source training leads to the following result.
1 pesu
yxp (2: -1 RaV (px1x). Pxly (B4.)
Theorem 4.4 (Average TV error bound for EBMs under multi-source training, Proof in Appendix D.3). Let PX be the likelihood maximizer defined in Equation (2) given Sup-
(fx
2n
priultt with conditional distributions in Equation
To eliminate the randomness, we average over random runs for each simulation and report the mean results.
pose = [0, 10dh and Ni = W(L. W, S, B) with constants

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Empirical TV Error
0:20
Empirical (Multi)
Empirical (Single)
0.55
Theoretica Boune
0.25
Empirical (Multi)
Empirical (Single)
0.7
Thoorctical Bound
Empirical TV Error
0.11
Empirical (Single)
0.30
The0 Aerelc eiiical Boune
0:45
Enipirical (Multi)
0.15
Emppriial TTV
0.5
0.25
0.35
0.15
0.09
0.10
0:25
0.10
0.7
0.07
0.20
Theorctical (Single)
Theorstical (Single)
Thcoretical (Single)
0.05
Theoretical (Multi)
0.15
0.05
Theoretical (Multi)
0.05
Theoretical (Multi)
0.15
3
10
15
100
1000)
XIDI
0.0
0.3
0.5 0.7
1.0
(a) Number of Sources
(b) Sample Size
(c) Distribution Similarity
Figure/ 1. Simulation results for conditional Gaussian estimation. Empirical values (solid lines) correspond to the left vertical axis, while theoretical values (dashed lines) correspond to the right. Single-source results are shown in orange, and multi-source results in green. The matched orders of empirical errors and theoretical upper bounds support the validity of results in Section 4.1
Order of the average TV error about K. We range the number of sources K in [1,3, 5, 10, 15] with fixed sample size n 500 and similarity factor Saim 0.5. We display the empirical average TV error for each K in Figure [a), with Ty (PX])' colored in green and Ryy (Px]x col- ored in orange. Ignoring the influence of constants, it shows a good alignment between empirical errors (in solid lines) and theoretical upper bounds (in dashed lines), both scaling as Ö(VK).
and single-source training on conditional diffusion models and evaluate their performance to validate the guaranteed advantage of multi-source training against dingle-souree training proved in Section 3 (2) We investigate the trend of this advantage about key factors-the number of sources and distribution similarity- as discussed in Section 4
Experimental settings. We train class-conditional dif- fusion models following EDM2 (Karras et al.] (024) at 256x256 resolution on the selected classes from the ILSVRC2012 training set (Russakovsky et al., 2015). which is subset of ImageNet (Deng et 2009 containing 1.28M natural images from 1000 classes, each annotated with an integer class label from 1 to 1000. In our experiments, we treat each class as distinct data source. To control simi- larity among data sources, we manually design two levels of distribution similarity based on the semantic hierarchy of ImageNet (Deng et al. 2010; Bostock. (0188) as shown in Figure 2 in Appendix E. along with other experimental details.
Order of the average TV error about n. We range sam- ple size n in 300, 500, 1000, 5000] with fixed number of sources K 5 and similarity factor Seim 0.5. We display the empirical error for each in Figure[[b, with RTv (( colored in green and RTV (Px))) colored in orange. Ignoring the influence of constants, it shows that the orders of empirical error about n match well with the theoretical upper bounds which scale as (1/Vn).
Order of the average TV error about Baim- We range similarity factor Sam in [o, 0.3, 0.5, 0.7, 1] with fixed sam- ple size n 500 and number of data sources K = 5. We display the empirical average TV error for each Baim in Figure (c) to observe how similarity factor 3aim impacts the ad vantage of multi-source training. Concretely, as pre- dicted by the theoretical bounds, the changing of Baim will not influence the performance of single-source training but will decrease the error of multi-source training the order of Ō(Vd)) = Ö(V1 3. The results show that the theoretical bounds predict the empirical performance well.
For each controlled experiment comparing multi-source and single-soune source training, we fix K target classes within one similarity level Sim and train the models on a dataset S consisting of examples per class. Under multi-source training, we train a single conditional diffusion model for all K classes jointly. Under single-source training, we train K separate conditional diffusion models, one for each class. Please refer to Section 2 for the formal formulation of these two strategies. We set each factor with two possible values: the number of classes K in 3 or 10, distribution similarity Sim in 1 or 2, and the sample size per class N in 500 or 1000. This results in a total of 8 sets of experiments comparing multi-source and single-source training.
To sum up, our simulations verify the validity of our theo- retical bounds in Section [.I Moreover, in all experiments, osingle Rap (2xp))) is consistently smaller than Rap ( ), sup- porting our results in Section 3
We evaluate model performance using the average Fréchet Inception Distance (Heusel et al. (017)) (FID, a widely used metric for image generation quality) across all conditions to assess the overall conditional generation performance. Results are displayed in Table I Specifically, for multi- source training, we compute the FID for each class and take
5.2. Real-world experiments on diffusion models
In this section, we conduct experiments on diffusion models to validate our theoretical findings in real-world scenarios from two aspects: (1) We empirically compare multi-source
7

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
ImageNet
Table Average FID for single-souree and multi-source training. Under different amounts of classes K, similarity level Sim, and per-class sample size N, multi-source training generally achieves lower average FID than that of single-source training, which is consistent with our theoretical guarantees derived Section
Dog
Vehicle
Italian
French
Jeep
Tank
greyhound bulldog
Avg. FID & (Single)
Avg. FID (Multi)
Sim
1
30.03
29.94
Sim=2
Sim=1
10
30.18
29.28
500
32.69
30.69
2
Figure 2. Similarity level in ILSVRC2012 training
10
30.54
28.75
3
28.01
26.41
K=3
1000
10
3
27.49
30.58
25.84
29.35
Advantage FID
0.04
K=10
Adventage on FID
0.12
K-10
0.088
0.10a
2
10
29.01
27.81
0.03
0.027
0.08
0.087
0.02
0.015
the average over all K classes. For single-source training, we compute the FID for each of the K separately trained models on their respective classes and calculate the average. Relative advantage of multi-source training is measured by Avg. FID (Single)- - Aug. FID (Muli)) displayed in Figure 3 Avg. FID (Single)
0.011
0.04
0,034
0.01
0.00
0.00
5tm
Sim=2
500
N-1000
Figure 3. Relative advantage of multi-source training. For any fixed similarity level Sim and per-class sample size N, a larger K yields a greater FID improvement than a smaller K. For any fixed K and N. higher distribution similarity leads greater FID improvement (illustrated by dashed lines). These results support the theoretical findings in Section 4
Experimental results In the following, we interpret the results sequentially from the view of our theoretical findings.
From T we observe that under different amounts of classes K, similarity level Sim, and per-class sample size N, multi-source training generally achieves lower average FID than that of single-souree training, which is consistent with our theoretical guarantees derived in Section
the realizable assumption. We further adapt their techniques to conditional generative modeling by introducing the upper bracketing number to quantify the complexity of conditional distribution space in Definition 3.1 and modify the proof' to handle conditional MLE in Appendix A.1
From Figure 3 we observe that for any fixed similarity level Sim and per-class sample size N, the relative advantage of multi-sources training with a larger K (the green bars) is larger than that with a smaller K (the nearby orange bars). Additionally, for any fixed K and N, the relative advantage of multi-sources training with larger distribution similarity larger than that with a smaller distribution similarity (as shown through the dashed lines). These results support our theoretical insights in Section that the number of sources and similarity among source distributions improves the advantage of multi-source training.
Theory on multi-task learning. Multi-task learning is well-studied topic in supervised learning (Caruana 1997 Baxter 2000). It typically benefits from similarities across tasks, sharing some commonality with multi- training, However, theoretical analyses in supervised learning often assume a bounded objective (Ben-David & Borbely 2008)
Maurer et al., 2016 Tripuraneni et al., (2220), whereas our MLE analysis imposes no such restriction.
6. Other related works
Advanced theory on generative models. Among gener- ative models based on (approximate) MLE (LeCun al.) 2006 Uria al. 2016 Ho et al. 2020), diffusion mod- els have been extensively studied theoretically its score approximation, sampling behavior, distribution estimation, and scalability (Oko et al. |023)|Chen Chen et al. [2023a] b Fu et al. 20224 /Zheng et al.] (025). This paper focuses on dis- tribution estimation for general conditional generative mod- eling. Incorporating existing literature could be a promising
Distribution estimation guarantee for MLE. Classical approaches investigate distribution estimation for MLE in Hellinger distance based on the bracketing number and the uniform law of large numbers from empirical process the- ory (Wong& & Shen, 1995] Geer 2000), which yields high- probability bounds of similar order as Theorem Ge et al. (2024) extend the analysis derive TV error bound under
8

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
direction for future work.
References
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.1, knowledge storage and extraction. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a.
7. Conclusion and discussion
This paper provides the first attempt to rigorously ana- lyze the conditional generative modeling on multiple data sources from a distribution estimation perspective. In par- ticular, we establish a general estimation error bound in average TV distance under the realizable assumption based on the bracketing number of the conditional distribution space. When source distributions share parametric sim- ilarities, multi-source training has a provable advantage against single-source training by reducing the bracketing number. We further instantiate the general theory on three specific models to obtain concrete error bounds. To achieve this, novel bracketing number bounds for ARMs and EBMs established. The results show that the number of data sources and the similarity between source distributions en- hance the benefits of multi-source training. Simulations and real-world experiments support our theoretical findings.
Allen-Zhu, Z. and Li, Y. Physies of language mod- els: Part 3.3, knowledge capacity scaling laws. CoRR, abs/2404.05405, 2024b.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., al. Qwen technical report, arXiv preprint arXiv:2309. 16609, 2023.
Balcan, M.-F., Khodak, M., and Talwalkar, A. Provable guarantees for gradient-baie based meta-learning. In Interna- tional Conference on Machine Learning, pp. 424-433. PMLR, 2019.
Bartlett, P. L.,, Foster, D. J. and Telgarsky, M. Spectrally- normalized margin bounds for neural networks. In Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 6240-6249, 2017.
Our thcoretical setting differs from practice in some as-
pects, e.g.. language models have no explicit conditions, and image generation models are commonly conditioned descriptive text involving multiple conditions. However, our abstraction provides a simplified framework that preserves the core properties of multi- source training and isolates how individual source distributions are learned. Moreover, re- cent studies suggest adding source labels, such as domain names, the start of training text for language models can enhance performance (Allen-Zhu & Li 2024b Gao al.
Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. Nearly-tight ve-dimension and pseudodimension bounds for piecewise linear neural networks. J. Mach. Learn. Res., 11:63:1-63:17 2019.
2025), which may become a standard practice in the future.
Baxter, J. A model of inductive bias learning. J. Artif. Intell. Res., 12:149-198, 2000.
Acknowledgments
Ben-David, S. and Borbely, R. S. A notion of task related- ness yielding provable multiple-task learning guarantees. Mach. Learn., 73(3):273-29) 2008.
This work was supported by NSF of China (Nos. 92470118, 62206159); Beijing Nova Program (20220484044); Bei- jing Natural Science Foundation (L247030); Major Innova- tion& & Planning Interdisciplinary Platform for the "Double- First Class" Initiative, Renmin University of China; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (22XNKJ13); the Natural Science Foundation of Shan- dong Province (ZR2022QF117) the Fundamental Research Funds of Shandong University. G. Wu was also sponsored by the TaiShan Scholars Program (NO..sqn202306051,
Bostock., M. Imagenet hierarchy, 2018 URL
https://observablengg wtt.:/ooommmmcecee @mbostock/ imagenet-hieraan
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voos A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J.. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
2020, December 6-1 2020, virtual, 2020.
Caruana, R. Multitask learning. Mach. Learn., 28(1):41-75, 1997.
9

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Chen, J., Yu, J., Ge, C.,, Yao, L., Xie, E., Wang, Z., Kwok, J. T., Luo, P., Lu, H., and Li, Z.. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
image synthesis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Fu, H.,, Yang, Z., Wang, M., and Chen, M. Unveil condi- tional diffusion models with classifier-free guidance: A sharp statistical theory. CoRR, abs/2403. 11968, 2024.
Chen, M., Huang, K., Zhao, T., and Wang, M. Score approx- imation, estimation and distribution recovery of diffusion models on low-dimensional data. In Krause, A., Brun- skill, E.,, Cho, K., Engelhardt, B., Sabato, S., and Scarlett, (eds.), International Conference on Machine Learn- ing, ICML 2023. 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 4672-4712. PMLR, 2023a.
Gao, T., Wettig, A., He, L., Dong, Y., Malladi, S., and Chen, D. Metadata conditioning accelerates language model pre-training. arXiv preprint M0Kiv:2501.0995555 2025.
Ge, J., Tang, S., Fan, J., and Jin, C. On the provable advan- tage of unsupervised pretraining. In The Tweifth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Chen, S., Chewi, S., Li, J., Li, Y. Salim, A., and Zhang, A. Sampling as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b.
Geer, S. A. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
He, H., Yan, H., and Tan, V. Y. Information-theoree char- acterization of the generalization error for iterative semi- supervised learning. Journal of Machine Learning Re- search, 23(287): 2022.
Chidambaram, M., Wang, X., Hu, Y., Wu, C.,, and Ge, R. Towards understanding the data dependency of mixup- style training. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Dandi, Y., Stephan, L.., Krzakala, F., Loureiro, B., and Zde- borová, L. Universality laws for gaussian mixtures in generalized linear models. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020. NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei- Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Hu, J. Y.-C., Wu, W., Lee, Y.-C., Huang. Y.-C., Chen, M., and Liu, H. On statistical rates of conditional diffusion transformers: Approximation, estimation and minimax optimality, arXiv preprint arXiv:2411.1752 2024a.
Deng, J., Dong, W., Socher, R. Li, L..J., Li, K., and Fei-Fei, L. Imagenet summary and statistics, 2010. URL https://tex stackexchange. com
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z. Fang. Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm: Unveiling the potential of small language models with scalable training strategies. CoRR, abs/2404.06395. 2024b.
aebll----wwebpppa- page
Du. Y. and Mordatch, Implicit generation and modeling with energy based models. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché Buc, F., Fox, E. B.. and Garnett, R. (eds.), Advances in Neural Information Pro- cessing Systems 32: Annual Conference on Neural Infor- mation Processing Systems 2019, NeurlPS 2019, Decem- ber 8-14, 2019, Vancouver, BC, Canada, pp. 3603-3613, 2019.
Jiao, Y., Lai, Y., Wang, Y., and Yan, B. Convergence analysis of flow matching in latent space with transformers. arXiv preprint arXiv:2404.02533, 2024.
Esser. P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., and Rombach, R. Scaling rectified flow transformers for high-resolution
Jose, S. T. and Simeone, O. An information theooreti anal- ysis of the impact of task similarity on meta-learning. 2021 IEEE International Symposium on Information Theory (ISIT), pp. 1534-1539, IEEE, 2021.
10

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamies of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
Ou, W. and Bölcskei, H. Covering numbers for deep relu networks with applications to function approximation and nonparametric regression. CoRR, abs/2410.06378, 2024,
Peebles, W. and Xie, S. Scalable diffusion models with transformers. IEEE/CVE International Conference on Computer Vision ICCV 2023, Paris, France, October 1-6, 2023, pp. 4172-41888 IEEE, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F., et A tutorial on energy-based learning. Predicting structured data, 10)) 2006.
Pires, T., Schlinger, E., and Garrette, D. How multilingual is multilingual bert? In Proceedings of the 57th Confer- ence of the Association for Computational Linguistics, ACL. 2019, Florence, Italy, July 28- August 2019, Vol- ume 1: Long Papers, pp. 4996-5001, Association for Computational Linguistics, 2019.
Ledent, A., Mustafa, W., Lei, Y., and Kloft, M. Norm-basee generalisation bounds for deep multi-class convolutional neural networks. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Confer- on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 6279-8287. AAAI Press, 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, IA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022.
Lin, S. and Zhang, J. Generalization bounds for convolu- tional neural networks. arXiv preprint arXiv:1910.01487, 2019.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-25, 2015. doi: 10. 11033/1112330010 0112263--1---0 015-0011--886--
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C.,, Ruan, C., et al. Deepseek-v? v3 technical report. arXiv preprint urXiv:2412.19431 2024.
Salimans, T. and Ho, Should ebms model the energy or the score? In Energy Based Models Workshop-ICLR 2021, 2021.
Maurer, A., Pontil, M., and Romera- Pareeee, B. The benefit of multitask representation learning. J. Mach. Learn. Res., 17:81:1-81:33 2016.
Shen, G. Exploring the complexity of deep neural networks through functional equivalence. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024. Vi- enna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Montanari, A. and Saeed, B. N. Universality of empirical risk minimization. In Conference on Learning Theory, PMLR, 2022.
pp.
Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. Quality not quantity: On the interaction between dataset design and robustness of CLIP. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Shen, G., Jiao, Y., Lin, Y., and Huang, J. Non-asymptotic excess risk bounds for classification with deep convolu- tional neural networks. arXiv preprinl arXiv:2 205.00992..2 2021.
Song, Y. and Ermon, S. Generative modeling by estimat- ing gradients of the data distribution. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buss F., Fox, E. B., and Gamettt R. (eds.), Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1 11895-11977, 2019.
Oko, K., Akiyama, S., and Suzuki, T. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu. Hawaii, USA, volume 202 of Proceed- ings of Machine Learning Research, pp. 26517-26582. PMLR, 2023.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In 9th Interna- tional Conference on Learning Representations, ICLR
OpenAL. Video generation models as world simula- tors. www.://wwwwwiiiddooooctiiinoooon as-world-simmlltoos/, 2024.

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021.
Zheng, C., Wu, G., and Li, C. Toward understanding gener- ative data augmentation. Advances in neural information processing systems, 36:54046-54060, 2023b.
Suzuki, T. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Zheng, C., Huang, W., Wang. R., Wu, G., Zhu, J., and Li,
On mesa-optimization in autoregressively trained trans- formers: Emergence and capability. Advances Neural Information Processing Systems, 17:49081-49122 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
Zheng. C., Zhang, X., Wang, R., Huang, W., Tian, Z., Huang, W. Zhu, J., and Li, C. Scaling diffu- sion transformers efficiently via ppp arXiv preprint arXiv:2505.15270,20000 2025.
Trauger, and Tewari, A. Sequence length independent norm-based generalization bounds for transformers. In International Conference Artificial Intelligence and Statistics, pp. 1405-1413 PMLR, 2024.
Tripuraneni, N., Jordan, M. I., and Jin, C. On the theory of transfer learning: The importance of task diversity. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Informa- tion Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Uria, B., Côté, M., Gregor, K., Murray, I., and Larochelle, H. Neural autoregressive distribution estimation. J. Mach. Learn. Res., 17:205:1-205:37, 2016.
Wang, K. and Thrampoulidis, C. Binary classification of gaussian mixtures: Abundance of support vectors, be- nign overfitting, and regularization. SIAM Journal on Mathematics of Data Science, 4(1):600-244 2022.
Wellner, J. A. Empirical processes in statistics: Methods, examples, further problems, 2002.
Wong, W. H. and Shen, X. Probability inequalities for likelihood ratios and convergence rates sieve mles. The Annals of Statistics, pp. 339-362, 1995.
Xie, S. M., Pham, H., Dong, X., Du N., Liu, H., Lu, Y., Liang, S., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training. Advances in Neural Information Processing Systems, 36:69798-699 69818 2023.
Zhao, M. Bao, F., Li, C., and Zhu, J. Egsde: Unpaired image-to-image translation via energy snergy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
Zheng, C., Wu, G., Bao, F. Cao, Y., Li, C., and Zhu, J. Re- visiting discriminative vs. generative classifiers: Theory and implications. In International conference on machine learning, pp. 42420 42477. PMLR, 2023a.
12

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
A. Proofs for Section 3
A.1. Proof of Theorem 3.2
Proof of Theorem 3.2 This theorem applies to both discrete and continuous random variables, while we use integration
notation the proof for generality. In the following, we first present an elementary inequality (in Equation (TO) which serves toolkit for the subsequent derivations. Then decompose the TV distance and derive its complexity- upper bound (in Equation
[22 using the former inequality. Finally, after specifying certain constants in this upper bound, clearer order w.r.t is revealed (in Equation
[3
Intermediate result induced by union bound. Let be real number that > 0 and be integer that p S co. Let B be an c-upper bracket of Pxiy w.r.t. T.* (XC) such that [B] = Ni (<((<) Pxix, (2)).
B. p' (2, v) 2 0 on Xx According to the minimum cardinality requirement, we obtain a proposition of B that: for any p' E B.
Let's first consider IT:=1 V (xi as random variable on S, where we suppose px,y (2, Us) > since (I ys) are
sampled from p and thus Px (mr: of O. By applying the Markov inequality, we have: given any 0 < 8 < 1,
p' ( y:) Piiy (x;
p' ( 31) p' (ax
Prs
2 8
Es
S 8.
(8)
Applying the union bound on all p' B, further have:
Prs
Vp' E B, II
( v1)
< Es
p' (x mn
PX (2:
pi Y (2r: lu:)
(2,,, v/)
n
p' (o 1ss) pxx1 (ce; (1:))
=1 Prs
3p E B,
z Es
px (y:)
8'
( y/) px x (2:())))
Ya
// ( y/) (ax (36))
=1 Prs
Es
p
WEB
p' (ai, wi) p Px
// (xx wi) p pxxy
21 Σ Prs
II
Es
(by union bound)
D'EB
21 N (N Px ('0))''
(by Equation (
L By denoting that 5 = (x ('3))' 8', have: it holds with probability at least - 8 that for all p' E B.
(x
12
1/ (x Ui)
T
pl (ar ya)
Es
(x
Px Px (2:
13

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Taking logarithms at both sides, we have
No No (((ePx)r,L
log (2 V/) Px (mi (16)
p' (z Mi) p (mr (11)
< log Es
+og
2
5
Ni PxIx, (x()
log
II i=1
B Bippor
x Uii 12 (ara lys)
log
8
((xi) are i.i.d. sampled from p
No c; (xxx I. (2(
p((x,)) 3)
n log Ex,y
+ +log
V P pxxx (20ly)
6
Nii c: P PXIr,
p((x, v) \V px)
= n log By
Exir
+ log
6
No PxIx, (2())
D p' v) (x
n log By
da
+ loog
n log Fy
1f wP' (a (arly)
+ log
5
As log S 1 for all x > O, the inequality can be further transformed into
Ni E; PxIx, (
log p' p(( vi)
S In Ey Us / (xr, (y -1
-1
+ log
(9)
2 1=1
P P
8
Elementary inequality for MIEE estimators.
Since the real conditional distribution px is in Px v. for the likelihood
maximizers PxIY E PxY. we have Ls(xx() = I (x.lu) > Ls(P'|r) = I Pxix (æ; ly:),, and thus (aslw) " log (m:ls)) _ log IT. (m:lve)
I Åx (m:
log 0. According the definition of upper bracketing number, there exists some p' B such that given any y E y, holds that: (i) Vr E X' (xx, v) 2 Px v). and (ii) (p
(xii(ll)) fx (=. pxlr dx S e. Applying (i), we have:
i (o 9i)
1
Dxly (xx(ye) log
log 1=1
2 0. Px (x (B)
2
P (z (x,
2
1=1
Combining this with Equation 9 and rearranging the terms, we have: holds with at least probability that
Nn Nn (x)
1 B- Ey Ug (1
1
log
(10)
n
This serves as an elementary toolkit for deriving the subsequent upper bounds.
Decomposing the square of the TV distance. Recalling that TV Px(x) fx- - -PX (æly) dx, we will decompose its square and then bound each term sequentially. First, we use the above 6xxx,) y) as an intermediate term to decompose the square of (TV( ,Xi()) into parts that can be effectively upper bounded:
(TV(EX).PXx))) - ( (xx(((ll)) (x
2 (Xxxxxx - pxIr p (wlw))do))
2 )²+ - ( li' (xc y) - pxIx (alw)ldx)
+
-X (x ld
(
14

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
For (I), have
( x
(xxxx((tlv) - (x)r) dx) )=)² - (1 - p
(xxx)( - (x + (a v) - px (xl1))d) (L (x) PxIx (xly)) 10'( y) - -P (all(()))))
(L (0xI)((a))) - pa v) da
[XxIx( + xxxx(()))+ (x, v) - Pxxr v)l 1ixpx + (xx
<( 4)e.
The first inequality holds for the triangle inequality |a + bl < |a| + |b] and the reverse triangle inequality ||a] - |6|l z
la - 6). The second inequality holds for the normalization property of conditional distributions (Jx Px (x]y) da and Ixx x (aly) equal 1) and the property of the c-upper bracket (/2 (xx y) -xxy (x) (aly))da S c).
For (II), have
(L (x, - (1
(xl)))) 2
VD((a,y) + (xll) dx) (VB(aa / ( 2
dr
(by Cauchy-Schward inequality)
(f'(x.)) + (x) dx) (f pl(a,s) + px (xly)) 2 f v)xxiy (xly)de
(by (a b) <2 2(a² + b))
(L (((2,)) - (xlv) + (xly) + pxxy (all)dar)
( p'(x0, v) -ix y (xll) + -xII (xlw) + p (xll)) - 2 ((( uwxx (ao )d)
<2(E + 2)( (2( (6+2-2) 2 f v u
(a (xll)dar
x
(by Pxlx (xlv))da = fox 12x1x (2t ldke = 1 and (x', v) - - 5)
Putting together (I) and (II), we get:
P
// (f (xII (xlw)
2
< 2 V (e (c+ 2(E + 2) () c+ 2+2-2 o Sov//VVrr (0((x)))xx1 (xly)dx)
(11)
15

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Bounding the average TV error. Based the above results, upper bound the average TV error (defined in Equa- tion of fxly follows:
R =By TV(xxIr.Xxxx>
V/ (c++)++ ( -2 / (2, u ( (dl
- Ey
2(c
(by Equation
2
6 - 2 L (a u)pxIy (alu)dar)
By (c (0+4)c+2((0))
21
(by concavity of f(x) = v and Jensen's 's inequality)
(e (e+ 2 +) c+2 1 By U, (1)xx)) ( 1)
1
21
(by the linearity of expectation)
Recalling the elementary inequality we derived formerly in Equation (10), we have: it holds with at least probability 1 - 8 that
Ni Mi(E:PxY,LL
1
2
R - (c+ +)+c++2(2++2) c+ log 2
(12)
n
8
Recalling that 8 1/2 and for non-empty P (;: Pxix ( (x)) 2 we have N1 ((: (xxxxx,, 18 18222 et.
Taking 1/n in Equation (12). it then holds with probability least 1 5 that
Px L(()
RTV(PxIV) v
+
4)
+
2(
+ 2)
- + log
2
n
5
No
4: Pxir, (23
5
1
2
$
+
log
(by 4/4 s 1)
2
11
71
11
Ng Px (2())
Ni
Px)x,,
1
10
4
log
+6
log
2
n
6
n
Ni P ((3))
(by < 2 log
Nu PHIx
Ni |:: Px L2))))
34
\
log
33
log
2
8
5
log.N1
P
1 llogg 5
3
(13)
71
Until now, we have completed the proof of this theorem.
16

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
A.2. Proof of Proposition 3.3
Proof of Proposition 3.3
As defined in Section
2 it holds that Puuiti X]YY" C Piipple Then, for any PX|Y
multi E P there
X|X
exists some P
single
Pxly such that Pxiy = Px|Y Given any c > and S p S oo, let geingle be c-upper bracket
single
_P()) ( According to the definition of c-upper bracket (as in w.r.t. LP(() for P such that [Beingle| =Ni (eippl Pxlx
Definition 3.1p there exists some p' E Briugle such that given any y E y, it holds that: Vx E It Xt,p (z. y) 2 PxlY PX| (rl)), and (::'(:))) - pXp Le(x)- = (:((()))))) - Px|y p (2(()) S c. Therefore, Bsingle is also a c-upper bracket w.r.t. (2)) for Pxpy' and thus Ni c:pxpy' . = |Beinglo| = Ni (:: Pxxi (I())
single (zly) =
B. Proofs for Section 4.1
B.1. Bracketing number of conditional Gaussian distribution space
According to Theorem 3.2 to derive the upper bound of average TV error, we need to measure the upper bracketing number for the conditional Gaussian distribution space. This result mainly follows the bracketing number analysis of Gaussian distribution space in Lemma C.5 in (Ge al. 2024), and slightly modifies it to conditional Gaussian distribution space.
Theorem B.1 (Bracketing number upper bound for conditional Gaussian distribution space under multi-source training). Let B be constant that 0 < B < co, suppose that 4 = [-B, BJ4, v [[-B, Bja-dd, and conditional distributions in Prppi are formulated as in Equation 5. Then, given any 0 < c < 1, the c-upper bracketing number of P w.r.t L' (X) satisfies
No Pxppii ( 1
(2(+ <2(L+ +1)
(K (K-1)d.+d
Proof. According to the assumptions, the conditional distribution space expressed by the parametric estimation model is
multi P
(x/l)) = o 1(v=k)
K
II k=1
K
(2m) 1 211== (6u.-(6u ((o=k) : E( W B, BJd di
X
1=1
(sc. - I(y=k)
multi For any PX1Y (mly)= = _ (2x)
Pmmltti let's first divide the mean vector (dk, 1/) into E XTY
width grids with a small constant 17 > 0 (the value of 7 will be specified later): If (ók); [iv, (i + +))) for some j e Z. let (Φx); in and σx = ((ēx),,-... (ox)), Similarly, if (tb); [in, (j + 1)n) for some j Z, let (15); = jn and J == (()))...,(()).... (V)a-d.). In this case, we have [[( v) (Øk: 15 < dn².
Let
1| (0m00)13+ I(y=k)
(0 II (2T) 10m)-2e- 2e
According to the definition of the bracketing, we want prove that p' pxx v) 2 pxlx (xll))) By completing the square w.r.t x, we have
Ci
o (Ōx, 51 + c2 - (-b11c - (dx, 3111
2
ci (0h, 15 (cxe, v)
(1 a))
x
C1
[1 13) - (0x, 10112++ 202
2
- c2
c1
2
17

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Further taking C
?/ and cg d(1 1))n/2, have
(1 -1)
ci ci(kk: xb) (Øk, 2b)
C1
||(bk, 15) (Øk, + +))l2+222 20²
1-c1 C1
1-c c1
2
1
2 +
(ok,, (5) - -(ox, 35)
1
1-
| 25) - (0x. +b) 2 +22
202
(c1 = 1 1)))
1-
12
1 !/ |1 x)) -OoR; +)]l++ 202
(7
"
2 1 1-11 drj² + -202
(1( 2) - -OOk, 25)1 - dn)²)
1
d(1 1)m + d(1 17)1 = O.
Therefore, it holds that for all y E y,
Va X: pl (2, v) PX (arly).
multi
(14)
Moreover, given any 0 < S 1, we take 77 ,... and thus c₁ = 1 -ंं and c2 = [1(( 1fa) f Since d N, we have 7) S 2 and 02 1.- Then, (p((:)) (-l)) (x)) can be bounded as
11 (-wiillc)))) / 1I v)
-Xxl (zlv))ldx
px(x, y)dx /
multi PX (xly)dx 1 60
(/x e 21121/ e
(2x)
x
Vai
1
(1 + 2022) 1
(6" 1 + 2x for ())
(1 + d(1 7)n)) -
(c) = 1 - 1) and c2 = d(1 -n)/2)
/1 17
<(1 + ()(1 + d(1 7)n) -1
VI= I 1 for [0. 211
-n(1 + d(1 - -2)) $ (111 + d) =
(15)
Combining Equation 14 and Equation 15
we know that for any muili (zly) E Pimuhi and 0 << there exists some
p' (xx, u) E B such that given any E y, holds that Vas E X (2, v) 2 Pxjy (æly), and (Ip((:)) Pxxx (lw0) S e, where
B = p'(x. y) = II (2m)- de - 11-- (5. I(E=))
K
(ox)), (1)) E [-B,
Recalling the definition of the upper bracketing number in Definition 3.1 we know that B is an e-upper bracket of PXY w.r.t. L (X)). Therefore,
N1 e; Pripi L. L'((3)
<|8| = ((on)KEn 5: (. (()-B n=2)
Kd, Kdg+d-dj
1B +1
(21+d))1 d)B +1)
(K-1)di+d
which completes the proof.
18

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
B.2. Proof of Theorem 4.1
Proof of Theorem 4.1
As 0%
2, vi* E V, and p the maximizer of likelihood (s(xx(r) in Puulti according
Theorem 3.2
know that
X|y (H: L (12))
TV Fmulet)
33, n
log.Vi
log
According to Theorem B.1 it holds that
(² Panpi 12() - (2 d)Bn 1) (K-1)dr-d
N
Therefore, obtain the result that
PXxx S 3 //2 ((K-
((KK-)d 1)d1 + +d) log(2(1 + d) Bn + 1) + log )
Omitting constants about n, K, d1, d, B, and the logarithm term we have Ryy (2 = ②6
B.3. Average TV error bound under single-source training
Theorem B.2 (Average TV error bound for conditional Gaussian distribution space under single-source training). Let ssingle be the likelihood maximizer defined Equation given Pxly with conditional distributions as in Equation 5). Suppose = - B)d, v [-B, B]d-ds with constant B > O, and oz 4, v0" V. Then, for any 0 < < < 1/2, holds with probability at least - 8 that
RTv priugle
-0(
=Ō
Kd n
Proof. The proof is very similar to that in the multi-source case. According to the assumptions, the conditional distribution space expressed by the parametric estimation model is
K
K II (2m) - ( ((w=k) : El B] S/ E(-B, Bjadt
Paingle X|Y
single (xll) II Pxly
(Po. vn (2:(k)
1=1
single For any PXiY (xly) = - (2n)) - f (ou.))11) k(
E paingle let's first divide the mean vector (Øk, tba) into w-idh grids with small constant > 0 (the value of will be specified later): If (Φx); E [jn. (j + 1)m) for some j E Z, let (Φx); = ju and de = ((xx) 1,, (()))))))))) Similarly, if (vxx); E [in, (j + 1)7) for some j E Z, let (vx)): jn and Jh = ((b4)1,.... (Wx))-dd)) In this case, we have (Øx,, shx) - (Øx, 2h )1 < dn
Let
K (2m) 4e (e- K(u=)) k=1
(2)
(2m)
We need (x,
(xly) by the definition of the bracketing. By completing the square w.r.t. æ, we have Px|Y
single
C1 x (0e, 3 + c2 - (-51lee- (-551le-(0ns
(dh, ww)ll
ci (o,, Fx) (che, sha)
2 (1 c1)
C1
(1xx, 17)) (0x. wxx)11 202
1- c1
19

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Further taking C1
7) and cg d(1 1))n/2, we have
2
x + ci (Øx; vhx) - (dxs v/x)
(1 0i) )
Ci
[|(0k, thx) - (shu, aba) 11 202
1 -
1
C1
2
1
11 1
cibo, vu)) - (ox, )bxo)
1-
1)
(0xk (ohk: 112 202
(a 1 -n))
1 1-11
1 1- 17
2
(|x, +x) - (0u. 22)112 + 202
(17 (1>00 0)
17
1 17 dip² + +22
(1( 1)) -60k s
1)
d(1 n)n- + d(1 771- O.
Therefore, it holds that for all y E y.
te E x:: p'x v) 2 Pxly (x g)
single
(16)
Moreover, given any 0 < e S 1, we take
= r and thus c₁ = 1 - and c2 = 2
14a) J+1
Since d E N, we
pinnle have 7) S 1 and 02 3. Then, [p'(: ( (-ww)lla>co) can be bounded
ll -PxIx (-19)llax 1 In'(z y) - -PX
pinele
pinnlee (all) dx
1/ 1l (xx, y)dar [
singlo 1 (all)ddr= = Vc1
1 6 -1
(3x e 21le1l/ = (2x)
=
x
< 1
(1 + 2002) -
(6" S 11+ 2x for x E (0)
Ve1
V1 - (1 + d(1 - n)m) -
(c) = 1 - 1) and c2 = d(1 - 1)1122
<(1 + 7))(1 + d(1 - n)n)
VI- S 1 + for x [0, ;
(n(1 + d(1 -n²) S 11(1 + d) =
(17)
Combining Equation 16 and Equation 17 we know that for any single (xly) PXx and < S there exists some
p' (ax, y) E B such that given any E y, it holds that Vx E X p((x, v) 2 PxlY (ælly,, and (p'(;;) ())) px|x (-1x))laaco)) S e, where
K II (2T)- ((2) te -110 (0l0 ((==k)
B (x(,))
(ox) (Wh)s E [- B, B nZ
Recalling the definition of the upper bracketing number in Definition 3.1 we know that is an cuupper bracket of paingle w.r.t. L (2)). Therefore,
x|y
No c: x|y
Psingle
-L))
< = (lāe)i 1
Kdi+K(d-a))
2B
+1
11
2(1 d)B Kd
+1
6
20

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Besides, according to Theorem
we know that
RT (Pxlx
5
No
IA(3)) + log 1 5) n X|Y
Pxlx
Peincle
$ 8 /
Kd + d)Bn + 1) + log 1 8)
Omitting constants about n, K, di,dd,,, B, and the logarithm term we have RT (Txx(X)) =Ö(V Kd
C. Proofs for Section 4.2
C.1. Preliminaries for evaluating the bracketing number of neural networks
Our results build on the intrinsic connection between the bracketing number of conditional distribution spaces and the covering number of neural network models. There is rich body of work on the complexity of ReLU fully connected neural networks, also known as multilayer perceptrons (MLPs) defined in Definition
4.2 from various perspectives, including Rademacher complexity (Bartlett et al. 2017), VC-dimension (Bartlett et al. 2019), and covering numbers (Suzuki 2019] Shen 2024 Ou & Bölcskei 2024). Specifically, prior results indicate that the logarithm of the covering number of an MLP scales as O LS), where L is the depth and S is the sparsity constraint. Furthermore, Ou & Bölcskei (2024) establish a lower bound, showing that for 2 1 and W, L 2 60, the covering number scales as (LLS). Their proofs share a common idea. To enhance clarity, we include a detailed derivation below following these prior works.
Definition C.1 (e-covering number). Let be real number that > 0 and p.q be integers that S p.q S co. An e-cover of a function space F with respect to ||-Il is a finite function set C C JF such that for any f E F, there exists some f' f' C such that [fx f'(x))l (P(x) S e. In particular, when
oo, it requires ||llf(x) - f( (L((2) =
the cardinality of the smallest c-cover with supmexllf(x>) - ( (=)]l>> S c. The c-covering number / (c, 111--la,,,xc))) F, respect to 1|=)1
Lemma C.2 (Lipschitz property of ReLU and sigmoid, Lemma A.1 in (Bartlett et al., 2017)). Element-wise ReLU
ReLU(-): Rd + Rd and sigmoid (zx Tкe- are 1-Lipschitz according to 1- for any p 2
Lemma C.3 (Covering number of a composite function class). Suppose we have two classes of functions, 9 &onssstinng of functions mapping from X, to X2 and F consisting of functions mapping from X2 to Xz. We denote by F o G all possible composition of functions F and G that F G ofog g: E F.g E 9). Assume that any I E F is -ip-iihchitz w.r.l. I1 i.e., for all x2, x'y2 E X2, (1)()))) f(x2))lo= S KF 222 2'2loo. Then, given constants EF, Eg we have
(EF [EFF+KFGG KFEG; F o 9, (x)))))) s N ( |1 (x)))) ) Eg:
and Co be an co-cover of G w.r.t (x)))) such that |Cgl = N NN cg: 11-1111111111 (111),,, For any f o ggg E Fo G. there exists f CF and
Proof. Let CF be an cF-cover of F w.r.t. 11-1l,o, such that |= |=1l= N F,
g' E Cg such that
Vaz X2, (1f()))) f'222)loo sen, and Va, E [1, (x1)))) S cg.
Then, for any 31 E X1, we have
o g(()) 8 o o' S o ff g( III og()) S (x
< kxllg(z)))))) g (x
Therefore, we have Cr Cg is an 6f cover of F o G, and thus
(( W(FF+Kr(s: Fog, 11 s [r ccl $ Cr||Col = -N
21

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Lemma C.4 (Covering number of MLP class). Given any constant 8 O, the covering number with respect to (0))) with X C [0, 1 of an MLP class F(I, W, B) defined in Definition 4.2 can be bounded by
can
(W + +1)
2B 8
8
$
+
Proof. Fix any a E [0, 1, Given any network f E W, expressed as
f(x) = RcL)) b(L) o (A + b(
let fi(x)= = (A ReLU() + b()) o...o (A + b( for | = , and (xx) = A + +00)
Sup-norm of the output at each layer. We first prove the statement that [[f(()))loo S (BV1) +1 for all E [4] by induction. When 1,
[[ 1|A x+ 11 S max/| (t.:1lll1 + 1|60
SWB WVB+1 BB
(Wo < W. |4 S B, 11x00 S c
B(W + 1) < (BVI)((W-1)
which implies the statement is true for I 1. Assume that for some = i 2 1, [ff(())) (BV1) 1) then have
ff+1(())llo = | ReLU((((x))
b 66++1) 1l00 S max/ max|| ( ReLU (fi()))ll + [| x
( + B
(W, S W. (++1)) || S B, bb + 11 S B) (ReLU(-) is 1-Lipschitz continuous for Lemma C.2 and ReLU(() 0)
WB)lff(z)ll +B
< WB(BVI)((M 1)'+
(WB++((+11)) +Bi+I(W +1), B 1, < W(W +1)+ (W W1),
B <<1
= (WVI)*I(W+I)) + +1)+1,
which implies the statement is true for / = + 1, completing the induction steps. Therefore, holds that for all / E [4].
[f s (BV 1) +1).
(18)
For any two different neural networks f. E F(L, W,S, B) expressed by Parameter-Lippchhtzzees at each layer.
f(x) (A(4)) b((4))
(A + b())), f((x) (A(DRRLLU() b(l)) (A + b(1))),
with parameter distance that max/
| A ()) A |~x Vl - B(0' 11:0 S 8, we prove the statement that [[((()) f( 1ox S
(B v 1) + +1)88 for all / [L] by induction. When / = =
[ff f((=)lo (1)
(1) x + 6(1) 10) x - (11) 11o0
S (l(A() (1)
)al|o [ b/2) 11
W8 +8
(Wo S W, (A(1)) A 1bo (,,1(((0) - -b00 5 ())
(W + +188
(BV (W + +1)',
22

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
which implies the statement is true for I Assume that for some = 1, f(())l s SiBBI))- 11)8. then we have
(x)) - -1+1
+ b(i+))
A ReLU B(et1) 112
< [1(+t)) ( [e + ||A (ReLU(f((z))
ReLU (RAUU))) - lb
+1[b(t1) - -b(it3) 1|00
< [f]]) + WB (c (
ReRU (f(L()(()))]
+
(W; W.,11 -
A 11+1)
1o0
26 11 1loo S B, - bli+1) 1| 5))
< + (z) -
f((=)]
(ReI.U(.) 1-I. continuous for Lemma C.2 and ReI.U(0) = 0)
< WW( = +
(Equation (88
W +)
$ (VVOVVI((=++1---W/VVVV)v))) +1 +1)+++ +1
(1+))
B) (W B21, + 6, B<1,
= (i+ ((+
which implies the statement is true for / = i + 1. completing the induction steps. Therefore, it holds that for all / E [4],
- - f/ - 1(Bv1))- + 1)'5.
(19)
Discretization of entry space. Let Sentry (((A (()) (Seutry (A Sentry (bO)) with Sentry (A) de- notes the value space of all entries in A and Sentry (b) denotes the value space of all entries in b. Now we discretize the value spaces of F(L, W, S, B) into 8-width grids and get a finite class of neural network as Faz(L, W,S. B) == {f e F(L, W, S, B): : Sentry (1 (1), (( [-B, BJnBBBnaa i 5Z]. where oZ = [ E 2} Then, for any I E F W. B) expressed as fx) = (A. ReLU(-)
==
(z) (e)U(-) + 6(4) 600))..... o...o (A x + b()), there exists f E Fsz(L, W, B) expressed as f'(x) (ALLR)) + b(2)') o...o o (A x + b01' such that max Al - A |1x V ( - b(0)' 1bx S 8. Ac-
cording to Equation 199 we have for any æ E [0. 1, [[)())) fa < L(B V 1)a (W + 1)²8. Therefore,
Foz(L, W,S, S, B) is an L(B v 1) ( + é-cover of F(L, W. S, B) with respect to |I- (x))) and thus we have
N(LBVI) (W + 1) B)S || 1.))
S|Fae(L, W,S, B) [((A9) 6 Hz Sentry(AA)) -Seutry (b:)) = --BB
2B
S
<
which completes the proof.
Here, we further establish the Lipschitz property of MLPs, which is useful in the following proofs for deriving the covering number for MIPs with an embedding layer applied to input data.
Lemma C.5 (Lipschitz property of MLPs about the input). For any f E F(L, W, S, B) defined in Definition 4.2 f is Biischip continuous w.r.f. olo about on X, i.e., for any x. E C Ro, it holds that
[[((x) f(x))l < B'W | xl|mx
Proof. Fix any x, x' e X. Given any f E F(L, W, B) expressed as f(x) = (A ((A let fx( = RelU)) + b()) .... (1)))) + b(1) for 2,..., I. and fi() = A0)) + b(1). We prove the
23

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
statement that [|]((z)) lo s B'W x 2l for all / E [Z] by induction. When =
[1f((z)) - f1(2))l>>> = || || x (
A x' 1l = |40 - (a==
S [W] a x 1l
(Wo - W. || S B)
= B' | 2'1000
which implies the statement is true for / 1. Assume that for some / 1= 1= then we have
2 f((x')l B' 2210
[1)))()) f1+1(x0)l
|A (+11) (+LRRe + 6(+1) A ReLU 1(1t1)1
| ((()) (eILU()((z)) (eI(((((z'))) 1loc
W B()R)E((((((()) - (e
(W, S W. ( S B)
( - f1
(ReLU(-) is 1-Lipschitz continuous for Lemma C.2
S WBB'W' | 2o100
B'+'w"' x x' 2/11
which implies the statement is true for / = + 1, completing the induction steps. Therefore, it holds that for all I E [7]]
[f) - f((x')ll> & B'W 2'1100
Thus, when L, we have ff(x)) - f(x'))()l 20o -If((()) f= S B' x x loo.
C.2. Covering number of the logit space of ARMs
We first characterize the output function space of the neural network without softmax operation, i.c., the unnormalized distribution parameter space, commonly referred to as logits in ARMS. This result serves as the foundation for deriving the bracketing number of the conditional probability mass function for each dimension. The derivation carefully analyzes the covering number of outputs for the entire network, which consists of the embedding layer, encoding layer, and an MLP. This analysis makes use of the previously established Lemma CemmmaC.44am and Lemma C.5
Lemma C.6 (Covering number of the unnormalized distribution parameter vectors). Let Ho v) MxD with ha v) = fw 10 (Ev,,
[hop1 (x. v) ho y)]
X
.vx (x, ( X + RM
as defined in Section 4.2 Let Ho (Ho(æ, v) w E W(L, W. B) Au [-B, Dxd,, by E [- B) Vx
[o, Mxx, Vy E (0, 1 R xd) with constants L, W, W,S.B. B Then, given any O, we have
(( + 1)4 S+D+(D+M4K)
N Ho, II (ll. S
Proof. For any [D], ha can be written fa
Evy. Vx Let the embedding space
Ga = (n)
v 10
Ev.. vx (x, v) ,..,, v 10,
Ev vx (x, :a = (Aa, bo. Vx. Vr),
bo
An (lBB, B Dxd, bo I B| Vx E [0, Mxd. Vr 10, 1xx.).
where
(AooL, Vy Vl
+bol1] +T
= (diag <ddiig(AeEEv AA.Ev..ss + E 10
Voo.b. o UV,.vxxx,1)
a (A. bbo
Given any 8 > 0, we first evaluate the 5-covering number of (2r, y) w.r.t. |1
24

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Covering number of the embedding layer. Let Sentry (A) denote the union of value spaces of all entries in A and Sentry (a) denote the union of value spaces of all entries in a. We first discretize the value spaces of Ga into 8-width grids to get a finite embedding function class:
g0,57 = {Go E Ga : Sentry (An) = Sentry (bo) = [-B, B] n oz, (ey) = SSnntry (Vx) =
Denote by ||e - a'lloc = sup (1 AA - -Ablloo bbo, 1000-b111,, VVV - V llo || Vx - [o For any G. E 9a with a = {Ao, bo. Vx, Vy there exists Go E 90 with a' = (A byy Vx, Vy- such that ||a - a' a'looo S Then we have for any 2, E X x y.
(a( v) Go (o.
1 '00
Ev,, vx (0,,1) 11 b
10D
Evy. vs (2, ".; v Vooib Vx v 100
Evy, (2. v)
(cr
VAo.bo Evr, vx (x. 3) b. Evy, (x. (x.y) y)
(diass(Ann AD (x, ()) + bu) - ddag Ab (V). - 56)
< diag AsEvr, + bn diag (v - 66
(o) is 1-Lipschitz continuuus for Lemma C.2
Ao dd, E vx (x. v] Abld, E... vx (x. vd. | + |1bo bbll.
sup (D(
(Ao[d, Abid, Evy, vx (x. wldd = + Agid, : :| - Evg. (2, + 8
sup dD(
< sup do dE(n)
EVy, vx (x. dd, :11 + Ev. Evg, (2c w)
+
(Ao E [-BB B) Dxd. .IAu Abllo 56)
d.6 Vy Vyly, :|
+ Vy [v. +
d 1= 1,
suP ded
Vx Vx de Vx vala--iil + d = 2, 2..... D,
S d.5 + d.B5 8 (1 de de B)ô.
(| S 1, Vx S 1, V Vf S 8, Vx
Vkllo 56)
Therefore, G0,82 is an (1 + de + de B)S-cover of Ga w.r.t. |1-1),,, and thus we have
N W((1dd + dd.B); (aaaa.1Hllas (ol,,, s G0,52l
=|sa- (Ap. bp, Vr, Vx): Sentry (Ao) = Sentry ( B] 82, Sentry (Vv) Sontry(Vv) [0.1] 6z)
2B
Dd.+D +D
1
Md. + Kd.
5 +1
(20)
Composition of an MLP. Let beaney
= Scch:aaanss-i000----AAyyooooooo-
w.t.t Il-l ((01)))))))))))))))))))))))))( such and Cg be an cg = (1+ + B. of Sa w.n.. II-1lcc,
that |Cxl -
such that |Col = N (E0:: Ga. ll (=x))), For any Ha = [fw 1] ... fw Gal; DI E Ho. there exists f' E CF and G², E Cg such that
Ve [0, aep...1 (1) (v)llo SEF, and Vz,y E X x y, [G. v) G2 S Eg.
25

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Let H3 [ G.l: 1]
f Cal DI] We have for all n. E Y.
|| G 1] - ... fw Gali Sw Gali DIl
Hg
H 2 x
sup Sw Golid] Clddll
sup(1
- - Gol: + [ Golid]- S' G.lddla
sup eF + B
(fw is B - continuous as in Lemma C.5
= + B".
Therefore, CH (H ff Gali,]] f/ G. S' E CF, Ga Cg} an EF BL Eg cover of Ha w.r.t. I1 and thus
x((1 (((L(BV))--QO + +1) + BL de + deB) 8: Ha. 11 (lx))))))
s|Cu| s |Cr||c||
(LL(BBI) (W) B, 11 (.). de det
2B +1
$
2B +1
+d
( 8+1)
Kdd
(Lemma C.4 and Equation (20
(2B 1))
S+Dd+DD+Md
3(BV
S+D+(D+M+K)d
(BV1) 1)
Taking e (LBV IIB 1)4 + BL
+ +1 d. we have
3(BV (LBVVI)((BVI)) + 1)th d1 S+D+(D+M+K)d,
(WAp:///wWWWWAYYM//A/V//////A/AD S+D+(D4M-K44
(de, (BV1)2) 1) (DD+(D++*K)d
3/BV1)([BVI)++ + 11 + + ( (1)dd.)))
3BB++))(BV))+22)) 11)4 S+D+(D+MKK4
C.3. Bracketing number of conditional probability space on each dimension
Lemma C.7 (Bracketing number of conditional probability space on each dimension). Let Po(x,y) = (po(y) ... po(==<D v)] = softmax/ Ho (x, X x + [0, M where softmax(H) denotes element-wise soft- max operation that (iltmax(H)imme d] = EN m.a... Given a class of autoregressive conditional distributions that
H[m..a]
Ho E Ho with po (aly) as defined in Section 4.2 then for PxxY = pp (xll)) = p po(y)) ... po any holds that
No D. s SN( E
Sed Ho. Holllllo,,,,
26

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Proof. Let CH be an ex-cover of Hy that |Cx| = (E(xs Ho. Ix According to the definition in Section 4.2
po(x) = p(21: po(y)) P(ED:Po(Z<D..))
Pol:.)) ... p(D:: (o. = III
plau; Po (ll, = II d=1
Pa u2 d]
d=1
D M = II II
M
I(s==m)
(softmax(H))) ()) d))
(as=m)_
Ho (c..ollid)
d=1 m==1
d=1 m=
==
Then, for any po E Pxiy, there exists Hy Ch such that for all a, x y, Ho (r. y) (z y) S en which equals to Vm E [M]. E [D), Hz (2 d] en S Ho(æ, v)mm d S Hg y) (,, dl + cM- Let p/o (2(1)) = and denote (a u)( d = II
I(xa=m)
[(za=m)
ITM
m=1 EE E MOmDdd--
Mr -
Hole,s)mm Iou
We Ho(e,)))mm
immediately have: for all 0, EX x y, (x 2 (v)l)) since Vm [M]. d E [D], el
and (ww))itdd) S Ho(m.))idd Moreover, we have for all x, y E X x y.
Po(a, V) dI- Po(x, u)(aa, df
M
[(x==))
M
e (o
((x=m)
II Hilwwwiddddcc
II
i= (a
muml
m=
M
Hig (a(
c Ho (m))
((x==))
II
Σ i= e (onwilidd
m=1
1(aamm)
M
(f(aa.))mmmd Ho(a.)))mmd
D M Ho(a.))idd + Holaa M EM - -e Ho(.s))
E ((x))l) A Holevv)l
m=1
M II
Holeww/mm. स
Σ M Ho(a,)) Ho Hole.wllomdd
Σو M
Hblawwhlddtenee 1(x=m)
Di=l
Σi= (.)()))) i.d
M
Ee= (oi
m=1
(1ea cbl S cavbla a- - bl and Ho( v) -H v\ll s en)
M
He 3CH Σ M (m))))))
Ho(m:)) M (l T(zumm)
==
M (i EM Ny (m.s)(iid)
m=1
Ho)(( Hx S)ll S (x)
M Hel 3 N Helewwlmdddoee K(a=m)
())((z))) y) H3 v)ll s s()
ΣM Ho(
m=1
M =(3ene + cHEN
mo
[(a=m)
1 Ha (o.w)))i
BeN sen + EEMEEH)
Pa w) [z df $ 4e den Po(x,y))au d].
(21)
Given any y E y, denoting a.* : Emgne(M] (2 - po( for d 1, 1,.... ... D. we have the following recursive
27

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
formula for {ad}de(p)
d II
ad Σ
(
po(
Σ
Pa [
II Po (z, 12
ssuE|M j=1
d-1
d-1
E Mc
II j=1
II
Po (a و
d-1
Σ
II
Po (z,
df Po(x,))(za
M j=1
d
Σ
pa( - pro(=<a-11))
Σ [dE[Na]
Pa(e,, y) (ad, + dene
Σ Po y)
#gd Pesd-E(MM) 1E(M]-1
(Equation 21
add-1 Σ
Po(a,y)(aa eeN
Σ po(23d--19)
[ME[M]]
=6 -d-- + 4e ten
0
(caE(M] vala,, d] 2 and Eagee Po (o( 1)
According to this recursive relation, and
a1 Σ
16l (u) (o(llv))
Σ P)l(x,))(a11 Po ()(1, E[M]
Z|E|M]
Σ
4e den Po v) 4eH
(Fquation (1
[]E[M]
have aa S 4eye Aew 2den
Therefore,
-
Σ #E MM
(pb(=ly) - (o(xly))p((<zl))
Σ
(p<(=<ply) po (a(xx
BSDE[MP
= S 4execen 4eh 2den -1 2CN -1 I 4exeeee
2den - 2e (22den - 1).
2EH
Suppose that 67 E (0, ta). we have e & ed S e and 2den 1 S 4dex as ex S + 2x for x E [0, 1]. and thus deN (c2dex - 1) s Sede. Therefore, given any y E y, the (X) distance between pb((ly) and po(-ly) can be bounded as
(1pb(-1e) - po(-10)ll
Σ
(x(l)))) - (p(xx w) s tex ( 2deR
11 -1) Sedex.
[(xx=m)
Therefore, Bp = pb (2(1) = - IIM
Hgte.
Hig E Cu is an Seden-upper bracket w.r.t.
EM EN 1ooowee
L' (X) of Pxpr, and we have
Ny (8edexx; (2)) < |Bpl < [CH1 = N( Ha, Ho..1
Letting Sedcy = (0, 1], we have CH I Bed < 1
4n0 and thus
(sPxxyy, SN sedi Ho. 11 (1.b,,,
Ho.
28

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
C.4. Proof of Theorem 4.3
Based on the relation between the bracketing number of conditional distribution space P and the covering number of output logit space of the neural network Hy derived in previous lemmas, obtain the final result.
Proof of Theorem 4.3
With conditional distributions as defined in Equation (6) we have
PXIY"
ppo (ally) = p( po(y)) ... papp (LLD:Po(<<DD) po( Hg E Ho)
where Hg Ha( v) X + RM as defined in Section 4.2
(ar, v)
ha. v)] x RMXD with he ( So (BV.
According to Lemma
C.7 and Lemma C.6
Ni Pxrx SN(
Sedn' Ho, Ho,,ll-llaaa (mx)))))))) s (24edn(T.+ + 1 1)44
S+D+D+++*K)
According to Theorem 3.2 we arrive at the conclusion that
Nog (n: XIV (2() log 8)
RTV(PXI
smulth
S
5 3 (S+ (S+ D+ (D+ (D+ M
K)d,) log (2g(24en 3) + 1)7)) log 3
Omitting
constants
about
n, K, doy L, W,S,B, B, and the logarithm term
we
have
L (D+
C.5. Average TV error bound under single-source training
Theorem C.8 (Average TV error bound for ARMs under single-source training). Let ssingle be the likelihood maximizer defined in Equation (3) given Paingle with conditional distributions as in Equation 6 Suppose that & [0, 1]d and W(L, W,S, B) and assume 0% E $, up" E W. Then, for any 1/2, it holds with probability at least 5 that
RTv(e = δ KL D 1)de)
Proof. As formulated Section 2 and with conditional distributions as Equation (6) we have
K
psingle X|Y
II
(po, (zlly) 1(y=R) (x) = p(11: Po. (v)) ppxp: po, Po, (E<D,V)) Ho, E cHo H
where
Hop (x,
(x, her D (x.v) v)]
X x
RMxD with hor v)
\Op fwr v
U dood (Vv. (2x,)) : x y M defined in Section 4.2
where
fwr oevie. wa W(L, W,S, V[k. 10.1) Uo { {u (2ll))
29

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
For all & E [K], let Bp, be an - upper bracket of PxIr = Pee (ally) = p(r1: Pex (v)) p((DD: pe (Z<D, v)) (.b.))))) He E Hou
w.r.t. L' ('C) such that [Bp=1 = ~N ( L' (2)). According to Lemma C.6 Lemma C.7
we know that
S+D+(D+++1Jd
Bp,, SN
- (24edn(I + () 8edn Ho 11 10
1) 4+2 (W (W + +1)4
For any p(æly) 1)) = II (po, (ally) I(w=k) E PXir there exists ph, E BBp....., .o. E Bpx such that for all & E [K], we have: Given any y E y, holds that Vo P', (xly) 2 pa. (ally) and llpe, (+1) po. (ll1)llaa $ n
Let (a(lv) = II (v6. (zllv) ((=
then we have that given any y y.
vo E x, ('cl)) = II Po. (als))
K
1(x=k)
2
II
k
(po, (xl "(=k)) =p(xly),
k=1
and
1
11p(--1)) pl-l-lllll s sup l1pbr (-lv) Pea (100)l - kE[K]
ple E BPx) } is an #/ -upper bracket Psingle w.r.t. L (X). Thus 1(v-
Therefore, Bp
(((x))) = -II (pe, (zlwy)
have
Ni : x1x L' (23) < |Bpl
U NE[R]
Bp.
II BPal
RE(K]
24edn(L (3) + 11)4) S+D+(D+M+1)4
II [K[
(B +1) K(S+D+(D+M-Ij)))
According to Theorem
3.2
we arrive at the conclusion that
log.V1 [2: Paiugle [2(3)) loog 1
R P 5
s 3 /
(
K(S [K(S+D D (D M 1)d.) log (2g(2Adn(L++3)) ()BV))))))))))))))) + 1) + log 1
Omitting
constants about
n, K, do, L, W, S, B, and the
logarithm term
have
RTV(Dipp
δ (L(S+D+(D+M--14)))
D. Proofs for Section 4.3
D.1. Covering number of the energy function class
Lemma D.1 (Covering number of the energy function class), Given Ug = {uo(all)) = fu ey (2, (2,v), y)} with un (a(=ly)
X x y + R as defined in Section 4.3 Suppose [ik. E [0, 1]de, w E W(L, W,S, B) with constants L, W.S, S, B > 0, and
= 0, 11,, = [K]. Then, given any e >> O, we have
StKd.
3(L+ 1) 1 +1)²
<
30

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Proof. As defined, Ug can be written as
Up = = { v) = for oev((. : F,,,,, E&V
Ev,
where εv = {ev (z. y) = Vly,
Ve E [0.1]xxd). Denote by X1 = X x y = [0, dl x [K] and X2 = [0, 1jd+tD.
Given any 6 > 0, we first evaluate the 6-covering number of &y w.r.t. I1
Covering number of the embedding layer. Let Sentry (V) denote the value space of all entries in V. We first discretize the value spaces V into S-width grids to get a finite embedding function class:
V0, fev Ev Sentry (V)) [0, n6z].
For any ev E Ev, there exists ev E Ev,a2 such that I V'I S 5. Then we have for any æ, y E X1,
|| V
-
[w v., -
[|ev(z.)) 9) ev (x, y)
[Br :1]
Vl, oc
$ V V'll S..
(22)
Therefore, sv 5z is an 8-cover of &v w.r.t (o0 and thus we have
1 kd. V Sentry (V)) [0.1] <
N (1 s &1v =
( +1 (1+1)
Composite energy function.
According to Lemma
C.3 given any EF, > the covering number of Uo is bounded by
M((= + RKFE; 11 (1-,)),. s
/ <c: &v, I1. (1011)),,
According to Lemma
C.5 Ky = B-W4, Further taking ey = L(B v 114-1))--1111 + 1) and Eg = 6, we have
cy KyCe L(B v 1W + 1 B" (I(BVI) (W 1)t + BLW 8. According to Lemma
C.4 and Equation
we have
8 (30 +1)=
11 kd.
and V ( (lee, - 8 11+1
Therefore,
N( (((IBVV1)=- + +1 + 3,w4));u6,1l1l (111,,,00)
2B
Kd.
3
+1
+
8
8
(2B ABVII 1)
StKd.
8
2
S+Kd,
8
3(B V 1)
S+Kd.
(Bv1) 1)
8
Taking c (L(B v 1)~-1((W 1)th B'WL)), we have
S+Kd.
((BV))(((BVI)- (W + 1)" + B'W")
N(c::60:1-lboo..ccc
StKd
3(BVI)(IBBII)((W + +1)4 (BVI)+(W + +1)4)
StKd.
3(I 111+1)(BVI)++ + +124
which completes the proof.
31

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
D.2. Bracketing number of the conditional distribution via the energy function
Lemma D.2 (Bracketing number of the conditional distribution via the energy function). Given class of energy-based
conditional distributions that Pxly po(ally) Ix Male dl uo E Un .frr any 0 < S 1, it holds that ano
e :oo, Mo, 1440011-loo... (1x),,,))) No Pxxxx,, [11)) s SN(
4e
Proof. Let Cie be an ey-cover of Uo w.r.t. I such that [Cul N (uulloo Il (ob, (xx3)) For any po(lyy)
sp(alo? dol E PxIy, there exists u/g E Cu such that for all æ E X and E y, [[ (2012//)) (2o())))))
u S eu, which equals (all)) eu I ( S u y) + ex
Let 1/ e -2
Jx dn Then we immediately obtain that: given any E y,
ue (mla))
Vo E (b Jx uglolul ou doc Sxx (olld) = po (x)
(23)
since for all X, y E y. tm 2 e (m(lu))) and fx ub(alv) au der S fxx (mll) da. Moreover, bound the I (3) distance between (-1)) and po(:ly)
(1b6( po po(-b))lzaax 10' (ally) po (dx
/
ab(arlu)2eu
(arl)))
dx Jx e e-uo(alu)
Ix e e- ds
e ((((oll))++ 20u
Sx e uul ds
-- (o(e Jx (ala))
dx
Ix us e- (o ds
(((()/))+ - e ())
Ix -- (a(()) ds
+ t e uol (Jz mo (s e-
dx
xx
fx e u. e co
(2/22/1) (2/) (xll))
Ix us (elv) ds + t uo( Sx Sg (ololl)))) -ubblelw) ds
dx
fx (ollo do 1x euooeluid
x
c- u/y (zc) 2 (x)l)) (mll)) + 202r fx e- -olella) + 6 uollv)) fx f ((g00/ (uos)))) - (16()/1)) ds
dx
fx e
e (alu)) do I e uollli
(/Ea -ell 1l0 e I 1l60 eavb caabla (00- bl)
c (c(((r))))+ +((( BEu fx e ua(slu)ds + mo(alv) fxe e- ubb((l))) tou cuds dx
fx e (elv) fx (s(y)))
x
(Vx E X. E y, (uo()ly)) 2<(()/)) S au)
{
3eye Beye=mo(olo)+aaz
u (6(l)) ds que (o((lu) fx e 16(()))) fx Ix (g( ds J uo(llld
ds da
(Vr E K. y, ( ub((=/y)) S se))
Sg - ub(olu) ds + e Sx e- u' dx 1x e ds ds Ix e uo(alu)ds
1x
(3eue + e dx 2 + ayeau < (eeye
(24)
fx uo
32

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Therefore, Bp = po((olu)
(gg(m)))) uug (2) E Cu is an 4ezje dupper
-upper bracket of Pxir w.r.t L (2)).
Ix (y(wla)
do
Thus have
Ni Ni (e Pxxy, (20) S S1ppl
Cul N
M( eux (1lo,,,,,
Let (eyee(ee)vv we have 4eu uuiiii S 1 and thus 4eyee4au)vv 4eEu, that get Eu de Therefore, have for any 0 < < 1,
Po( 1212)) s SN( :o 11 :
D.3. Proof of Theorem 4.4
Based on the relation between the bracketing number of conditional distribution space Pxiy and the covering number of energy function space Us derived in previous lemmas, we obtain the final result.
Proof of Theorem
4.4
With conditional distributions as defined Equation we have
(=
Pmmutt X|Y
(Po(xly)= = Jx e- uolalvidd ug E Uo y.
where
Up = { uo(x)ly) WLL
[0. 1
0, according to Lemma D.I we have N
Let be constant that eu (1 StKd.
s
Then with Lemma D.2, we further obtain that
Ni Punuti XY (2)) <N(
1 ( Uoo ll (N(1))) den
bold:::lllloo.eeoo (1 s (12(LL+1(BBV)+++ +1)+ StKd
According to Theorem
3.2 we arrive at the conclusion that
1 8) log.Ni PxIpt L'(2)) + log
(::
(pmulti) R RTV(PXIV
S 3/
n
1
(S log()nn 1) 1) +))) log )
$ 3
n
L(S + (do) log (1) (W + +1)) + log 2)
= 3/ n
Omitting constants about n, K, de, I., W.S, B, and the logarithm term we have RTV ( = 001 LIS+
(dd) Kd.)
D.4. Average TV error bound under single-source training
Theorem D.3 (average TV error bound for EBMs under dingle-souree training). Let ssingle be the likelihood maximizer defined in Equation given Pringle with conditional distributions as in Equation 77. suppose that & = [0, 1] and I = W(L, W, S, B) and assume of E 4, is E V. Then, for any 0 < < 1/2, it holds with probability at least - 8 that
RTV (exigg))) =
LK(S + de)
33

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Proof. As formulated in Section
2 and with conditional distributions as in Equation 7 have
K
(mly)
I (pe (2) (z>ly) [(v=k) -po, (xl1) = Jx te (alu'
x|Y
uo E Uox sr
where
{ (uor (2)1)) fux o (evia,) we E WLL, W,S, Vik, E 10, 11,
Ue
For all & E [K], let Bp, be an + -upper bracket of PxIr,
pea (xly)
(mlu)
us E Uox j w.r.t. L (X) such
Ix oo do
that [Bpl Ni h: PMIM L' (2)). According to Lemma
D.1 and Lemma
D.2 know that
1 ( :em ll. (x (12em(L+ + 1) (VV (W+))²) std.
|Bp S N
For any p(zly) w) = II (pa, (all)) k(w=)) Exinpe there exists vo, E BBI....., E Byx such that for all & E [K], we have: Given any y E y, holds that Vo E X, Po, (xly) 2 pa, (æly), and ll (-1y) po. (-11))laaa $ n²
Let p'<ll)) II (p6. (<c1v) ((==))
then we have that given any E y
Vo E p/(lv) = II Pe. (mllv) k(v=)) 1=1
2 II Pew (ar (arl))) (( =p(all),
and
1
(1'(-l9) ((((l))-((lb)llecss s sup I1 (-1y) po. po (190)l1200) kE[A]
n
Therefore, Bp = pf((x/1) = -I1 (po. (zlw)) 1(v=k)
: Pbo E &Brg } is an 1 -upper bracket of X w.r.t. L (X)) Thus
we have
Ng ( pindle 11(3))
|Bpl
U Bp. S II BPy kE(K]
KE[K]
Std.
II kE[K]
12em (orn([+1)BBB))))) (W +1)
(12en(L + 1)(BvI) +ww+ll 11)4) k(S+d.)
According to Theorem
3.2 we arrive at the conclusion that
log.Vi
( psingle 11(23) + log 1
R RT (P
s 3,
m
71
1 3
log log (12enL I) (e++:(W++))) +1)) + loy 1
$
LK(S de)
(W
Omitting constants about n, K, de L, W. S, B, and the logarithm term we have RTv(P (Prinle)) = =0(1 (LK(Std.)
34

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Table 2. Hyparameters of our experiments. 'Ic' denotes training from single-source, and others denote training from multi-source which contains 3,5, and classes.
Setup
Iterations (kimg)
Learning rate
Decay (kimg)
lc
184549
0.005
2500)
3c
268435
0.006
4000
10c
1610612
0.012
6000
Table 3. Standard deviations of FID five times of sampling.
N
Sim
K
Std Dev (Single)
Std Dev (Multi)
3
0.0086
0.0057
10
0.0018
0.0336
500
3
0.0160
0.0158
0.0056 10
0.0035
3
0.0034
0.0064
10
0.0028
0.0250
1000
0.0047
0.0051
10
0.0013
0.0084
E. Supplementary for experiments
E.1. Additional detail of real-world experiments
Implementation. Following EDM2, we use the Latent Diffusion Model (LDM) (Rombach al. 2022 to down-sample each image 3 256 to corresponding latent R for training diffusion models.
All experiments are trained and sampled on 8 x NVIDIA A800 80GB, 8 x NVIDIA GeForce RTX 4090, and 8 x NVIDIA GeForce RTX 3090 on the Linux Ubuntu-22.00 platform.
For a fair comparison, we set different hyperparameters for experiments with different numbers of sources as shown in Table but these parameters are the same within each similarity.
Based these trained models, we perform multiple samplings using five different random seeds estimate the randomness in calculating the FID scores. The standard deviations of FID scores over multiple samplings reported in Table 3
corresponding to Table in the main paper.
The selection of sample sizes and the number of classes. In the real-world experiments, we set the number of classes K in and 10, and the sample size per class N in 500 and 1000. We would like to clarify that this selection is influenced by several inherent characteristies of the ILSVRC2012 dataset: (1) Sample sizes: The maximum number of images per class in ILSVRC2012 is 1300, so we selected sample sizes of 1000 and 500 images per class, which are common choices. (2) Number of sources: Given that distribution similarity levels were manually defined, it was difficult establish large number of structured subdivisions. To be specific, to ensure reasonable similarity levels for the controlled experiment, we designed a two-level tree structure for the dataset, as shown in Figure Overall, we divided the whole ILSVRC2012 into 10 high-level categories (mammal, amphibian, bird, fish, reptile, vehicle, furniture. musical instrument, geological formation, and utensil). Each category was further subdivided into 10 subsets (e.g.. for mammals, we have Italian greyhound, Border terrier, standard schnauzer, etc.). Defining such semantically meaningful and mutually exclusive divisions is not trivial. As a result, the number of classes within each similarity level in our experiments limited 10.
While our experiments are not on large-scale datasets, there existing studies that provide valuable empirical observations for large-scale multi-source training, including: cross-lingual model transfer for similar languages (Pires et al.) 2019).
pretraining with additionla high-quality images to improve overall aesthetics in image generation (Chen et al. 2024). and
knowledge augmentation on subsets of data to enhance model performance on other subsets (Allen-Zhu & Li 2024a They have offered relevant findings that inform our work.
Connection between FID and the theoretical guarantees. Our theory provides guarantees for the average TV distance
35

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
Table 4. TV with the numher of sources K simulations on ARMs.
K
3
5
Single-source
0.0763
0.1212
0.1519
0.1787
0.2127
Multi-source 0.0763
0.1145
0.1318
0.1364
0.1369
Table 5. TV errors with the sample size in simulations ARMs.
n
1000
3000
5000
10000
30000
Single-source
0.5680
0.3516
0.2882
0.2036
0.1212
Multi-source 0.5491
0.3467
0.2747
0.1922
0.1145
Table 6. TV errors with the sequence length D in simulations ARMs.
Dt
10
12
14
16
18
Single-source 0.2036
0.3785
0.5932
0.7242
0.7505
Multi-source
0.1922
0.3530
0.5068
0.5747
0.6289
(Equation which quantifies distribution estimation quality but is incomputable without access to the conditional distributions. Therefore, in real-world experiments, use FID practical alternative. FID measures the similarity between generated and real data distributions by comparing their feature representations in pretrained neural network. It is widely used to evaluate image generation quality and serves as the best available metric for our setting.
Connection with the theoretical analysis of EBMs. Additionally, we would like to discuss the connection between our real-world diffusion model experiments and the theoretical analysis of EBMs. As mentioned Section [ EBMs are a general and flexible class of generative models closely connected to diffusion models. To be specific, first, the training and sampling methods in (Song& & Ermon 2019 Song et al.) 2021) are directly inspired by EBMs. The distinction is that EBMs parameterize the energy function, while diffusion models parameterize its gradient (the score function). Second, Salimans & Ho (2021) shows that under a specific energy function formulation (Equation (5) in their paper). EBMs are equivalent to constrained diffusion models. Their experimental results (Table 1, Rows A and B) indicate that the constraint has a minor impact on generative performance. Thus, our diffusion model experiments provide insight into EBMs' behavior in real-world settings to some extent.
E.2. Supplementary Simulations on ARMs
We conduct additional simulations on autoregressive models (ARMs) to examine how empirical total variation (TV) errors align with the thcoretical predictions. The empirical results are summarized in Tables 4 and 6
Each ground truth source distribution is defined discrete categorical distribution supported on the set [M]', where M is the vocabulary size and /) is the sequence length. The variable Y is drawn uniformly from 21,2,... K). A multi-source dataset of size is sampled from the joint distribution of (X, Y)) by first drawing n samples of Y, followed by sampling X Y conditionally.
The network architecture exactly follows the setup in Section 4.2 It consists of two embedding matrices to encode Y and the first dimensions of into d, d.-dimensional vectors. These embeddings are processed by a single encoding layer, followed by a multi-layer perceptron (MLP) with width W, depth L, and a softmax output. The conditional distribution parameters are computed autoregressively using a masked input vector.
For multi-source training, single model is trained the full dataset. In contrast, single-source training involves training K separate models, each using data from its corresponding source. In all experiments, we fix M = 2 and use network configurations with de W = 64, L = 5, and batch size B = 1. We vary the number of sources K E (1,3,5, 5,7, 10), the total number of samples n E {1000, 3000, 5000, 10000, 30000). and the sequence length E {10, 12, 14, 16, 18) to assess the alignment between empirical total variation (TV) error and the theoretical bounds. For each configuration, the batch size and learning rate are selected from (100, 300, 500} and 5 10-1, 10-3), respectively, for maximum likelihood.
36

---

A Theory for Conditional Generative Modeling on Multiple Data Sources
f(x)
1
(x)
0.75
B (x((x) = ke: = 1,2, -
1,2,
by (x) 0.75
0.5
b2 (x) 0.5
F = (f(x) = c: x [0,1), [0,1]]
0.25
by (x) 0.25
0
Figure Illustration of e-upper brackets constant function class with 0.25. Each bracket function (horizontal line) lies above the function it covers, with a difference at most
F. Additional discussions on the notion of Bsem
The notion of Bsim in Section 4 is defined by induction based on our three specific model instantiations. directly measures the model parameter sharing across different sources, and thus reflects the source distribution similarity under our theoretical formulation in Section 2.2 As such, its exact formulation varies depending on the model instantiation.
Specifically, in the Gaussian model (Section 4.1 Beem بーي measures the proportion of shared mean vector dimensions, which seems to correspond to the property of the ground truth distribution. While for ARMs EBMs (Section 4.2 and Section 4.3 Beim- = s+d. is based on shared model parameters, which do not explicitly represent the data distribution itself. Despite this difference, in both cases, 3esm fundamentally represents the extent of parameter sharing across sources. The distinction arises from the modeling paradigm: the Gaussian case assumes a parametric form for distributions, where model parameters (e.g., mean vectors) explicitly encode data properties, whereas EBMs use neural networks function approximator fit probability densities without an explicit distributional form, making no explicit connection between parameters and data.
As result, Asim cannot be directly computed from general datasets without model-specific assumptions. We remark that rigorously quantifying dataset similarity in practice is still direction under exploration. Possible approaches might include: (1) From a practical perspective, a small proxy model can be used to estimate source distributions' interaction (Xic et al.) 2023). (2) From theoretical perspective, several existing notions in multi-task learning and meta-learning could be adapted for this purpose, such as transformation equivalence (Ben-David & Borbely 2008), parameter distance (Balcan et al. 2019). and distribution divergence (Jose & Simeone 2021
G. Intuitive illustration of the upper bracketing number
The c-upper bracketing number (Definition 3.1 | is a way to quantify the complexity of an infinite set of functions. The key idea is to construct a collection of "brackets" that enclose every function in the set within a small margin.
To illustrate this, consider a simple example. Suppose we have the function set F= [f())= c [o, C [0, 1]}. which consists of all constant functions taking values in the interval [0, 1]. We construct c-upper bracket for F by defining the finite set B- {b(x:) = ke A 1,..., [1/e]] }. Then, for any function f < F. there exists bracket function b E B such that: (1) For all æ E [o, 1]. the bracket function is always an upper bound: b(x) 2 f(x). (2) The total "gap" between b and f, measured by the integral fo' b(x) f(x)
most e. Intuitively, this means we can cover the entire function class using small number of simple approximating functions that "overestimate" each function just slightly. This concept is visualized in Figure 4 where show such brackets for 0.25.
In our paper, we extend this idea to conditional probability spaces. There, each condition defines its own function set, and we construct corresponding upper brackets that ensure every conditional distribution is approximated with a small error uniformly across conditions.
37


Reviews:
[{"review_id": "r6Lbdx4xwJ", "reviewer": "Reviewer_7LUU", "summary": "The paper establishes a distribution estimation error bound in average total variation distance for conditional maximum likelihood estimation. The main result is based on the bracketing number; it shows that when source distributions share certain similarities and the model is expressive enough, multi-source training guarantees a sharper bound than single-source training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: Experiments for ARMs or EBMs\n\nWe thank the reviewer for the valuable comment. \n\nFollowing the reviewer's suggestion, we have conducted supplementary simulations for ARMs according to the formulation in Section 4.2. Experimental settings and results are presented below. Generally speaking, the empirical TV errors exhibit similar trends as theoretical bounds in Theorem 4.3 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$.\n\nIn all experiments, we define a ground-truth sequential discrete distribution, enabling exact computation of the practical TV error. We fix the vocabulary size $M=2$, neural network configurations with $d_e=W=64$, $L=5$, $B=1$. Then we vary $K$ in $[1, 3, 5, 7, 10]$, $n$ in $[1000, 3000, 5000, 10000, 30000]$, and $D$ in $[10, 12, 14, 16, 18]$ to examine alignment of practical TV error with theoretical bounds. For each setting, the batch size and learning rate are selected from $\\lbrace 100, 300, 500 \\rbrace$ and $\\lbrace 10^{-5}, 10^{-4}, 10^{-3} \\rbrace$ for the minimum likelihood. Empirical TV errors are presented in the following tables: \n\n| $K \\uparrow$ | 1      | 3      | 5      | 7      | 10     |\n| --- | --- | --- | --- | --- | --- |\n| single       | 0.0763 | 0.1212 | 0.1519 | 0.1787 | 0.2127 |\n| multi        | 0.0763 | 0.1145 | 0.1318 | 0.1364 | 0.1369 |\n\n| $n \\downarrow$    | 1000   | 3000   | 5000   | 10000  | 30000  |\n| --- | --- | --- | --- | --- | --- |\n| single | 0.5680 | 0.3516 | 0.2882 | 0.2036 | 0.1212 |\n| multi  | 0.5491 | 0.3467 | 0.2747 | 0.1922 | 0.1145 |\n\n| $D \\uparrow$ | 10  | 12  | 14  | 16  | 18  |\n| --- | --- | --- | --- | --- | --- |\n| single       | 0.2036 | 0.3785 | 0.5932 | 0.7242 | 0.7505 |\n| multi        | 0.1922 | 0.3530 | 0.5068 | 0.5747 | 0.6289 |\n\nThe results show consistent trends between empirical and theoretical values with respect to $n$, $K$, and $D$, i.e., the TV error decreases as $n$ and increases with $K$ and $D$, and multi-source training generally outperforms single-source training.\n\nAdditionally, we would like to clarify the connection between our diffusion model experiments and the theoretical analysis of EBMs. As mentioned in lines 51-55 in our submission, EBMs are a general and flexible class of generative models closely connected to diffusion models. To be specific, first, the training and sampling methods in [1,2] are directly inspired by EBMs. The distinction is that EBMs parameterize the energy function, while diffusion models parameterize its gradient (the score function). Second, [3] shows that under a specific energy function formulation (Equation (5) in their paper), EBMs are equivalent to constrained diffusion models. Their experimental results (Table 1, Rows A and B) indicate that the constraint has minor impact on generative performance. Thus, our diffusion model experiments provide insight into EBMs' behavior in real-world settings to some extent.\n\nWe will provide the implementation details and simulation results for ARMs in the revised version of our paper, along with the above discussions for EBMs.\n\n[1] Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. \n\n[2] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations.\n\n[3] Salimans, T., & Ho, J. (2021). Should EBMs model the energy or the score?\n\n# Q2: Quantifying similarity for general datasets \n\nWe thank the reviewer for raising this insightful question.\n\nIn our paper, $\\beta_{sim}$ is defined by induction based on our three specific model instantiations in Section 4. It is not an inherent, directly measurable property of the source distributions themselves, meaning it cannot be directly computed given general datasets.\n\nA fundamental question underlying the reviewer's inquiry might be: *How can we quantify dataset similarity in practice with theoretical guarantees?* We acknowledge that there is no single method currently that provides a solution to this problem, and we are still exploring ways towards this goal. \n\nPossible approaches might include: (1) From a practical perspective, a small proxy model can be used to estimate source distributions' interaction [4]. (2) From a theoretical perspective, several existing notions in multi-task learning and meta-learning could be adapted for this purpose, such as transformation equivalence [5], parameter distance [6], and distribution divergence [7].\n\n[4] Xie, S. M., Pham, H., Dong, X., et al (2023). Doremi: Optimizing data mixtures speeds up language model pretraining.\n\n[5] Ben-David, S., & Borbely, R. S. (2008). A notion of task relatedness yielding provable multiple-task learning guarantees.\n\n[6] Balcan, M. F., Khodak, M., & Talwalkar, A. (2019). Provable guarantees for gradient-based meta-learning.\n\n[7] Jose, S. T., & Simeone, O. (2021). An information-theoretic analysis of the impact of task similarity on meta-learning.", "claims_and_evidence": "The paper focuses mostly on the theoretical part of the claims, which is backed by detailed proofs. Simulations and real-world experiments on diffusion models partly validate the results.", "methods_and_evaluation_criteria": "The evaluation criteria seems reasonable.", "theoretical_claims": "I am not very familiar with this particular problem. Due to time limit, I did not carefully check the proofs.", "experimental_designs_or_analyses": "I am not entirely sure if I missed something. But it seems that the paper characterizes the bracketing numbers for conditional Gaussian\n estimation, autoregressive models and energy-based models, but the real-world experiments focus largely on a particular diffusion model, i.e., EDM2. There seems to be a discrepancy between the proposed theory and its numerical validation.", "relation_to_broader_scientific_literature": "The paper establishes a new framework for the analysis of multi-source training in conditional generative modeling, where each condition represents a distinct data source. This could be potentially helpful for general multimodal data learning.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Please see the ``Experimental Designs Or Analyses`` section.", "other_comments_or_suggestions": "I would like to see more experimental results regarding autoregressive models or EBMs to validate the theoretical results.", "questions_for_authors": "Sorry if I missed this part, but can the authors advise on how to quantify the source similarity $\\beta_{\\rm sim}$ for general datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "SPvbV9XgGt", "reviewer": "Reviewer_uwo5", "summary": "This paper provides a theoretical framework proving that training conditional generative models on multiple data sources outperforms single-source training when sources share similarities. The authors instantiate their theory across Gaussian distributions, autoregressive models, and energy-based models, demonstrating that both the number of sources and their similarity improves multi-source training benefits. Simulations and experiments with diffusion models validate the theoretical findings, explaining why large generative models trained on diverse but related data often perform better than specialized models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: Close alignment in Figure 1\n\nWe appreciate the reviewer’s careful examination of Figure 1.\n\nAs detailed in lines 339-340 of our submission (the caption of Figure 1), the empirical and theoretical values are plotted on **separate vertical axes**: empirical values correspond to the left axis, while theoretical values correspond to the right axis. This visualization normalizes differences in constants between empirical results and theoretical bounds, emphasizing the comparison of trends rather than absolute values. \n\nWe will highlight the dual-axis annotation in Figure 1 in the revised version to avoid any potential confusion.\n\n# Q2: Experiments for ARMs\n\nWe thank the reviewer for the valuable comment. \n\nFollowing the reviewer's suggestion, we have conducted supplementary simulations for ARMs according to the formulation in Section 4.2. Experimental settings and results are presented below. Generally speaking, the empirical TV errors exhibit similar trends as theoretical bounds in Theorem 4.3 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$.\n\nIn all experiments, we define a ground-truth sequential discrete distribution, enabling exact computation of the practical TV error. We fix the vocabulary size $M=2$, neural network configurations with $d_e=W=64$, $L=5$, $B=1$. Then we vary $K$ in $[1, 3, 5, 7, 10]$, $n$ in $[1000, 3000, 5000, 10000, 30000]$, and $D$ in $[10, 12, 14, 16, 18]$ to examine alignment of practical TV error with theoretical bounds. For each setting, the batch size and learning rate are selected from $\\lbrace 100, 300, 500 \\rbrace$ and $\\lbrace 10^{-5}, 10^{-4}, 10^{-3} \\rbrace$ for the minimum likelihood. Empirical TV errors are presented in the following tables: \n\n| $K \\uparrow$ | 1      | 3      | 5      | 7      | 10     |\n| --- | --- | --- | --- | --- | --- |\n| single       | 0.0763 | 0.1212 | 0.1519 | 0.1787 | 0.2127 |\n| multi        | 0.0763 | 0.1145 | 0.1318 | 0.1364 | 0.1369 |\n\n| $n \\downarrow$    | 1000   | 3000   | 5000   | 10000  | 30000  |\n| --- | --- | --- | --- | --- | --- |\n| single | 0.5680 | 0.3516 | 0.2882 | 0.2036 | 0.1212 |\n| multi  | 0.5491 | 0.3467 | 0.2747 | 0.1922 | 0.1145 |\n\n| $D \\uparrow$ | 10  | 12  | 14  | 16  | 18  |\n| --- | --- | --- | --- | --- | --- |\n| single       | 0.2036 | 0.3785 | 0.5932 | 0.7242 | 0.7505 |\n| multi        | 0.1922 | 0.3530 | 0.5068 | 0.5747 | 0.6289 |\n\nThe results show consistent trends between practical TV error and theoretical bounds with respect to $n$, $K$, and $D$, i.e., the TV error decreases as $n$ and increases with $K$ and $D$, and multi-source training generally outperforms single-source training.\n\nWe will add the simulation results and implementation details for ARMs in the revised version.\n\n# Q3: Real-world experiments are on small datasets\n\nWe would like to clarify that the selection of sample sizes and the number of classes in the experiments in Section 5.2 was influenced by several inherent characteristics of ILSVRC2012 dataset:\n\n- Sample Sizes: The maximum number of images per class in ILSVRC2012 is 1300, so we selected sample sizes of 1000 and 500 images per class, which are common choices.\n\n- Number of Sources: Given that distribution similarity levels were manually defined, it was difficult to establish a large number of structured subdivisions. To be specific, to ensure reasonable similarity levels for the controlled experiment, we designed two-level tree structure for the dataset, as shown in Figure 3 on Page 35 of our submission. Overall, we divided the whole ILSVRC2012 into 10 high-level categories (mammal, amphibian, bird, fish, reptile, vehicle, furniture, musical instrument, geological formation, and utensil). Each category was further subdivided into 10 subsets (e.g., for mammals, we have Italian greyhound, Border terrier, standard schnauzer, etc.). Defining such semantically meaningful and mutually exclusive divisions is not trivial. As a result, the number of classes within each similarity level in our experiments is limited to 10.\n\nWhile our experiments are not on large-scale datasets, there are existing studies that provide valuable empirical observations for large-scale multi-source training as mentioned in our Introduction section (lines 8-13, right column), including: cross-lingual model transfer for similar languages [Pires et al., 2019], pretraining with additionla high-quality images to improve overall aesthetics in image generation [Chen et al., 2024], and knowledge augmentation on subsets of data to enhance model performance on other subsets [Allen-Zhu & Li, 2024a]. They have offered relevant findings that inform our work.\n\nWe will provide a more detailed explanation of our experimental settings in the revised version.\n\nTo summarize, we sincerely thank the reviewer for the constructive comments regarding our experiments, which we believe can improve the quality of this paper.", "claims_and_evidence": "The paper's claims are supported by theoretical proofs and empirical evidence that appear convincing.", "methods_and_evaluation_criteria": "Yes. \n\nThis paper uses bracketing numbers as a theoretical tool to measure distribution space complexity, which is well-suited for analyzing generative model estimation errors. \n\nAnd three representative model types (Gaussian, autoregressive, energy-based) are selected that cover key generative modeling approaches.\n\nThe evaluation framework connects TV error to FID, and systematically varies K and βsim identified in the theory, making the approach well-aligned with the problem being studied.", "theoretical_claims": "I'm not an expert in theoretical deep learning, but I feel the structure of the proof is sound and clear. Although I haven't checked the details of the proof, it seems solid to me.", "experimental_designs_or_analyses": "I noticed that Figure 1 shows a very close alignment between theoretical and empirical results. This perfect alignment is somewhat suspicious. From my understanding, the theoretical bounds are derived using worst-case analysis and typically contain constants that are not optimized, making perfect alignment with empirical results unusual. In most papers comparing theory and practice, you'd expect to see similar trends but with some gap between theoretical bounds and empirical measurements. Can the authors explain more about that? \n\nThe theoretical results cover three model types (Gaussian, ARM, EBM), but real-world experiments focus only on diffusion models, with no validation for autoregressive models. This should be a more important experiment. \n\nThe theory addresses large-scale generative modeling, but experiments use relatively small datasets (500-1000 images per class) and only up to 10 classes, raising questions about how well the findings generalize to truly large-scale settings.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I really like the topic of this paper and believe this is very important to the field. \n\nEven though the theoretical proof seems solid, the experiment part is weak. The whole section focuses on simple cases, and hard to evaluate the generality of the theory. There are more factors that are not involved, such as ARM.", "other_comments_or_suggestions": "See the sections above.", "questions_for_authors": "See the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E82Ef57BTT", "reviewer": "Reviewer_QALB", "summary": "This paper investigates conditional generative models with multiple data sources. It establishes a general upper bound on the MLE error. The theoretical result is then specialized to conditional Gaussian distributions, autoregressive models, and energy-based models. Finally, the theoretical findings are validated through both simulation studies and real-world experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: Characterization of multi-source advantage\n\nWe thank the reviewer for the insightful comment.\n\nWe would like to clarify that the advantage of multi-source training is indeed measured by the model parameter sharing, while the degree of the model parameter sharing reflects the source distribution similarity under our theoretical formulation (lines 94-108, right column).\n\nWe understand the reviewer’s concern. In the Gaussian model (Section 4.1), $\\beta_{sim} = \\frac{d - d_1}{d}$ measures the proportion of shared mean vector dimensions, which seems to correspond to the property of the ground truth distribution. While for EBMs (Section 4.3), $\\beta_{sim} = \\frac{S}{S+d_e}$ is based on model parameters, which does not explicitly represent the data distribution itself.\n\nDespite this difference, in both cases, $\\beta_{sim}$ is fundamentally defined by the extent of parameter sharing across sources. The distinction arises from the modeling paradigm: the Gaussian case assumes a parametric form for distributions, where model parameters (e.g., mean vectors) explicitly encode data properties, whereas EBMs use neural networks as a function approximator to fit probability densities without a predefined distributional form, making no explicit connection between parameters and data.\n\nWe will add detailed clarification on the relationship between parameter sharing, distribution similarity, and the advantages of multi-source training in the revised version.", "claims_and_evidence": "I think the advantage of multi-source training is not very convincing, and the characterization is somewhat confusing.\n\nIntuitively, multi-source training should only be beneficial when there are similarities among different classes. For example, in the Gaussian distribution setting in Section 4.1, when $d_1 = d$, i.e., there are no shared features across all sources, single-source training should perform just as well as multi-source training. Therefore, a reasonable characterization of the advantage should involve conditions on the ground truth data distribution. However, in Section 4.3, the advantage of multi-source learning is quantified by $S$ and $d_e$, both of which are parameters of the distribution family rather than direct information about the underlying data distribution.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "The experimental designs are reasonable.", "relation_to_broader_scientific_literature": "This paper provides theoretical guarantees for conditional generative models, with results applicable to both large language models and diffusion models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "This paper is overall well-written and presents a solid theoretical framework for multi-source learning.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FcHqc3YSEf", "reviewer": "Reviewer_MrF2", "summary": "This paper takes the first step toward a rigorous analysis of multi-source training in conditional generative modeling, where each condition represents a distinct data source. Specifically, the article establishes a general distribution estimation error bound in average total variation distance for conditional maximum likelihood estimation based on the bracketing number. \nResult shows that when source distributions share certain similarities and the model is expressive enough, multi-source training guarantees a sharper bound than single-source training. \nThey further instantiate the general theory on conditional Gaussian estimation and deep generative models including autoregressive and flexible energy-based models, by characterizing their bracketing numbers. Simulations and real-world experiments validate this theory.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: Intuitive explanation for upper bracketing number\n\nThe $\\epsilon$-upper bracketing number is a notion to quantify the complexity of an infinite set of functions. The key idea is to construct a finite collection of \"brackets\" that enclose every function in the set within a small margin.\n\nTo illustrate this, consider a simple example. Suppose we have the infinite function set $\\mathcal{F} = \\lbrace f(x) = c:x\\in [0,1];c \\in [0,1] \\rbrace,$ which consists of all constant functions taking values in the interval $[0, 1]$. We can construct an $\\epsilon$-upper bracket for $\\mathcal{F}$ by defining a finite set $\\mathcal{B} = \\lbrace b(x)=k\\epsilon : k=0,1,\\dots,\\lceil 1/\\epsilon \\rceil \\rbrace,$ which contains $\\lceil 1/\\epsilon \\rceil+1$ functions. Then, for any function $f \\in \\mathcal{F}$, there exists a bracket function $b \\in \\mathcal{B}$ such that: \n(1) For all $x \\in [0,1]$, the bracket function is always an upper bound: $b(x) \\ge f(x)$. \n(2) The total \"gap\" between $b$ and $f$, measured by the integral $\\int_0^1 b(x) - f(x) dx$, is at most $\\epsilon$. Therefore, the $\\epsilon$-upper bracketing number of $\\mathcal{F}$ is at most $\\lceil 1/\\epsilon \\rceil+1$.\n\nIn our paper, we extend this idea to conditional probability spaces. There, each condition defines its own function set, and we construct corresponding upper brackets that ensure every conditional distribution is approximated with a small error uniformly across conditions. \n\nWe will include additional intuitive explanations and diagrams to make this idea more accessible in the revised version.\n\n# Q2: Definition of the estimation error\n\nYour interpretation is essentially correct. In our paper, we define the error in terms of the average TV distance (see Equation 4 on line 147, left column). This metric evaluates the accuracy of conditional distribution estimates across all $K$ sources by averaging the error over each source. Therefore, even for single-source training, the error bound is related to $K$ because it aggregates the errors from the $K$ separate models. \n\n# Q3: Guarantee on one specific source\nYou have correctly captured the main idea. Our theory demonstrates that, in terms of the average distribution error, multi-source training has a better guarantee than single-source training. However, this does not necessarily imply that for every individual source, the corresponding multi-source model will yield lower error than a dedicated single-source model. Thus, your example is consistent with our theoretical findings.\n\n# Q4: Real-world experiments with larger $N$ and $K$\n\nWe would like to clarify that the selection of sample sizes and the number of classes in the experiments in Section 5.2 was influenced by several inherent characteristics of ILSVRC2012 dataset:\n\n- Sample Sizes: The maximum number of images per class in ILSVRC2012 is 1300, so we selected sample sizes of 1000 and 500 images per class, which are common choices.\n\n- Number of Sources: Given that distribution similarity levels were manually defined, it was difficult to establish a large number of structured subdivisions. To be specific, to ensure reasonable similarity levels for the controlled experiment, we designed two-level tree structure for the dataset, as shown in Figure 3 on Page 35 of our submission. Overall, we divided the whole ILSVRC2012 into 10 high-level categories (mammal, amphibian, bird, fish, reptile, vehicle, furniture, musical instrument, geological formation, and utensil). Each category was further subdivided into 10 subsets (e.g., for mammals, we have Italian greyhound, Border terrier, standard schnauzer, etc.). Defining such semantically meaningful and mutually exclusive divisions is not trivial. As a result, the number of classes within each similarity level in our experiments is limited to 10.\n\nAdditionally, for the 10-dimensional Gaussian example in Section 5.1, we used a maximum sample size of $N = 5000$ and $K = 15$ (see Figure 1(a) and (b)), which we believe are sufficiently large to verify the theoretical predictions in that case. \n\nWe will add the above explanations for the experimental settings in our revised version.\n\n# Q5: Experiments for ARMs or EBMs \n\nFollowing the reviewer's suggestion, we have added supplementary simulations for ARMs and further illustrations for EBMs. Generally speaking, for ARMs, the empirical TV errors exhibit similar trends as theoretical bounds in Section 4.2 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$. For EBMs, we clarify their connection with the diffusion model experiments. Due to space constraints in the rebuttal, please refer to our response to Reviewer 7LUU (Q1) for details.\n\n# Q6: Consistency of models used for multi/single\nYes, for both theoretical analysis and empirical experiments, the models used for multi-source and single-source training are exactly the same across all settings, such as model architecture, number of parameters, initialization, and optimizers.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "It makes much sense to the problem at hand.", "theoretical_claims": "Yes, I have checked them.", "experimental_designs_or_analyses": "Yes, I have checked them.", "relation_to_broader_scientific_literature": "The author has listed these findings in the related work and preliminary section.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "## Strength\nThis article is well organized and well written, using mathematical notations clearly and making it easy to understand. The new theory establishes a theoretical bridge between the general error bound of single-source training and multi-source training in conditional generative modelling. This will guide the researchers in choosing the data sources and models empirically and theoretically.\n\n## Weakness\n\nThis article has a small issue. Providing intuitive explanations for some theoretical concepts would be beneficial. Additionally, the assumptions in this work deviate slightly from real environments.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Can you provide an intuitive explanation of the ε-upper bracketing number? An example would be helpful.\n\n2. I initially felt some counter intuitiveness about the error bound of a single source. I believe one model training with one dataset has no relation to the whole number of datasets K, but I realize that this work computes the accumulated error of all models, so the error bound will be related to K. I don't know if I understand you correctly. I would like to discuss this with the author.\n\n3.  In real-world training, the model with large parameters for multi-source training does not always achieve better results than small expert models with single-source training. For example, when K=10 for Model=10, counting all errors of these 10 models corresponding to one dataset will be higher than a model for all these 10 datasets. However, the error of each model corresponding to one dataset will not always be higher than the model with 10 datasets. What is your opinion of this example, and does this example match your theory?\n\n4.  I would like to know if the N and K in Table 1 are set to larger values, such as N=1500 or 2000 and K =15 or 20, the results will have a similar tendency to those in Table 1.\n\n\n5. The theoretical analysis for the error bound is based on EBM and ARM; however, the article does not provide numerical results for these models but with diffusion models. I think the author should provide some experiment results on ARM or EBM, which would be better.\n\n6. Is the model for multi-source training the same as the single-source training in theoretical analysis and empirical training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3CiKPl9WT", "reviewer": "Reviewer_8TSP", "summary": "This work analyzes the effect of training with multiple data sources on conditional generative models. The authors establish a bound on the total variation distance between true and model distributions in terms of the bracketing number.\n\n## update after rebuttal\n\nThe other reviews and the rebuttal have increased my confidence in my initial estimate of the quality of this paper, so I have increased my score by 1 accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Experimental suggestion: Error estimate for FID scores\nWe thank the reviewer for the valuable suggestion regarding error estimation.\n\nWe would like to clarify that the real-world experiments in Section 5.2 were run only once due to the long training time. Based on these trained models, we additionally performed multiple samplings using five different random seeds to estimate the randomness in calculating FID scores following the reviewer's suggestion. The mean values and standard deviations of FID scores over multiple samplings are reported in the table below (corresponding to Table 1 of our submission).\n\n| $N$  | $\\mathrm{Sim}$ | $K$ | Avg. FID (Single) | Std Dev (Single) | Avg. FID (Multi) | Std Dev (Multi) |\n| --- | --- | --- | --- | --- | ---| ---|\n| 500  | 1  | 3   | 30.03  | 0.0086  | 29.94 | 0.0057 |\n| |   | 10  | 30.18  | 0.0018 | 29.28 | 0.0336 |\n| | 2 | 3 | 32.69  | 0.0160 | 30.69 | 0.0158 |\n| | | 10  | 30.54 | 0.0056 | 28.75 | 0.0035 |\n| 1000 | 1 | 3   | 28.01| 0.0034  | 26.41| 0.0064 |\n| | | 10  | 27.49 | 0.0028 | 25.84 | 0.0250 |\n| | 2  | 3   | 30.58 | 0.0047 | 29.35 | 0.0051 |\n| | | 10  | 29.01 | 0.0013 | 27.81 | 0.0084 |\n\nWe will include these results in the revised version.\n\n# Typo: Missing parentheses\nThank you for pointing this out. We will correct this in the revised version.\n\n# Q1: Experiments on diffusion models and the theory for ARMs & EBMs\n\nWe thank the reviewer for the insightful question. \n\nEBMs, as mentioned in lines 51-55 in our submission, are a general and flexible class of generative models closely connected to diffusion models. To be specific, first, the training and sampling methods in [1,2] are directly inspired by EBMs. The distinction is that EBMs parameterize the energy function, while diffusion models parameterize the score function, which is the energy function's gradient. Second, [3] shows that under a specific energy function formulation (Equation (5) in their paper), EBMs are equivalent to constrained diffusion models. Their experimental results (Table 1, Rows A and B in their paper) indicate that the constraint has a minor impact on generative performance. Thus, our diffusion model experiments provide insight into EBMs' behavior in real-world settings to some extent.\n\nAdditionally, we have added supplementary simulations for ARMs according to the formulation in Section 4.2. The empirical TV errors exhibit similar trends as theoretical bounds in Theorem 4.3 regarding several key factors—the number of sources $K$, sample size $n$, and data length $D$. Due to space constraints in the rebuttal, please refer to our response to Reviewer 7LUU (Q1) for detailed experimental settings and results.\n\nWe will add the above discussions for EBMs, and simulation results along with implementation details for ARMs in the revised version of our paper.\n\n# Q2: Obstacles for practical application\n\nAs discussed in Section 7 in the submission (lines 422-432, right column), our theoretical formulation of multi-source training through conditional generative modeling abstracts real-world scenarios to some extent. In practice, conditions may not be explicitly given (e.g., in language models) or may involve multiple source labels (e.g., large-scale image generation).\n\nOur analysis provides a first step toward understanding multi-source training under a simplified yet reasonable setting. Extending it to more complex, fine-grained multi-source interaction scenarios is a valuable direction for future work. Possible approaches might include: characterizing distribution similarity without explicit conditions [1,2] or investigating the multiple-label case by compositional generative modeling [3, 4].\n\n[1] Ben-David, S., & Borbely, R. S. (2008). A notion of task relatedness yielding provable multiple-task learning guarantees.\n\n[2] Jose, S. T., & Simeone, O. (2021). An information-theoretic analysis of the impact of task similarity on meta-learning.\n\n[3] Okawa, M., Lubana, E. S., Dick, R., & Tanaka, H. (2023). Compositional abilities emerge multiplicatively: Exploring diffusion models on a synthetic task.\n\n[4] Lake, B. M., & Baroni, M. (2023). Human-like systematic generalization through a meta-learning neural network.\n\n# Q3: Connection between FID and theoretical guarantees\n\nOur theory provides guarantees for the average TV distance (lines 142-155, left column), which quantifies distribution estimation quality but is incomputable without access to the true conditional distributions.\n\nTherefore, in real-world experiments (Section 5.2), we use FID as a practical alternative. FID measures the similarity between generated and real data distributions by comparing their feature representations in a pretrained neural network. It is widely used to evaluate image generation quality and serves as the best available metric for our setting.\n\nWe will add the above discussion to clarify the choice of FID in the revised version.", "claims_and_evidence": "The experimental data aligns well with the theoretical bounds for the Gaussian case. For ARMs and EBMs, the connection to the experiment with diffusion models should be more clearly delineated.", "methods_and_evaluation_criteria": "Yes, the combination of a simple setup with Gaussians and an empirical evaluation based on diffusion models for image data fits well.", "theoretical_claims": "I skimmed the appendix but did not check the correctness of the proofs in detail.", "experimental_designs_or_analyses": "There is no explicit experimental design. The standard deviations on the results seem reasonably small based on the smoothness of the curves but it would be great to report an error estimate for the FID scores.", "relation_to_broader_scientific_literature": "\\-", "essential_references_not_discussed": "\\-", "other_strengths_and_weaknesses": "I don't see any specific weaknesses but I am not knowledgeable enough about this field to judge the importance of these results to the wider community. My impression is that this paper examines a practically relevant setting theoretically in a sound way and and validates the main results empirically.", "other_comments_or_suggestions": "- I believe in line 61 there should be parentheses around $K - 1$.", "questions_for_authors": "1. How exactly does the diffusion model experiment relate to your bounds for ARMs and EBMs?\n2. What are the obstacles for applying these bounds to practical generative models?\n3. Is there a direct connection between the FID and your theoretical guarantees?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a theoretical framework for analyzing multi-source training in conditional generative modeling, establishing distribution estimation error bounds that demonstrate when and why multi-source training outperforms single-source approaches. The authors provide an analysis through the lens of bracketing numbers, instantiating their theory across conditional Gaussian estimation, autoregressive models, and energy-based models. Reviewers acknowledged the paper's theoretical soundness and appreciated the authors' thorough responses to concerns about experimental validation, particularly their additional ARM simulations and clarification of the connection between EBMs and diffusion models. While some reviewers initially questioned the characterization of source similarity and the scale of experiments, the authors addressed these points in their rebuttal, providing additional results and explaining practical constraints. Given the paper's theoretical contribution to understanding an important phenomenon in generative modeling and the authors' willingness to enhance their experimental validation, I recommend acceptance of this paper to ICML 2025.
Decision: Accept (poster)