On Statistical Rates of Conditional Diffusion Transformers: Approximation,  Estimation and Minimax Optimality
Authors: Jerry Yao-Chieh Hu
Weimin Wu
Yi-Chen Lee
Yu-Chao Huang
Minshuo Chen
Han Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Conditional Diffusion Transformer
Statistical Rates
Approximation
Estimation
Primary area: generative models
TL;DR: We investigate the approximation and (minimax optimal) estimation rates of Conditional Diffusion Transformers (DiTs) with classifier-free guidance.

Abstract: We investigate the approximation and estimation rates of conditional diffusion transformers (DiTs) with classifier-free guidance. We present a comprehensive analysis for “in-context” conditional DiTs under various common assumptions: generic and strong Hölder, linear latent (subspace), and Lipschitz score function assumptions. Importantly, we establish minimax optimality of DiTs by leveraging score function regularity. Specifically, we discretize the input domains into infinitesimal grids and then perform term-by-term Taylor expansions on the conditional diffusion score function under the Hölder smooth data assumption. This enables fine-grained use of transformers’ universal approximation through a more detailed piecewise constant approximation, and hence obtains tighter bounds. Additionally, we extend our analysis to latent settings. Our findings establish statistical limits for DiTs and offer practical guidance toward more efficient and accurate designs.

Published as a conference paper at ICLR 2025
ON STATISTICAL RATES OF CONDITIONAL DIFFUSION TRANSFORMERS: APPROXIMATION, ESTIMATION AND MINIMAX OPTIMALITY
Jerry Yao-Chieh Hu
Weimin Wu
Yi-Chen Lee
Yu-Chao Huang*!
Minshuo Chen
Han Liu'
Worthweettern University
National Taiwan University
(jhu, wwm @u www.hwesteern...om edu, b10202055@ntu. .dw..tedddutc.....
yuchachuangeg ntt..nuudde edu. tw, (minshuo. chen, hamiiu (ommiiu/@northnnn @hoorhwwestern.dd
ABSTRACT
We investigate the approximation and estimation rates of conditional diffusion transformers (DiTs) with classifier-free guidance. We present comprehensive analysis for "in-context" conditional DiTs under various common assumptions: generic and strong Hölder, linear latent (subspace), and Lipschitz score function assumptions. Importantly, we establish minimax optimality of DiT's by leveraging score function regularity. Specifically, we discretize the input domains into infinites- imal grids and then perform eerm-by-term Taylor expansions on the conditional diffusion score function under the Hölder smooth data assumption. This enables fine-graine fine-grained use of transformers' universal approximation through more detailed piecewise constant approximation. and hence obtains tighter bounds. Additionally, extend our analysis to latent settings. Our findings establish statistical limits for DiTs, and offer practical guidance toward more efficient and accurate designs.
INTRODUCTION
We investigate the approximation and estimation rates of conditional Diffusion Transformers (DiTs) with classifier-free guidance. Specifically, we derive score approximation, score estimation, and distribution estimation guarantees for "in-context" conditional DiT's introduced by Peebles and Xie (2023). We provide comprchensive analysis under various data and score function assumptions, in- cluding generic and strong Hölder, linear latent (subspace), and Lipschitz score function assumptions. Moreover, show that the analysis of both conditional DiT's and their latent variants lead to the first known minimax optimality of unconditional DiTs under the strong Hölder data assumption.
Transformer- based conditional diffusion models are at the forefront of generative AI due to their success as scalable and flexible backbones for image (Zhao et al., 2024; Wu et al., 2024a; Bao et al., 2023; Batzolis al., 2021) and video generation (Liu al., 2024; Ni et al., 2023; Saharia et al., 2022; Voleti et al., 2022). However, the theoretical understanding of conditional DiT's remains limited. On the one hand, while prior work (Hu et al., 2024) reports approximation and estimation rates of DiTs using the established universality of transformers (Yun et al., 2020), their results are not tight and are limited unconditional diffusion. On the other hand, existing theoretical works on conditional diffusion models only focus on ReLU networks (Fu et al., 2024a; Yuan et al., 2024), model-free settings (Ye et al., 2024; Guo et al., 2024) or generative sampling process (Dinh et al., 2023), without considering the transformer architectures. This work addresses this gap by providing a timely analysis of the statistical limits of both conditional and unconditional DiTs.
In this work, we present comprehensive analysis of conditional DIT and its latent setting under four common data and score function assumptions. We also establish the minimax optimality of unconditional DiT and its latent version by deriving the tight distribution estimation error bounds. Our techniques include two key parts: (i) Discretizing the input domains into infinitesimal grids. (ii) On each grid, performing term-by-tern Taylor expansions the conditional diffusion score function under generic and stronger Hölder smooth data assumptions, motivated by the local diffused
Equual contribution. Full version and future updates available arXiv.

---

Published as a conference paper at ICLR 2025
Table 1: Summary of Theoretical Results. The initial data d, d--dimeniionl,, and the condition is d,- dimensional. Furthermore, is the feature dimensions of transformer network function class (Definition 2.2). For latent DIT. the latent variable is do o--immnniiona.. of = 1-- e is the denoising scheduler. The sample size is and 0 <, < 1 represents the precision parameter. While report asymptotics for large d2, do, we reintroduce the dependence in the estimation results to emphasize sample complexity convergence. Lastly, we adopt standard ().). (()),(()))))))))) to omit constant factors and (()) to omit logarithmic factors.
Assumption
Score Approximation
Score
Dist. Estimation (Total Variation Distance)
Minimax Optimality
Estimation
Generic Hölder Smooth Datu Dist. (Sections 3.1 and 3.3)
(((1og(2)))(o /o8)
n --(1144) (log (ld.)
n 1-000 (log n) (ld))
x
Strong Hüilerr Smooth Data
Ō(P%o7)
n-8(1/4) (log ()44))
--0010 (log ((.))
Dist. (Sections 3.2 and 3.3)
Latent Subspace + Generic
Hölder Smooth Data Dist.
(1))())))))))
n 9 (log n) ((d))
B(1/da) (log n) Olie
x
(Appendix A)
Latent Subspace + Stronger Hölder Smooth Data Dist.
0(2%o7)
-- -(11)) (log n) ((4.)
((1/4)) - (log n) (io)
(Appendix A)
polynomial analysis (Fu et al., 2024a; Oko et al., 2023). These techniques leverage the nice regularity of the score function imposed by the Hölder smoothness data assumptions and hence enable fine-
grained use of transformers' universal approximation (Kajitsuka and Sato, 2024; Yun et al., 2020) through a more detailed piecewise constant approximation. Consequently, we obtain tighter bounds.
Contributions. We summarize the theoretical results in Table 1. Our contributions are threefold:
Score Approximation. We characterize the approximation limit of matching the conditional DiT score function with a transformer-base score estimator. The approximation results explain the expressiveness of conditional DiT and its latent version, and guide the score network's structural configuration for practical implementations (Theorems 3.1, 3.2 and A. The results also show that the latent version achieves a better approximation for the score function.
Score and Distribution Estimation. We study the score and distribution estimation of conditional DiTs in practical training scenarios. Specifically, we provide a sample complexity bound for score estimation (Theorems 3.3 and D.3), using norm-based covering number bound of transformer architecture. Additionally, we show that the learned score estimator can recover the initial data distribution in both conditional DiT and its latent setting (Theorems 3.4 and A.2).
Minimax Optimal Estimator. We extend our analysis unconditional DIT and investigate whether the generated data distribution achieves the minimax optimality in the total variation distance. Specifically, we show that the upper bounds on the distribution estimation error match the lower bounds under stronger Hölder smooth data distribution (Theorem 3.5 and Remark A.3).
Organization. Section presents preliminaries and the problem setup. Section presents the results of conditional DiTs. Appendix A presents the results of latent conditional DiTs. Appendix C. presents related works' discussions. The appendix contains an extended and improved version of (Hu et al., 2024) on conditional DiTs (Appendix E), additional results, and detailed proofs.
The index set {1..... T} is denoted by [ where I N We denote (column) vectors by lower case letters, and matrices by upper case letters. Let a[i] denote the i-th component of vector a. Let Aij denotes the (i, -tth entry of matrix A. | || and || denote the Euclidean norm, 1-norm, and infinite norm. |[W [|2 and || ||ll, denote the spectral norm and Frobenius norm. Lastly, we adopt standard ()),. e(-) to omit constant factors and Õ-) to omit logarithmic factors.
Notations.
2 BACKGROUND AND PRELIMINARIES
In this section, we provide high-level overview of the conditional diffusion model with classifier-free guidance in Section 2.1 and conditional Diffusion Transformer (DiT) networks in Section 2.2.
2.1 CONDITIONAL DIFFUSION MODEL WITH CLASSIFIER-FREE GUIDANCE
Forward and Backward Conditional Diffusion Process. In the forward process, conditional diffusion models gradually add noise to the original data zu E Rd Give condition y Rd, and 2n ~ P-y)).). Let x, denote the noisy data at the timestamp t, with marginal distribution and density (:(- and pe(- ly). The conditional distribution Pr (y) follows Noaao, of la ) where
2

---

Published as a conference paper at ICLR 2025
Reshape Layer
Reversed Reshape Layer
R(-)
Rdxt
Concal
Thin
(+)
R
R
Embed
gaxa
Transformer Network
Label Timestep
Figure I: Conditional DiT Network Architecture. The architecture consists of reshape layer R(-), reversed reshape layer R (-), and the embedding layers for label y and timestep The embeddings of y and t. are concatenated with input sequences and then processed by transformer network fr E TThis...
a nondecreasing weighting function. In practice, the forward process terminates large enough T' such that Pr is close to N (0, la,). In the backward process, we obtain z; by reversing the forward process. The generation x depends on the score function V log (-(v)). See Appendix F.1 for the details. In below, when the context is clear, suppress the notation dependence of x4 on the time step t.
or - of = e and w(() >
Classifier-Free Guidance. Classifier- free guidance (Ho and Salimans, 2022) is the standard workhorse for training condition diffusion models. It approximats both conditional and unconditional score functions using neural networks sw with parameters W. It uses the following loss function:
E(zo. yi sw) 1 T to E, R (( [|sw (ee,Ty, t) Va, log oe (Ie) dt,
where 22, log 6: (zz( = (xx - 20/ to is small cutoff to stabilize training'. T = denotes the unconditional version, id denotes the conditional version, and P(7 0) P(r id) = 0.5. To train sw, we select n i.i.d. samples {2o,5, Wil.n. where To, ~ Po(-lys). We use
((w) = : Σ (100,11155 165 3658w)
(2.1)
as the empirical loss. In addition, denote population loss as C(sw). See Appendix F.2 for details.
2.2 CONDITIONAL DIFFUSION TRANSFORMER NETWORKS
We use a transformer network as a score estimator sw. Our notation follows (Hu et al., 2024).
Transformer Block. Let (SA), RdxL + R denote the self-attention layer. Let h and denote the number of heads and hidden dimension in the self-attention layer, and then we have
((S)) (2) = 2 + Σ wy (Wi-2) Softmax [W (Wi [[W)2)
(2.2)
where Wi. Wi W E Raxd and W gdxs are the weight matrices. Next, we define the feed-forward layer with MLP dimension
((F) (Z) = Z W.ReL +b, b2.
(2.3)
where W1)) e grxd and ((2) ddxr are weight matrices, and ((1) e R2, and 6(2) gd are bias.
Definition 2.1 (Transformer Block). We define a transformer block of h-head, s-hideen dimension. r-MLP dimension, and with positional encoding E E Rdd2 as
hh.s.r (2) = ( (S(S)) (2+ F?) RdxL + IR
Now, we define the transformer networks compositions of transformer blocks.
foo is the carly stopping time prevent the score function from blowing up (Fu et al., 2024a; Chen et al., 2023c; Dhariwal and Nichol, 2021; Song et al., 2021).
3

---

Published as a conference paper at ICLR 2025
Definition 2.2 (Transformer Network Function Class), Let Thisk denote the transformer network function class where each function E ETH.s, is composition of transformer blocks fhr,.., i.e.
The = {T TR R + RdxL | fhiss ..... fhe...
Conditional Diffusion Transformer (DiT). Let f E Thisir be transformer network, and (x; y, t) E _ x 1d x [ T" be the input data. We follow the "in-context conditioning" conditional DiT network in (Peebles and Xie, 2023) as in Figure I. The following reshape layer converts a vector input &d into the sequential matrix input format Z R for transformer with d₂ d L.
Definition 2.3 (DiT Reshape Layer R(-)). Let R(-) Rd + R be a reshape layer that transforms the dy-dimensional input into a d x L matrix. Specifically, for any d=, = x i image input, R(-) converts into sequence representation with feature dimension d p2 (where p 2 2) and sequence length == (i/p)². Besides, we define the corresponding reverse reshape (flatten) layer R (-) R d the inverse of R(-). By de dL, R. R are associative w.r.t their input.
We define the following transformer network function class with the reshape layer. To simplify, we define W = (WK) Wq and Wov = WoWy.
Definition 2.4 (Transformer Network Function Class with Reshape Layer TR Th....). The transformer network class with reshape layer T (CT, C CKQ Cg Cov, CE C2 02.00 CCp, LT) satisfies:
T = {R- o fr o R: Rd + Rd fr E Th.ssr
Transformer network output bound: supz [ S CT:
Parameter bound in F(FF) max([] 12,00 || S c max[ [2ll2) S Cb:
Parameter bound in F((A); |[Wxoll2 & CKQ. ||ovll2 S Cov. ||Kooll..o S OKQ
[|Wov S C || || 112,00 CE., where co-norm follows 12,00 max 1
Lipschitz of fr Th.s.r (lfT(Z1) - (T2))l S LT Zell for any Z1. Zg RdxL
These norm bounds are critical to quantify the interplay between model, performance and data.
STATISTICAL LIMITS OF CONDITIONAL DITS
In this section, we present a refined decomposition scheme for the fine-grained analysis of score approximation, score estimation, and distribution estimation in conditional DiT. Our analysis con- siders two assumptions on initial data distributions: (i) generic Hölder smooth data assumption (Section 3.1 for approximation, and Section 3.3 for estimation), (ii) stronger Hölder smooth data assumption (Section 3.2 for approximation, and Section 3.3 for estimation). This new scheme leads to tighter bounds, including the minimax optimality of the unconditional DiT score estimator.
3.1 SCORE APPROXIMATION: GENERIC HÖLDER SMOOTH DATA DISTRIBUTIONS
We present a fine-grained piecewise approximation using transformers to approximate the conditional score function under the Hölder smoothness assumption on the initial data (Fu al., 2024b). At its core, we introduce a score function decomposition scheme with term-by-temn tractability.
We first introduce the definition of Hölder space and Hölder ball following (Fu al., 2024b).
Definition 3.1 (Hölder Space). Let a E z and let 3 = K + denote the smoothness parameter, where k, = [ and 7 E [0, 1). For a function f : Rd + R. the Hölder space H (Rd)) is defined as the set of a-differentiable functions satisfying: ((Rd) = { R2d + RR | < oo). where the Hölder norm (R) satisfies:
| f(x) 8 f(x'l
[[]l)))())) = Σ sup 0 f(x)]
+
max [al|:
sup 2Øt2'
a: a:||a]]
| - 11
||<]]] <ki
We also define the Hölder ball of radius B: H. (R., B) = : .d R [1l < B).
Let 3o E dRd denote the initial data, and E [0, 1j the conditional label. With Definition 3.1, we state the first assumption on the conditional distribution of initial data To.
4

---

Published as a conference paper at ICLR 2025
Assumption 3.1 (Generic Hölder Smooth Data). The conditional density function oo( is defined on the domain IRd [0. 1 1dd and belongs to Hölder ball of radius B > for Hölder index > 0,
denoted by Po y) E H d x [0, 10. B) (see Definition 3.1 for precise definition.) Also, for
any E [0, 1],, there exist positive constants C1. C2 such that po ) S C1 exp - 1l
Remark 3.1. The Hölder continuity assumption captures various smoothness levels in the conditional density function. The light-tail condition relaxes the bounded support assumption in (Oko et al., 2023). Moreover, Assumption 3.1 only applies to the initial conditional distribution and imposes no constraints on the induced conditional score function. This is far less restrictive than the Lipschitz score condition in prior works (Yuan et al., 2024; Lee et al., 2023; Chen et al., 2022).
In our work, we aim approximate the conditional score function V log pr ly) using transformer architectures. Hu et al. (2024) analyze the unconditional DiTs based on the established universality of transformers (Yun et al., 2020). These theories discretize the input and output domains into in- finitesimal grids and employ piecewise constant approximations to construct universal approximators with controllable errors. However, such methods do not yield tight bounds for DiT architectures (Hu et al., 2024). To combat this, we build on the key observation by Fu et al. (2024a)²:
(= = SSL. Tie (2T)d./2
dæo
||a,tto - xel 20
(ooll
exp
(3.1)
serderddr Taylor polynomial
Roke-order Taylor polynomial
A term-by-term Taylor expansion of the above conditional distribution under Assumption 3.1 enables more fine-grained analysis. As result, we propose a fine-grained version of piecewise constant approximation for conditional DiTs, allowing transformers to approximate the conditional score func- tion with tighter error bounds. In particular, we utilize a refined transformer universal approximation modified from (Kajitsuka and Sato, 2024) (see Appendix G. for details).
Our score approximation procedure has two stages: first, we construct a score approximator by incorporating the approximation of and Vp; using a Taylor expansion, then use transformers to approximate the score approximator. These lead to provably tight estimation results in Section 3.3.
We state our main result of score approximation using transformers under Assumption 3.1 follows:
Theorem 3.1 (Conditional Score Approximation under Assumption 3. Assume Assumption 3.1. For any precision parameter 0) < < 1 and smoothness parameter /3 > 0, let I O(N-2) for some N N. Let Ca, C. > 0 be some absolute constants. For any y E [0, 1]d, and E [N-C2 C. C. log N]. there exists a Tacore (I. y, E &T Th.e.r such that
Tocare (x, y, () - - Po (xly)) = o ( N (log N))+
B²
Notably, for O(N- the approximation error has the upper bound O(log (2) / The parameter bounds for the transformer network class are as follows:
CKQ: CQQ O ( (log. N) 2d)++) Cov, Cor O(N
CF, CF,02.00 (N (log N) 8++
): C: = 0(1; - O(viog N 1o?)
where 0(-) hides all polynomial factors depending on dz, d, L, C2.
Remark 3.2. N is the resolution of the input domain discretization. We remark that domain discretization is essential for utilizing the local smoothness of functions under Hölder assumptions. Furthermore, Cn and C, control the stability cutoff and early stopping time, respectively.
Sketch. Sketch. Recall that V log Pe (xly) Vpe(sly) (=lp)
Our proof follows three steps:
Step 1: Smooth Local Approximations. A ki-th order and a k2-th order Taylor expansion for Po(#ly) (y) and exp(-) yield two explicit functions fi fi y, and fa(x, y) that approximate pr(xly)
Recall that (=(x)l)) -Jaa, p(zoly) (e (=. dar with P.(-ly) N af Jaz). In below, when the context clear, we suppress the notation dependence of x, on the time step
5

---

Published as a conference paper at ICLR 2025
and (zl() in Lemma H.3 and Lemma H.4, respectively. Both fi gx) and fz(2. y, t) inherit the Hölder smoothness. Altogether, this gives score approximator by Pe (æly) (p.(xll))
(=(a))
Step 2: Transformer Approximation on a Bounded Domain. We leverage the universal approxi- mation capabilities of transformers approximate the score approximator
bounded domain.
Step 3: Extension to the Full Space via Sub-Gausiannity We extend the bounded-dones approximation full space using the target density's 's sub-Gaussian tails. Gaussian tail bounds cap the error outside the domain and maintain the overall approximation accuracy.
Error Matching. The overall enror includes ErrorTaylor and Errory. Given fixed discretization resolution N, Errortaylor remains fixed. However, the approximation error bound of the transformer can be an arbitrary value. We align Errory and ErrorTaylor optimize the final results.
Please see Appendix H for a detailed proof.
Remark 3.3 (Approximation Rate), Given fixed resolution N, the approximation error scales inversely with the smoothness 3. As the smoothness increases, we get a tighter approximation error
Remark 3.4 (Comparing with Existing Works), Fu et al. (2024a) provide approximation rates for conditional diffusion models using ReLU networks. We the first to establish approximation error bounds with transformer networks. Additionally, Oko et al. (2023) establish approximation rates under a compactness condition on the input data. We mitigate this compactness requirement by applying a Hölder smoothness assumption to control approximation eπror outside compact domain.
3.2 SCORE APPROXIMATION: STRONGER HÖLDER SMOOTH DATA DISTRIBUTIONS
Next, we study the conditional DiT score approximation problem using our score decomposition scheme under the stronger Hölder smoothness assumption from Fu al. (2024b, Assumption 3.3).
Assumption 3.2 (Stronger Hölder Smooth Data). Let function f E H. (R x [0, 111T, B). Given a constant radius B, positive constants C and C22, we assume the conditional density function f y) and f(ro. y) 2 C' for all (zo. y) dd |0, 1]
p(zoly) exp [C
Assumption 3.2 imposes stronger assumption than Assumption 3.1 and induces a refined conditional score function decomposition. Explicitly, by Lemma I.1, V log Pt (zly) becomes:
-C2x
Vh(z,v,()
V Vlogp(
at C2a7 + h(5,8,1)
(3.2)
where f1zz v. = Sgu.
((D..) 87th (2x)dz/2 exp
[m0 1000,,01' dro. δ. oi and ag
20
Vat+Cart
ag a2+C2a7
We highlight that (3.2) leads to a tighter approximation error compared with Theorem 3.1. Intuitively. Assumption 3.2 imposes lower bound on the conditional density function and hence implies in better regularity of the score function. In contrast, under Assumption 3.1, 1, the score function lacks such regularity and may explode when Pt is small. These low-density regions act holes in the data support. They cause the score function to diverge near the boundary of these holes. To combat this, an implication of (3.2) is handy- - is bounded from zero, ensuring that the score function remains well-hehaved across the entire data domain. To elaborate more, two technical remarks are in order.
Remark 3.5 (Linearity). The first term on the RHS of (3.2) is linear in This makes part of V log Pe (xly)) linear function of ær enabling easy approximation with tighter bound.
Remark 3.6 (Tightened Approximation Induced by h's Lower Bound). Moreover, the introduction of h tightens the approximation error due to the lower bound imposed by Assumption 3.2 (i.e., f(x, v) 2 C). The second term on the RHS of (3.2) mirrors the form V log (.(l)) by replacing p with h. In the analysis of Section 3.1, especially in Step of the proof (resembling f2 to approximate Vp. (zly)), we have to impose a threshold on the denominator of Vp.(zlu) to (=(l))
Vpe(mly) P- (>())))
prevent score explosion under Assumption 3.1. This threshold introduces additional approximation error (Lemma H.6). Assumption 3.2 remedies this by ensuring lower bound on Pr (æly) through the
minimum values of f (xx y) and exp( - C22 | 222 within the compact domain after discretization. Setting this lower bound eliminates the need for threshold and improves the approximation.

---

Published as a conference paper at ICLR 2025
Consequently, decomposition (3.2) improves approximation result from Section 3.1. We state main result of score approximation using transformers under Assumption 3,2 as follows:
Theorem 3.2 (Conditional Score Approximation under Assumption 3.2, Informal Version of Theo- rem I.I.. Assume Assumption 3.2. For any precision parameter 0 < < and smoothness parameter
33 > 0, let e I ()N-8) for some N. Let Ca, C. > be some absolute constants. For any E [0, 1j4y and t E [N-C= Ca log N). there exists Tecore (x, 3, R sh.. such that
B of
Jdd ||Toooee (z, y, V log pr(zly)dz 0
N (log No)+1.)
Notably, for B), the approximation error has the upper bound ((E2/o?).
Intrinsically, is proportional to the size of the transformer network Tanore (see Theorem 1.1). Hence, the precision parameter is inversely proportional the size of Tecore. Therefore, Theorem 3.2 specifies the required configuration (e.g., size) of Tscore for a desired score approximation accuracy.
Proof Sketch. The proof closely follows Theorem 3.1, I. but uses a different conditional score function decomposition in the form of (3.2). We highlight this key distinction Lemma I.1.
Please see Appendix for a detailed proof, and see Theorem I.1 for the formal version.
Remark 3.7 (Comparing with Theorem 3.1). Let (()) hide the terms about to, log to, log n. In Theorem 3.2, the approximation rate (NN-2)) is faster than that of Theorem 3.1, i.e., ONN-))
3.3 SCORE ESTIMATION AND DISTRIBUTION ESTIMATION OF CONDITIONAL DITs
Next, we study score and distribution estimations based on the two score approximation results for two different data assumptions: Theorems 3.1 and 3.2. Let denote the trained score estimator.
Building on our approximation results from Sections 3.1 and 3.2, the next objective is to evaluate the performance of the score estimator trained with set of finite samples by optimizing the empirical loss (2. 1). To quantify this, we introduce the score estimation risk.
Score Estimation.
Definition 3.2 (Conditional Score Risk). Given score estimator 8, we define risk as the expectation of the squared & difference between the score estimator and the ground truth with respect to (1t, y. t):
(
8
E... ( v, - pa
T to
to
Given set i.i.d sample {.;, jej] direct computation of Ex [R)) is infeasible due to the absence of access to the joint distribution P(z,,)) y). To address this, we: (i) Decompose the risk into estimation and approximation errors, (ii) Bound the estimation error using the covering number of transformers, and (iii) Bound the approximation error using Theorem 3.1 and Theorem 3.2.
Theorem 3.3 (Conditional Score Estimation with Transformer). Consider y E [0, 1]d and t E [to. T with 4g = N-C. and C. log N, where C., C. are absolute constants such that 4o Furthermore, let 16,3 + 12,3 and U2 = 20d, 4,3 + 18.
Assume Assumption 3.1. Then, it holds
E [R(3) O (n "ito, (log )1)
Assume Assumption 3.2. Then, it holds
(maw)) [R(3]] O 03 (log mognn)' )2
The upper bounds in Theorem 3.3 arises from the fundamental bias-variance trade-off. A key aspect of the risk analysis comes from treating n i.i.d. training samples as random variables drawn from the same distribution. Bounding this expectation relies on the log-covering number of transformer network class, characterized by 21, 1/2 and 1/3 Specifically, the difference between the risk computed from the actual sample and computed from the closest element in the cover remains small. Controlling
7

---

Published as a conference paper at ICLR 2025
this gap ensures that the overall risk bound holds. In particular, A larger log-covering number corresponds to a more expressive model, leading to higher variance in the learned estimator. One the other hand, the bias term arises from the approximation eπror from Theorem 3.1 and Theorem 3.2. Reducing the approximation error requires increasing network capacity, while expanding the function class raises the log-covering number. The overall risk bound follows from balancing these two effects.
Proof. Please see Appendix J.2 for detailed proofs.
Remark 3.8 (Sample Complexity Bounds). To obtain e-error in terms of score estimation, we have the sample complexity ((e +38)// under Assumption 3.1 and ((e-(1x+63)//2)) under Assumption 3.2. Here (-)) ignores the terms about to, log to and log n. The Hölder data smoothness degree 3 affects the sample complexity. This indicates that the regularity of the initial data distribution determines the complexity of score estimation.
Distribution Estimation. Next, we study the distributional estimation capability of the trained conditional score network s(z, y, t) by analyzing the total variation distance between the estimated and true distributions. Our strategy uses a three-part decomposition: (i) the total variation between the true distributions at timestamps 0 and to. (ii) the total variation between the true distribution at to and the reverse process distribution using the true score function, and (iii) the total variation between the reverse process distributions using the true and estimated score functions at la.
Theorem 3.4 (Conditional Distribution Estimation). For [0, 11 let P.g (-lv) denote estimated conditional distributions to. Recall that Po(-ly) is the conditional distribution of initial data zo given y and t E [N- C. log N). Assume KL (Po (0, 1)) S. e for some constant c < co. Assume Assumption 3.1. With ii and 1/, specified in Theorem 3.3, it holds
Ey TV Pr (.v).. Po (8(ww) O (n (log n) () 2+
E 1
where min
C 2(1++C, +38)* t +38 2(m CC.-383
C.
Assume Assumption 3.2. With v1 and 12, specified in Theorem 3.3, for all e IR,,, it holds
E
By
TV (Pro (-1)), Pol (o(-lw)]] o (n (oogllog)) 7+3)
where @ = min
Co
Co +65
2(v1
+1+63
Proof. Please see Appendix J.4 for detailed proof.
3.4 MINIMAX OPTIMAL ESTIMATION OF UNCONDITIONAL DITS
We are now ready to present our minimax optimality result for unconditional DiTs. Recall the minimax optimal rate for distribution estimation under the strong Hölder assumption (Assumption 3.2).
Lemma 3.1 (Proposition 4.3 of (Fu et al., 2024b) and (Yang and Barron, 1999)). For fixed constant
C2 and Hölder index B O. We consider the task estimating a probability distribution P(x) with its density function defined within the following function space
p {p(x)) = [(xx) s(=))xx((Caalll1) f(x)) (R B), ())
Given n i.i.d data {- we have inf a suPpEr Eg [TV P)] Q(n 2オサ28 Here, the estimator ī ranges over all possible estimators constructed from the data.
Setting dy 0, show unconditional DiTs match the minimax optimal rate under specific conditions.
Theorem 3.5 (Minimax Optimality of DiTs), Recall Theorem 3.4. The minimax optimality of unconditional diffusion transformer holds under C, = 2C', 233 and 163(d + d,
Proof. Please see Appendix J.5 for detailed proofs.
8

---

Published as a conference paper at ICLR 2025
Remark 3.9. Since do not impose any assumptions condition y (in contrast to data x:), Theorem 3.5 establishes only the minimax optimality of unconditional DiTs by setting dy = O. We leave the minimax optimality of conditional DiTs for future works.
Remark 3.10 (Comparing with Existing Works), Oko et al. (2023) analyze the ReLU network and provide the near minimax optimal estimation rates in both the total variation distance and Wasserstein distance of order one. Fu et al. (2024b) utilize the ReLU network and provides the minimax optimality for distribution in total variation as well. Our results offer the first and exact minimax optimal guarantee for unconditional DiTs in distribution estimation.
DISCUSSION AND CONCLUSION
We investigate the approximation and estimation rates of conditional DiT and its latent setting. We focus the "in-context" conditional DiT setting presented by Peebles and Xie (2023), and conduct a comprehensive analysis under various common data conditions (Section 3 for generic and strong Hölder smooth data, Appendix A for data with intrinsic latent subspace).
Interestingly, we establish the minimax optimality of the unconditional DiTs' estimation by reducing our analysis of conditional DiTs the unconditional setting (Section 3.4 and Remark A.3). Our key techniques include a well-designed score decomposition scheme (Section 3.1). These enable a finer use of transformers' universal approximation, compared the prior statistical rates of DiTs derived from the universal approximation results in (Yun et al., 2020) by Hu et al. (2024).
Consequently, we provide three extensions the appendix:
In Appendix A, we extend the results from Section 3.2 by assuming the input data has an intrinsic liver-dimensional representation. Importantly, we establish the minimax optimality of estimation for such latent DiTs.
In Appendix D, we expand Appendix and extend our well-designed score decomposition scheme from Section 3 to the latent conditional DiT. Notably, we also obtain provably tight rate, i.e., for distribution estimation under Assumption 3.2 (Remark A.3).
In Appendix E, we extend the analysis of (Hu et al., 2024) the conditional DiT setting and provide an improved version. In particular, we analyze conditional latent DiT's under the following three assumptions from (Hu et al., 2024) and obtained sharper rates:
Low Now-Dimensiooal Linear Latent Space Data (Assumption A.1)
Lipschitz Score Function (Assumption E.2)
Light Tail Data Distribution (Assumption E.3)
In detail, we use a modified universal approximation the single-layer self-attention transformers (modified from (Kajitsuka and Sato, 2024)) to avoid the need for dense layers required in (Yun et al., 2020). This refinement results in tighter error bounds for both score and distribution estimation. Consequently, our sample complexity error bounds avoid the gigantic double exponential term 20/e)²* reported by Hu al. (2024), and obtain sharper rates than those of (Hu et al., 2024).

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
JH would like to thank Sitan Chen, Mimi Gallagher, Sara Sanchez, Dino Feng and Andrew Chen for enlightening discussions; Wei-Po Wang, Zhao Song, Jui-Hui Chung, David Liu, Yibo Wen, Maojiang Su, and Hude Liu for collaborations on related topics; and the Red Maple Family for support. The authors would like to thank the anonymous reviewers and program chairs for constructive comments. They also thank Maojiang Su, Ning Zhu, Mingcheng Lu, and Sophia Pi for pointing out typos.
JH is partially supported by the Walter P. Murphy Fellowship. HL is partially supported by NIH R0ILM1372201, Abb Vie and Dolby. This research was supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at North- western University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
REFERENCES
Silas Alberti, Niclas Demn Laura Thesing, and Gitta Kutyniok. Sumformer: Universal approximation for efficient transformers. In Topological, Algebraic and Geometric Learning Workshops 2023,
pages 12-86. PMLR, 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669-22679, 2023.
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13600 2021.
Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d-linear con- vergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations, 2024.
Clément L Canonne. A short note on an inequality between kl and tv. arXiv preprint arXiv:2202.07198, 2022.
Bo Chen, Chengyue Gong, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Mingda Wan. High-order matching for one-step shortcut diffusion models. arXiv preprini arXiv:2502.00668, 2025.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang, PLOT: Prompt learning with optimal transport for Janiunnl models. In The Eleventh International Conference on Leurning Representations (ICLR), 2023a.
Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735-4763. PMLR, 2023b.
Minshuo Chen, Kaixuan Huang. Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on -iwdiieesioonl data. In International Conference on Machine Learning, pages 4672-4712. PMLR, 2023c.
Minshuo Chen, Song Mci, Jianqing Fan, and Mengdi Wang. An overview of diffusion models: Ap- plications, guided generation, statistical rates and optimization. arXiv preprint 244 2024a.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint 12Xiv:2209.112/5. 2022.
Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode is provably fast. Advances in Neural Information Processing Systems, 36, 2024b.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
10

---

Published as a conference paper at ICLR 2025
Anh-Dung Dinh, Daochang Liu, and Chang Xu. Rethinking conditional diffusion sampling with progressive guidance. Advances in Neural Information Processing Systems, 36, 2023.
Zehao Dou, Minshuo Chen, Mengdi Wang, and Zhuoran Yang. Theory of consistency diffusion models: Distribution estimation meets fast sampling. In Forty-first International Conference on Machine Learning, 2024a.
Zehao Dou, Subhodh Kotekal, Zhehao Xu, and Harrison H Zhou. From optimal score matching to optimal sampling. arXiv preprint 1.Xiv:2409.0703222 2024b.
Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning (ICML), pages 5793-5831. PMLR, 2022.
Hengyu Fu, Zehao Dou, Jiawei Guo, Mengdi Wang, and Minshuo Chen. Diffusion transformer captures spatial-temporal dependencies: A theory for gaussian process data. arXiv preprint arXiv:2407.16134, 2024a.
Hengyu Fu, Zhuoran Yang, Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory. arXiv preprint arXiv:2403.11965 2024b.
Khashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv preprint arXhv:2404.18869 2024.
Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, and Zhao Song. Exploring the frontiers of softmax: Provable optimization, applications in diffusion model, and beyond. arXiv preprint arXiv:2405.03251, 2024.
Hanzhong Guo, Cheng Lu, Fan Bao, Tianyu Pang, Shuicheng Yan, Chao Du, and Chongxuan Li. Gaussian mixture solvers for diffusion models. Advances Neural Information Processing Systems (NeurIPS), 37, 2023.
Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusion models: An optimization perspective. arXiv preprint arXiv:2404.14743, 2024.
Alexander Havrilla and Wenjing Liao. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Satoshi Hayakawa and Taiji Suzuki. On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces. Neural Networks, 123:343-361, March 2020. ISSN 0893-6080, doi: 10. 10.1006...euuet2200 12.011.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Jerry Yao-Chieh Hu, Weimin Wu, Zhuoru Li, Sophia Pi, Zhao Song, and Han Liu. On statistical rates and provably efficient criteria of latent diffusion transformers (dits). In Thirty-eighth Conference on Neural Information Processing Systems (NeurIPS), 2024.
Jerry Yao-Chich Hu, Wei-Po Wang, Ammar Gilani, Chenyang Li, Zhao Song, and Han Liu. Fun- damental limits of prompt tuning transformers: Universality, capacity and efficiency. In The Thirteenth International Conference Learning Representations, 2025.
Yuling Jiao, Lican Kang, Huazhen Lin, Jin Liu, and Heng 7uo. Latent schr {\" o) dinger bridge diffusion model for generative learning- arXiv preprint arXiv:2404. 13309, 2024a.
Yuling Jiao, Yanming Lai, Yang Wang, and Bokai Yan. Convergence analysis of flow matching in latent space with transformers. arXiv preprint arXiv:2404.02538, 2024b.
Tokio Kajitsuka and Issei Sato. Are transformers with layer self-attention using low-rank weight matrices universal approximators? In The Twelfth International Conference on Learning Representations (ICLR), 2024.
11

---

Published as a conference paper at ICLR 2025
Junghwan Kim, Michelle Kim, and Barzan Mozafari. Provable memorization capacity of transformers. In The Eleventh International Conference on Learning Representations, 2022.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22882.
2022.
Holden Lee, Jianfeng Lu, and Yixin Tan, Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pages 946-985. PMLR, 2023.
Gen Li, Yu Huang, Timofey Efimov, Yuting Wei, Yuejie Chi, and Yuxin Chen. Accelerating convergence of score-based diffusion models, provably, arXiv preprint arXiv:2403.03852, 2024a.
Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards non-asymptotic convergence for diffusion- based generative models. In The Twelfth International Conference on Learning Representations, 2024b.
Gen Li, Yuting Wei, Yuejie Chi, and Yuxin Chen. A sharp convergence theory for the probability flow odes of diffusion models. arXiv preprint arXiv:2408.02322 2024c.
Yingyu Liang. Zhenmei Shi, Zhao Song, and Yufa Zhou. Unraveling the smoothness properties of diffusion models: A gaussian mixture perspective. arXiv preprint 240. 2024a.
Yuchen Liang, Peizhong Ju, Yingbin Liang, and Ness Shroff. Non-asymptotic convergence of discrete- time diffusion models: New approach and improved rate. arXiv preprint arXiv:2402.13900 2024b.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, al. Sora: A review background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177 2024.
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 18444-18455 2023.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning, pages 26517-26582. PMLR, 2023.
Sejun Park, Jaeho Lee, Chulhee Yun, and Jinwoo Shin. Provable memorization via deep neural networks using sub-linear parameters. In Conference on Learning Theory (COLT), pages 3627- 3661. PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), pages 4195 4205, 2023.
Peter Potaptchik, Iskander Azangulov, and George Deligiannidis. Linear convergence of diffusion models under the manifold hypothesis. arXiv preprint arXiv:2410. 099466 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan. Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 35:36479-36494 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pages 2256-2266 PMLR, 2015.
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2025
Rong Tang, Lizhen Lin, and Yun Yang. Conditional diffusion models Mannmmax-optimge and manifold-adaptive for conditional distribution estimation. arXiv preprint arXiv:2409.20124, 2024.
Wenpin Tang and Hanyang Zhao. Score-based diffusion models via stochastic differential equations-a technical tutorial. arXiv preprint urXiv:2402.074887 2024.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in neural information processing systems, 14:11287-11300 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 23(7):1611-1674, 2011.
Vikram Voleti, Alexia Jolicoeur-Martinen and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural information processing systems, 05:23371-23385 2022.
Junde Wu, Wei Ji, Huazhu Fu, Min Xu, Yueming Jin, and Yanwu Xu. Medsegdiff-v2: Diffusion- medical image segmentation with transformer. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6030-6038, 2024a.
Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical insights for diffusion guidance: A case study for gaussian mixture models. In Forty-first International Conference Machine Learning, 2024b.
Yuchen Wu, Yuxin Chen, and Yuting Wei. Stochastic runge-kutta methods: Provable acceleration of diffusion models. arXiv preprint arXiv:2410.04760, 2024c.
Yuhong Yang and Andrew Barron. Information-theorete determination of minimax rates of conver- gence. Annals of Statistics, pages 1564-1599, 1999.
Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stefano Ermon. Tfg: Unified training-fres guidance for diffusion models. arXiv preprint arXiv:2409.1570 15761, 2024.
Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang, Reward- conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36, 2024.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are trans- functions? In International Conference
formers universal approximators of Learning Representations (ICLR), 2020.
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. arXiv preprinl 4rX1v:2410.03456, 2024.
13

---

Published as a conference paper at ICLR 2025
Appendix
A Latent Conditional DiTs
15
Notation Table
17
Related Works, Broader Impact and Limitations
18
C.I Related Works
18
C.2 Broader Impact
19
C,3 Limitations
19
D Latent Conditional DiT with Hölder Assumption
20
D.1
Score Approximation
D.2
Score Estimation
23
D.3
Distribution Estimation
24
Latent Conditional DiT with Lipschitz Assumption
25
E.
Score Approximation
26
E.2
Score Estimation
................................
E.3
Distribution Estimation
27
E,4
Proof of Score Approximation (Theorem E.1)
E.5
Proof of Score Estimation (Theorem E.2)
34
E.6
Proof of Distribution Estimation (Theorem E.3)
F Supplementary Theoretical Background
43
F.1
Conditional Diffusion Process Classifier-free Guidance
43
F22
44
G Universal Approximation of Transformers
45
G.I Transformers as Universal Approximators Parameter Norm Bounds for Transformer Approximation
45
G.2
................................
52
H Proof of Theorem 3.1
56
H.1 Auxiliary Lemmas
................................
H.2 Score Approximation on a Bounded Domain
................................ ................................
57
H.3
Main Proof of Theorem 3.1
I Proof of Theorem 3.2
65
I.1
Auxiliary Lemmas
65
I.2
Main Proof of Theorem 3.2
69
J
Proof of the Estimation Results for Conditional DiTs
71
J.1
Auxiliary Lemmas for Theorem 3.3
71
J.2
Proof of Theorem 3.3
78
J.3
Auxiliary Lemmas for Theorem 3.4
82
J.4
Main Proof of Theorem 3.4
82
J.5
Proof of Theorem 3.5
85
14

---

Published as a conference paper at ICLR 2025
A
LATENT CONDITIONAL DrTs
Reshupe
-1/0?
Revensd Reshipe Layer
Wi
RD
27 Thaz
Incodar
2
"W
Embed
Trancfonner Netwark
Latent Decoder
Labell y
Timesspp
Figure 2: Network Architecture of Latent Conditional DIT. overall architecture consists linear layer of encoder and decoder WT and Wo that transform input E B& into linear latent space Re reshaping layer R(-) and R (-), embedding layer for label and timestep t. The embedding concatenates with input sequences and processes by the adapted transformer network Th."." Ri- o gr o o R.
In this section, we extend the results from Section by considering the latent conditional DiTs. Specifically, we assume the raw input Rd has an intrinsic lower-dimensional representation.
Assumption A.1 Low--Dimeniional Linear Latent Space). Initial data x has a latent representation via Uh, where U E Rd is unknown matrix with orthonormal columns. The latent variable IRdo follows distribution Pa with density function Ph.
Remark A.1. "Linear Latent Space" means that each entry of given latent vector is a linear combination of the corresponding input, i.e., x = Uh. This is also known as the "low-dimensional data" assumption in literature (Hu et al., 2024; Chen et al., 2023c). This assumption is fundamental in dimensionality reduction techniques for capturing the intrinsic dower-dimensional structure of data.
Score Decomposition and Model Architecture. To derive approximation and estimation results, we extend the techniques and network architecture presented in Section to latent diffusion by considering the dimeeeminiioal linear subpace". Under Assumption A.I, we decompose the score:
V log = U( of logr (UT alv) + +UT
1/o
4/ residual connection
(A.1)
=0 (UTTT..9.)) Rdo xR 16,21
Rdo
following Hu et al. (2024); Chen al. (2023c) (see Lemma D.1). Based this decomposition, we construct the model architccture in Figure 2. The network detail for approximate (A.1) are as follow: a transformer 9T (WJ H, v, E Thaarr to approximate q(UT x, y, l), a latent encoder WT E R.doxdd xds and decoder Wy E Rd to approximate UT E doo and E RRd, xdo and residual connection to approximate -z/a?. Importantly, do is the latent dimension.
For latent diffusion, we follow the standard setting by Peebles and Xie (2023). For each input E Rd, and corresponding label y E RV we use transformer network to obtain score estimator sw E Rd The key differences from Section 3 are follows: First, apply latent encoder WT do to map the raw data æ E gd. into low-dimensional representation h == W. x E Rdo where do S dz- Second, we reshape E dRdo into a sequence H E T using a layer R(-)) Rdd + R with do d L. Note that, by do d... d < d, and L L. Third, pass H R through the transformer 9T. Lastly, We then obtain the predicted score sw E Rd by applying the inverse reshape layer R ( Rx + Ro followed by the latent decoder Wu Ro + Rdd.
For our analysis, we study the cases under both the generic and strong Hölder smoothness assumptions on latent representation 2 do Specifically, we assume the "latent" data is Sy Hy-Hölder smooth with radius B, following Assumptions 3.1 and 3.2. We extend both approximation and estimation results from Section 3 to latent diffusion and establish the minimax optimality of latent conditional DiTs.
Score Approximation. We now present the approximation rates for latent score function under both generic and stronger Hölder data assumptions. Let h == WJ x E do and h == UTa E Bd be the estimated and ground truth (according to Assumption A. latent representations, respectively.
15

---

Published as a conference paper at ICLR 2025
Theorem A.1 (Score Approximation of Latent Conditional DiT', Informal Version of Theorems D.1
and D.2). Assume do = 9 log log N For any precision 0 < c and smoothness ,ó0 > 0, let
log N
e S ONN 8o) for some N N. Let Ca, Co > 0 be some absolute constant. For any y E 0, 1 1] and E [N-C., Ca log N,, there exists Tecore (a, y, () E Thhs. such that
Under Assumption 3.1, have
(
(Bi.
B N 90 (log Ny + +
240 ||soooo (h. y, t) V logg (Tlv) pl (ly)dd = =0
Notably, for OON the approximation error has the upper bound O(loog (+) /ot).
Under Assumption 3.2, holds
Tecoro (), t) V Flogp p (hlg)dh =0 o 0
( oi
N 280. (log N N)
R/00
Notably, for -O(NN-0) the approximation error has the upper bound (E
Proof. See Theorems D.1 and D.2 for the formal versions and Appendices and for proofs.
Remark A.2 (Comparing with Theorems 3.1 and 3.2). Recall d, > dg- The approximation error bounds O(log ) /at) in Theorem 3.1 and ((f2 in Theorem 3.2. Theorem A. shows that the latent conditional DiT achieves better approximation and has the potential to bypass the challenges associated with the high dimensionality of initial data.
Score and Distribution Estimation. Based on Theorem A.I, derive the score estimation bounds in Theorem D.3, and report the results for distribution estimation in next theorem.
Theorem A.2 (Conditional Distribution Estimation). For E [0, 1]d,, let (11)) denote estimated conditional distributions to- Recall that Po(-ly) is the conditional distribution of initial data zo given y and I E [N-Co
C. Ca log N]. Assume KL (Po(-ly) N /)) s e for some constant e < co.
Assume do I2( 12,0 + 2. it holds
Ing N log log N and Assumption 3.1. Let ii = 68,30 104C, and D2 12do +
(logm) By TV (o)) o (n x+))
Po(1b))]]
E E
where w min 2(2 +386) Vy + Co +33 21 +3310 J.
80
Assume Assumption 3.2. Let Bi =
- (12 + 31 6,3) 3+ + 12C, (12dg + 25d + 6) + 72C,.
For all x Rd= holds
B Ey TV (Po(-l))) (o(o)) o (n (r=O(log max Not3 12211) 1)
where Φ min
C. 2pP++0p.J
C.
%
By
Proof. Please see Appendix J.4 for a detailed proof.
Remark A.3 (Minimax Optimal Estimation). Following the same idea in Section 3.4, show that the estimation error bound in Theorem A.2 is the optimal tight bound for the latent unconditional DiT. Specifically, by applying Lemma 3.1 and substituting p(zly) and d, by p (Flyy and dp respectively in Assumption 3.2, we follow the setting specified in Appendix J.5.
16

---

Published as a conference paper at ICLR 2025
Concluding Remarks. We defer the discussion of results and concluding remarks to Section 4. We extend our analysis to the setting of (Hu et al., 2024) and improve their results in Appendix Importantly, our bounds avoid the gigantic 2(1/e)²' term reported by Hu et al. (2024).
B NOTATION TABLE
We summarize our notations in the following table for easy reference.
Table Mathematical Notations and Symbols
Symbol
Description
[/]
The index (1.... (}, where / E N+
a
The i-th component of vector The (i, entry of matrix A Euclidean norm of vector r
A
[
I-norm of vector æ 2-norm of vector z
Infinite norm of vector æ Spectral norm matrix W Frobenius norm matrix W (p. g) gommr matrix W, where p-norm over columns and q-norm over rows
W
WII.
lf
1.2 norm, where f is a function
(())))
1.2 (P) -norm, where is a function and is distribution
Lipschitz-norm, where
function
dp(f.9)
p-norn of the diffcrence between functions and g defined dp(f.9) = (f [f(x) p(x)p dz)) dz) Pushforward measure, where f function and P distribution Kullbbakk-eiiblee (KL) divergence between distributions P and Total variation (TV) distance between distributions P and Q Normal distribution with y and variance o² There exist constants C 0 such that Cb
f.P
QL(P,Q)
TV(P,Q)
N(p.
Sample
Data point original data space, x R'.. Conditioning Label, x E Rd,
Latent variable in low-dimensional subspace, h e Bd
R
h= U
Pk
The density function of i The matrix with orthonormal columns to transform to where U e (d
Radius Hölder for conditional density function p(Kly)
Bo
Radius of Hölder ball for latent conditional density function p(Pilx)
Hölder index for conditional density function p(z)g))
Hölder index for latent conditional density function p(A y)
D
Granularity in the construction of the transformer universal approximation
N
Resolution of the discretization the input domain
R
Score risk (expectation of squared 1 difference between score estimator and ground truth)
(f,,, IHI)
Covering number of collection F (sec Definition J.5)
T
Stopping time in the forward process of diffusion model Stopping time in the backward process of diffusion model
to
Discretized step size in backward process
The density function a at time The density function of i at time t (Conditional) Gaussian density function
Thas
Transformer network function class Definition 2.2)
Transformer block h-head, s-hidden size, r-MLP dimension (see Definition 2.1) Input dimension each token the transformer network Token length
transformer network DiT
I Latent data input dimension of each token in the transformer network of DIT Latent data token length in transformer network of DiT
L
Sequence input transformer network in DíT, where Rdx1
H
Sequence latent data input of transformer network DiT, where X g gixxz. Position encoding, where E E L
E
R(-
Reshape layer in DiTs, R(-) RR4 R.xr.
R()
Reshape layer latent DīTs. R(-) Rddo R
R (1)))
Reverse reshape layer DiTs, R-1())) RdxL - Rd
R-
Reverse reshape layer latent DiTs, R (-) Rxx Rd
Wb.
orthonormal matrix to approximate U. where Wir E B4. xdo
17

---

Published as a conference paper at ICLR 2025
C RELATED WORKS, BROADER IMPACT AND LIMITATIONS
C.I RELATED WORKS
In the following, we discuss the recent success of the techniques used in our work. We first give the universality (universal approximation) of the transformer. Then, we discuss recent theoretical developments (approximation and estimation) in diffusion generative models.
Universality of Transformers. The universality of transformers refers to their ability to approx- imate any sequenceeto-sequence function with arbitrary precision. Yun et al. (2020) establish this func-
by showing that transformers is capable of universally approximate eqquence-o-ssqueene tions using deep stacks of feed-forward and self-attention layers. Additionally, Alberti et al. (2023) demonstrate universal approximation for architectures employing non-standard attention mechanisms. Recently, Kajitsuka and Sato (2024) show that even a single-layer transformer with self-attention suffices for universal approximation assuming all attention weights rank-1. Moreover, Hu et al. (2024) leverage ( (Yun et al., 2020)'s 's universality results to analyze the approximation and estimation capabilities of DiT.
Our paper is motivated by and builds upon the works of Hu et al. (2024): Kajitsuka and Sato (2024); Yun et al. (2020). Specifically, we utilize and extend the transformer universality result from (Kajitsuka and Sato, 2024). We employ a relaxed contextual mapping property in (Kajitsuka and Sato, 2024) (see Appendix G.1). This generalization allows us to avoid the "double exponential" sample complexity bounds in previous DiT analyses (Hu et al., 2024, Remark 3.4) and establish transformer approximation in the simplest configuration single-layer, single-head attention model.
Approximation and Estimation Theories of Diffusion Models. The theories of DiT's revolve around two main frontiers: score function approximation and statistical estimation (Chen et al., 2024a; Tang and Zhao, 2024). Score function approximation refers to the ability of the score network to approximate the score function. It leverages the universal approximation ability of the neural network in L norms (Hayakawa and Suzuki, 2020), the approximation characterized as Taylor polynomial (Fu et al., 2024a) or B-Spline (Oko et al., 2023). Chen et al. (2023c) and Fu et al. (2024a) investigate score approximation under specific conditions, such as low-dimensional linear subspaces and Hölder smooth data assumptions, using ReLU-based models. Furthermore, Hu al. (2024) presents the first characterization of score approximation in diffusion transformers (DiTs).
The statistical estimation includes score function and distribution estimation (Wu et al., 2024b; Dou et al., 2024a; Guo et al., 2024; Chen et al., 2023c). Under a L22 accurate score estimation, several works provide the convergence bounds under either smoothness assumptions (Benton et al., 2024; Chen et al., 2022) or bounded second-order moment assumptions (Chen et al., 2023b; Lee et al., 2023). Chen et al. (2023c) provide the first complete estimation theory using ReLU networks without precise estimators. Oko et al. (2023) achieve nearly minimax optimal estimation rates for total variation and Wasserstein distances. Meanwhile, Dou et al. (2024b) define exact minimax optimality using kernel functions without characterizing the network architectures. In the realm of diffusion transformers, Hu al. (2024) introduces the first complete estimation theory. Jiao al. (2024a;b) demonstrate theoretical convergence for latent DiTs using ODE-based and Schrödinger bridge diffusion models.
Our paper advances the foundational works of Fu et al. (2024b); Oko et al. (2023); Hu et al. (2024). We adopt the Hölder smooth data distribution assumption', a more practical approach than the bounded support assumption in (Oko et al., 2023). Unlike the simple ReLU networks in (Fu et al., 2024b), we provide a complete approximation and estimation analysis for conditional DiTs and establish their exact minimax optimality. Furthermore, while Hu et al. (2024) analyze DiTs, their estimation upper bounds are suboptimal. We refine this by avoiding the substantial double exponential term 2(1/=)²* reported by Hu et al. (2024, Remark 3.4 3.4) and present a provably tight, minimax optimal estimation. Recent works on minimax optimality of diffusion model such as (Tang et al., 2024; Fu
'Of independent interest, many works investigate the convergence rates of diffusion models under various score and data smoothness assumptions or with different samplers. Please (Chen al., 2025; Li et al.. 2024a;hçe; Potaptchik al., 2024; Wu al., 2024c; Liang al., 2024b;a; Gatmiry al., 2024; Gu et al., 2024; Guo et al., 2023; Chen et al., 2024b; 2023b; 2022; Lee et al., 2023; 2022) and references therein.
'Recent work by Havrilla and Liao (2024) examines the generalization and approximation of transformers under Hölder smoothness and low-dimensional subspace assumptions.
18

---

Published as a conference paper at ICLR 2025
al., 2024b) demonstrate minimax-optime optimal under the total variation for ReLU network. Additionally, Tang et al. (2024) extend the analysis to a more general setting by considering different smoothness levels for the data z and condition y.
C.2 BROADER IMPACT
This theoretical work aims to shed light the foundations of generative diffusion models and is not expected to have negative social impacts.
C.3 LIMITATIONS
Although our study provides a complete theoretical analysis of the conditional DiTs and establishes the minimax optimality of the unconditional DiT, acknowledge three main limitations:
The minimax optimality of conditional DiT remains not clear.
We did explore other architectures such "adaptive layer norm" and "cross-attention" DiT. A potential direction is by establishing the universal approximation capacity of the transformer with cross-attention mechanisms.
Although achieve a better bound for the latent conditional DiT under the Lipschitz assumption than under the Hölder assumption, we do not show the minimax optimality under the Lipschitz assumption.
We leave these for future work.
Furthermore, there are limitations regarding the Hölder smooth data assumptions in Assumption 3. and Assumption 3.2. Our results in Section 3 and Appendix A depend on the Hölder smooth data assumptions. However, it is challenging to measure the smoothness of given dataset (e.g., CIFAR10), because it requires knowledge of the dataset's exact distribution. Conversely, it feasible create dataset with predefined level of smoothness. To illustrate this, we provide two examples.
Diffusion Models in Image Generation: When modeling conditional distributions of images given attributes (e.g., generating images based class labels), these assumptions hold if the data distribution around these attributes is smooth and decays. In diffusion-basel generative models, the data distribution often decays smoothly in high-dimenniioal space. The assumption that the density function decays exponentially reflects the natural behavior of image data, where pixels or features far from central region or manifold less likely. This is commonly observed images with blank boundaries.
Physical Systems with Gaussian-Like Decay: This applies to cases where the spatial distribution of a physical quantity, such temperature, is smooth and governed by diffusion equations with exponential decay. In physics-based diffuusion models, like those simulating the spread of particles or heat medium (e.g., stars in galaxies for astrophysics applications), the conditional density typically decays exponentially with distance from a central region.
19

---

Published as a conference paper at ICLR 2025
D LATENT CONDITIONAL DIT WITH HÖLDER ASSUMPTION
In this section, we elaborate more on Appendix A conditional DiTs under (i) low-dimensional linear latent (subspace) and (ii) Hölder data assumptions.
We extend the results on approximation and estimation of DiT from Section 3 by considering the latent conditional DiTs. Latent DiTs enables efficient data generation from latent space and therefore scales better in terms of spatial dimensionality (Rombach et al., 2022). Specifically, representation in a dy-dimensional subspace,
assume the
raw input & E I has an intrinsic lower-dimensionss where do dg. This setting is common in both empirical (Peebles and Xie, 2023; Rombach et al., 2022) and theoretical studies (Hu et al., 2024; Chen et al., 2023c).
Organization. We present the statistical results under Hölder data smooth Assumptions 3.1 and 3.2 and state the results in Theorem D.I, Theorem D.2, Theorem D.3, and Theorem D.4, respectively. Ap- pendix D.1 discusses score approximation. Appendix D.2 discusses score estimation. Appendix D.3 discusses distribution estimation. The proofs in this section primarily follow Appendices and I.
Let do denote the latent dimension. We summarize the key points of this section as follows:
K1. Low-Dimensions. Subspace Space Data Assumption. We consider the setting that latent representation lives in a "Low-Dimensional Subspace" under Assumption A.1, following (Hu et al., 2024; Chen et al., 2023c).
Assumption D.1 point x = Uh, where U E Rddxxd is an unknown matrix with orthonormal columns. The latent variable h de follows distribution Pn. with density function Ph.
Linear Latent Space (Assumption A. Restated)). Data
For raw data E Rdo we utilize linear encoder WT E gdoxda and decoder Wy E Idd to convert the raw E Rdd and latent h E Rddo data representations. Importantly, Uh with U Rd,xdo by Assumption A.
For each input x E Rdd and corresponding label y E Rdd, we use a transformer network to obtain score estimator sw E RRd To utilize the transformer network the estimator, introduce reshape layer to convert vector input E Rd to matrix (sequence) input H I Specifically, the reshape layer in the network Figure 2 is defined as R(-): Ie + R and its reverse R (-) - - Rdo, where do S dg, d < d. and L < L.
We remark that the fow-ddmmnsioonl data" assumption leads to tighter approximation rates than those of Sections 3.1 and 3.2 and estimation errors due to do < dg (Theorems D.1 and D.2).
K2. Hölder Smooth Assumption. For approximation and estimation results for latent conditional DiTs (Theorems D.I to D.4), we study the cases under both the generie and strong Hölder smoothness assumptions on latent representation h E IRd Specifically, we assume the "latent" data is (6- Hölder smooth with radius Bg following Assumptions 3.1 and 3.2. We extend both approximation and estimation results from Section 3 to latent diffusion and establish the minimax optimality of latent conditional DiTs.
Assumption D.2 (Generic Hölder Smooth Data (Assumption 3.1 Restated)). The conditional density function p (holy) defined on the domain R:do [0, 1]d, and belongs to Hölder ball of radius Bg > 0 for Hölder index So > O, denoted by p è (hol yy) E H x [0. 1|d, Bu) (see
Definition 3.1 for precise definition.) Also, for any y E [0, 1jdx, there exist positive constants
C1, C2 such that p (holy) S C1 exp (-C² //oll//2).
Assumption D.3 (Stronger Hölder Smooth Data (Assumption 3.2 Restated)). Let function f H (R [0, 11 Bo). Given a constant radius Bo. positive constants C and Cz, we assume the conditional density function p(holz) exp exp(-Ca) (2c(-C2llholl/// f(ho, v) and f(ha, y) 2 C for all (ho. y) dR 10
20

---

Published as a conference paper at ICLR 2025
K3. Latent Score Network. Under low-dimensional data assumption, decompose the score function following (Hu et al., 2024; Chen et al., 2023c) (see Lemma D.1):
V Pogg)(<<l)) U log (U xlu) + uT))o
2/O
(D.1)
= R [f..]]
residual connection
Based on this decomposition, we construct the model architecture in Figure 2. The network detail for approximate (D.1) are as follow: a transformer 9T ( x, y, E Thh,r to approximate q(U 3, y. (). a latent encoder W E IRdxxd= and decoder Wt E IRdx xds to approximate U E xdz and U E xda, and a residual connection to approximate -aptaateeer////// -x/o?.
We adopt the following transformer network class of one-layer single-head self-attention
Th.ar { sw (2,8, 1) = WUST (W),,,
1 of
(D.2)
residual connection
where 97 E This. (1-5 о fhre Bd + E.
Let h = WJ. E Rdo and h = U Rdd be the estimated and ground truth (according to Assumption A.1) latent representations, respectively. Here we construct a network sw (z, y, t) to approximate the score function in (D.1) (see Figure 2 for network illustration).
In Section give approximation results for score function V log (=lly) using conditional DiTs with one-layer single-head self-attention transformer. We use the similar transformer architecture to approximate latent score function V log p. (Tlly). Specifically, the density function takes the form
p (hly) = the (hehhh)px (Mly)dh,
where (-1h) is Gaussian ((Sth,ofTa) with Br e and of
Based on the latent network construction in (K3), we employ the same techniques presented in Section 3 for score function approximation and estimation. We restate for completeness. First, we decompose the conditional score function V log pt (hly) as following:
Vpt (h
V log (Fly)
(D.3)
ri (Mly)
By the definition of Gaussian kernel, we have
|| To
p (hly) -f (2To?)) de/2
Ph (hly)
exp
dh.
20
seder Taylor polyuomial
oodde
Reke-order Taylor polynomial
Similar to Section 3, our strategy is to expand above term-by-term with ki- and k2-order Taylor polynomials for fine-grained characterizations.
Remark D.I. Here in the latent density function, we have (2ma²) -da/2 instead of (2ma?) -do/22
However, the additional (2ma²)--dds (dd--d//// term does not affect the application Section into latent diffusion approximation.
Based on the low-dimensional data structure assumption, we have the following score decomposition terms: on-support score (U') x, y, f) and orthogonal score s sxx y,
21

---

Published as a conference paper at ICLR 2025
Lemma D.1 (Score Decomposition, Lemma I of (Chen et al., 2023c)). Let data x: = Uh follow Assumption A.1. The decomposition of score function V log (2( is
V (o)))(()) = UV log (hly) (ID UU x/o = --
(D.4)
1 (h.g.0)
where p (hly) vir (x(h)))) (11v) dh, (-) is the Gaussian density function of N N of la.). 3 = and of -
Following the proof strategy of conditional DīTs in Appendices and with differences highlighted in (KI), (K2), and the latent network (K3). To derive the approximation and estimation under generic and stronger Hölder assumptions results in Theorems 3.1 to 3.4 for data under low-dimensional data assumption, we just need to replace the input dimension d, L to d and L, and the input dimension d. with dg, and consider the 3 Hölder smoothness assumption on latent data.
To hegin, we clarify the relation between initial data admits to p(zly) E H x [0, 1j.1,,,, B), and under linear transformed data Assumption A.1 admits to p(hly) E Ho H x [o, 1] Bo) where 8o 3 and Bo < CB by Lemma D.2.
Lemma D.2 (Transformation
Stronger Hölder Smooth Data Distribution under Linear Mapping). Let f E H H (R2z x [0, 1]d, B) satisfy ((x,y) 2 C > O for all (x, v) E Rd x [0, 1] Consider the conditional density function:
C2 || ||=12 2
p(aly) = f(x, y) exp
(-2
Suppose the data undergo the linear transformation Uh, where U E R.da has orthonormal columns (U U
(U = Iá,) and fo fh(lly) (Uhlyy). The transformed density p(hly) becomes:
p(bly) = f(Uh, exp (-Ca 1
The following condition holds for Hölder smooth data that undergoes linear transformation: fo a (Rd. x 104, Bo) with Bo < CB, where Č = max(C", C""..
Proof. First, we compute the partial derivative of the transformed function fo(hly) f(Uhly). From the definition of Hölder space Definition 3.1, and let (a), oyy where + ay < k1. We compute the partial derivative up to the order of h; and show that it is bounded by some C", that is
((Uh,8) v) exp (- (-2 (1113)
80 p(hp))))))))))))))))))))) 8hhn yos
( (ofa-e) exp (- (-2 2 |M13 )
Σ asv
(0h ((Uh. v))
(By product rule)
From the relation 0r f(Uh,y) = U 0 f(U,, b)) where U the product of U entries corre-
spond to di). Therefore, 18 0 (hlp)) S C' for some C' depends on U and or Since satisfied Hölder condition and the mapping + Uh linear, for Hölder condition |a, + |a, kn there exist C such that
10th 0f fo(hly) 0. o5 (1(ll))) < B.
[1 v) (W', v
The bounded partial derivate up order K1 satisfied Hölder condition. This completes the proof.
22

---

Published as a conference paper at ICLR 2025
D.1 SCORE APPROXIMATION
We present the approximation rate of latent score function under generic Hölder and stronger Hölder data assumption in Theorems D.I and D.2, respectively.
Theorem D.1 (Latent Conditional DiT Score Approximation, Formal Version of Theorem A. l). Ing
Assume Assumption 3.1 and dg = O log log N For any precision 0 < < and smoothness A let (NN-9a) for some N N. Let Co C. > 0 be some absolute constant. For any y E [0, 1]dy and E [N- [N-C=, C. log N). there exists T Tscore (F, (r, TRTR.. such that
T (h, v) - Vlog Vlogg/ p. (hly)d Rd0
Bg of N
(log N) do 3q 9 N)do+
Proof Sketch. The proof closely follows Theorem 3.1, with differences highlighted in (K1) and (K2).
Specifically, we replace L and d, with d. L and dg respectively. Moreover, we consider the Ho-Hölder smoothness assumption latent data detailed in (K2). This completes the proof.
Please Appendix H for detailed proof.
Theorem D.2 (Latent Conditional DiT Score Approximation under Stronger Hölder Assumption, Formal Version Theorem A. Assume Assumption 3.2. For any precision 0 < c < and smoothness Bg 0, let < ONN -aa) for some N N. Let Ca, Co > 0 be some absolute constant. For any y E [0, 1] and & [N-C. C. Ca log N), there exists a Tacore (I, y. 1) E Thh...r such that
f Tscore (x. (h,
v log p (Fly)) (hlydd =
B
2 (log N) N))itt). 0z
Proof Sketch. The proof closely follows Theorem I.1, with differences highlighted in (K1) and (K2). We replace the input dimension d, L d and L, and the input dimension dz with dg in Theorem I.1 with the -ölder smoothness assumption on latent data detailed in (K2). This completes the proof.
Please Appendix for detailed proof.
Remark D.2 (Score Approximation for Low- Now-Dimensional Linear Latent Space). With the assumption of low-dimensionnl latent space Assumption 1, Theorems D. and D.2 provide better approximation rates than Theorems 3.1 and 3.2 under Hölder smooth assumptions
Assumptions 3.1 and 3.2, respectively. Specifically, from Lemma D.2 have 3g and Bo S B. Therefore, Theorems D.1 and D.2 deliver better approximation error over Theorem 3.1, where do S dz.
D.2 ScORE ESTIMATION
In this section, we provide the extended results for Section 3.3 on score estimation with the estimator Teeme We state the main results under Hölder data assumptions in Theorem D.3.
Theorem D.3 (Conditional Score Estimation with Transformer). Consider y E [0, 1]4y and E [lo, T] with to = NN C. and = Ca log N, where C., Ca are absolute constants such that to <
Assume Assumption 3.1. Then, it holds
2o. 3T 1336 13 (log n) )0,+2)
E (x,an), [R(3)] = [
Assume Assumption 3.2. Then, it holds
E Ez [R()) = (n
2.g F.+0o (log n) ( max (1,)8o+2))
23

---

Published as a conference paper at ICLR 2025
Proof Sketch. The proof closely follows Theorem 3.3, with differences highlighted in (K1) and (K2). By replacing the imput dimension d, L d and L, and the input dimension de with do in Theorem 3.3, and under the the Ho-Hölder smoothness assumption on latent data detailed in (K2), the proof is complete. Please see Appendix J.2 for detailed proof.
Remark D.3 (Comparing Score Estimation in Theorems 3.3 and D.3). Invoking Lemma D.2 where 80 = 3 and Bo S B the sample complexity in Theorem D.3 improves Theorem 3.3.
D.3 DISTRIBUTION ESTIMATION
In this section, we provide the extended results for Section 3.3 on distribution estimation with the estimator Tewore. We restate the main results under Hölder data assumptions in Theorem D.3.
Theorem D.4 (Conditional Distribution Estimation, Theorem A.2 Restated), For y E [0, 1]dx, let Pro (1y) denote estimated conditional distributions at l-- Recall that Po(-ly) is the conditional distribution of initial data zo given y and [N-C=, Ca log NJ. Assume KL (Po(-l) N( Z)) s for some constant c < 00.
Assume Assumption 3.1. Then, it holds
E
fa Him
By TV
(-1)), Po(lwi)]] (Hb), o (logn) 7+3)
where T min
C₁ 20. +336)
Co +C+ +33a
80 2 20
Assume Assumption 3.2. Then, it holds
By [TV (Po((
= =0 (n-g(oonn) x+3)
E [z )
where Ø- min
Co +636)
C.
8u
+636
in +696
Proof. The proof closely follows Theorem 3.4, with differences highlighted in (K1) and (K2). By replacing the input dimension d, L to and L, and the input dimension d,, with do in Theorem 3.4, and under the the 3-Höller smoothness assumption on latent data detailed in (K2), the proof complete. Please see Appendix J.4 for detailed proof.
Next, we present the distribution estimation result for low-dimensional input data.
24

---

Published as a conference paper at ICLR 2025
E LATENT CONDITIONAL DIT WITH LIPSCHITZ ASSUMPTION
In this section, we apply our techniques to the setting of (Hu et al., 2024) on DiT approximation and estimation theory under (i) low-dimensional linear latent (subspace), (ii) Lipschitz score and (iii) sub-Gaussian data assumptions. Specifically, we extend their work by using the one-layer self-attention transformer universal approximation framework introduced in Appendix G.I.
Compared to (Hu et al., 2024), we consider classifier-fret conditional DiTs, providing a holistic view of the theoretical guarantees under various assumptions. In particular, our sample complexity bounds avoid the gigantic double exponential term 2(1/e)** reported in (Hu et al., 2024). We adopt the following three assumptions considered by Hu et al. (2024):
(A1) now-Dimensional Linear Latent Space Data Assumption.
Assumption E.1 (Low-Dimensional Linear I Latent Space, Assumption A.1 Restated). Data point æ = Uh, where U E IR x is an unknown matrix with orthonormal columns. The latent variable h E de follows a distribution Ps. with density function Ph-
Under this data assumption, Chen et al. (2023a) show that the latent score function endows neat decomposition into on-support s+ and orthogonal s terms (see Lemma D.1).
Lemma E.1 (Score Decomposition, Lemma of (Chen al., 2023c), Lemma D. Restated), Let data x Uh follow Assumption A.I. The decomposition of score function V log Pe is
(o) UV log (hly) (LD UUT) x/0 h U (x,1)
(E.1)
(h.21.)
where pt (Tly) = f th (h(h)) (hly) dh, (-1b)) is the Gaussian density function of N(Bhh la.)), Br e-22 and of -t
(A2) Lipschitz Score Assumption. We assume the on-support score function 8 (h, y, to be Le, Lipschiizz for any h and y.
Assumption F.2 (I,, Lipschitz of 8+ (h, y, t)). The on-support score function 8+ (h, y, t) is Luy -Lipschiz with respect to any Ti IRdo and y E Rd for any E [o, T.. i.e., there exist constant L,g, such that for any h y and h y/:
|s |s y. s+ (Th, ,1)l2 S L._ Ft' 112 + Lo 11y 8 1122
(A3) Light Tail Data Assumption.
Assumption E.3 (Tail Behavior of Pa). The density function ps twice continuously differentiable. Moreover, there exist positive constants Ag, A1, A2 such that when ||hll2 2 Ao.
the density function Pı (hly) (2T) d2 - Axexp(- 1/2))
We note that, the assumptions (A1) and (A3) (A2) on the smoothness of score function is stronger than Hölder data smoothness assumptions considered in Section and Appendix A.
data, and (A2) are the score function. Notably,
Organization. We study latent conditional DiTs under low-dimensional data Assumption E, Lipschitz smoothness Assumption E.2, and tail behavior of P,, Assumption E.3 and states the results in Appendices E.1 to E.3, respectively. Appendix E. discusses score approximation, Appendix E.2 discusses score estimation. Appendix E.3 discusses distribution estimation. The proof in this section provided in Appendices E.4 to E.6. The proof strategy in this section follows (Hu et al., 2024).
Here we summarize the key settings of this section:
$1. Lipschitz Smooth Assumption and Tail Behavior. Following (Hu et al., 2024), we introduce assumptions Lipschitz smoothness for on-support score function 8 and tail behavior
25

---

Published as a conference paper at ICLR 2025
of Pa in Assumptions E.2 and E.3, respectively. The on-support score function is defined as $+ (U xx y, UV logp! (U xly) (see Lemma D.1 for score decomposition).
S2. now-Dimensional Space. We consider the setting of latent representation that the data lives in a "Low--Dmeensiona Subspace" under Assumption A.1, following (Hu et al., 2024; Chen et al., 2023c). The raw data x E TR is supported by latent E RRda where do S dz.
S3. Transformer Network. We follow standard setting of "in-context" conditional DiT's by Peebles and Xie (2023) on latent representation. Recall Appendix A for the transformer network setting.
E.1 SCORE APPROXIMATION
For completeness, we introduce the proof' from (Hu et al., 2024) for score approximation of the conditional latent diffusion model.
Instead of assuming Hölder smoothness of the initial conditional data distribution as in Appendix A, we use stricter assumptions on the latent density function. To be specific, we directly approximate the on-support latent score function, instead of approximating the denominator and nominator separately. From the score decomposition in (A.1), we define the on-support score function s+ as following:
(T.ee(())) (hly) dh (h h W)pp (N/ly) dh'
UU v. -U)
UV log | Jully).
(E.2)
Here we require two assumptions following the proof of (Hu et al., 2024) on tail behavior of density function and Lipschitz continuous for on-support score function. Assumption E.3 is the analogy of Assumption 3.1 for assuming the tail behavior of the density function. On the other hand, Assumption E.2 further assume the on-support score function 84 to be L,, -Lipshitz. Note that this assumption is stricter than Assumption 3.1 since we make the Lipschitz assumption directly on the score function instead of on the latent density function.
Theorem E. (Latent Score Approximation of Conditional DiT, modified from Theorem 3.1 in Hu al. (2024)). For any approximation enror >> 0 and any data distribution Po under Assumptions A. E.2 and E.3, there exists DiT score network Tacore (h. y. t) E T that for any & E [fo: 7). we have:
Thh.sir where W (Wu, Tecore } such
T (,,t) V (oggp(()) S /dg + dy/o?.
where of e- and the parameter bounds in the transformer network class satisfy
|Wlll2 = [[xll2 o(d (2+2 (log L)+):
Woll2.0 11,,02 0 c +1 [)
Woll2 = o (a+e.) 1l1oll2,x = 0 ((t
Wwll22 = ( ): |Wvll2,0 = =(d):
[Will2 O (Te - +) || 1,,00 O (dle" 5)
WWell O (de 1 1/ o (abe $)
E 1 O d T.3
Proof. Please see Appendix E.4 for a detailed proof.
26

---

Published as a conference paper at ICLR 2025
E.2 SCORE ESTIMATION
Theorem E.2 (Score Estimation of Latent DiT), Under the Assumptions E.1 to E.3, choose the score network (a, y. T hh.s"" from Theorem E.1 using e E (0, 1) and > 1. With probability - 1/poly(n), we have
to !. [1T)) (oog)())-VVooo/(o// Vlogp(()llecr)ddt= (1 n A log T. T. m)
where hides the factor about d,, dy, do, d and 1.2.-
Proof. Please see Appendix E.5 for detailed proof.
E.3 DISTRIBUTION ESTIMATION
In practice, DiTs generate data using discretized version with step size μ. Let Pro be the distribution generated by scoor (I, y, L) in Theorem E.2. Let Ph and ph; be the distribution and density function of on-support latent variable h to- We have following results for distribution estimation.
Theorem E.3 (Distribution Estimation of DIT, Modified from Theorem 3 of (Chen al., 2023c)). Let T = Olog n)) to = Omin 1/L, , }), where co is the minimum eigenvalue of T.,,, [hh With the estimated DiT score network T (z, y, t) in Theorem E.2, we have the following with probability 1 1/poly(n).
(i) The accuracy to recover the subspace U is
[|WowT UUT 2 Ie
(6(2 c0
5 -n a/d14E) llog m))
n
(E.3)
(ii) (WaU)P P.,, denotes the pushforward distribution. With the conditions KL (N Ia,)) < 0o, and step size μ E(n, to, L) (dov/lg do). There exists an orthogonal matrix U Bddd such that we have the following upper bound for the total variation distance
TV(P), PW. = =0(1) (113/2144) log"
(E.4)
to
where Ô hides the factor about de do, d, and Le..
(iii) For the generated data distribution Pro. the orthogonal pushforward (7 W WT Pr. (O, E), where Σ ~ alg / for constant 0.
Proof. Please see Appendix E.6 for a detailed proof.
Remark E.1 (Compare with Existing Work). In (Chen et al., 2023c, Theorem 3), the upper bound log" n Therefore, for > 1, Theorem E.3 gives tighter accuracy if 3d + 11 > 12/d + 16Ī where d < d and İ < L. On the other hand, under similar conditions for d and L, Theorem E.3 suggest to achieve similar total variation distance we only require Vlo early stopping time which is beneficial for empirical setting.
for total variation distance with ReLU network is ő 1/(cnto)n (n2(/ont)on//--s-))) 1/((+)))
E.4 PROOF SCORE APPROXIMATION (THEOREM E.
To begin with, we restate some auxiliary lemmas and their proofs here from (Chen et al., 2023c) for later convenience. Note that some of the proofs extend to the latent density function.
27

---

Published as a conference paper at ICLR 2025
Lemma E.2 (Modified from Lemma 16 in (Chen et al., 2023c)), Consider a probability density
function Ps. (hly) exp(-C1)11))) for Id and constant C' Let Th > be a fixed radius. Then it holds
20/ do exp(-Cri. Jmllssa Ph (1ly)) dh v (doo +1 h
2d do/2 CT -1
| (hly) dh v
do exp -C
Lemma E.3 (Modified from Lemma 2 in (Chen et al., 2023c)). Suppose Assumption Assumption E.3 holds and is defined as:
(h. U..) uh. - /
heb (h(h) Ph (hly) dh, h B x. Suh (1(b)) ps (Niyy) dh
Given O, with rh c Vd log(d//// log(1/e) for absolute constant c, it holds
l|e (h, v, 11/l/112 m (2())) for t E [to.I].
Lemma E.4 (Modified from Theorem 1 in (Chen et al., 2023c)). We denote
i ot 9 y. 1)
T(rx)=
sup kElt. rx ye
sup
sup
With (h, v,t) = have (h) (Aly) 1h (((l)))p and pa satisfies Assumption E.3, we have a coarse upper bound for (Fr))
T(r))= -0 (
(1+8) 88
[oo oz² Vaera) -dd Vdo).
Br
Proof of Lemma E.4.
ot (h, v, = T /
h 2
or vz (hll))) dh U (ht((hh))pa((M))) S vo (Fll))) (hly)) (ʃ (e (..6Fhhppuu (kly)dh))
dh
(h)(h))))) (hly)dh
14 [11.122 - (1 88 Th + 3 whe (h(h)))) (hly)
1
dh
(h(h)pp (hly)dh
hus (hi(()))) S 2 (11112 (1 B7) - vhe (Hl)) (ʃ (Mlla)))
(hly)dh dh
①
3r of
Erp. [n]1n[[]] - (1+ 3?) Cov [h/h] Tt)
where plug in Orh (Tll)/tt Be (1 (1) + 1.17111)) (h(h)//oo and collect terms in (i). Since Ph. has Gaussian tail, its third moment is bounded.
Then we bound ||Cov(A (h [i]] Ilon by taking derivative of $4 (h. y. t) with respect to h, here
84 (h. U 8 oo J (h
(W dh U h
h ((
28

---

Published as a conference paper at ICLR 2025
Then we have
0
31
2
f hhh "p(k, y)hh
hup(h. f hT ((h, y)dh t/v
s
(h, y.t) (k,
0h
0
3t
1 oī Cov(h(T))
U
!do
where
(hh
Along with the LRg Lipschitz property of 84, we obtain
of B3
Los 2/ oi
[Cor(h)/))lopp
Therefore, we deduce
T(m)) (0 -011t000 (0 Let +
1 -/2 o7
vdora =0 Le,
Br
as Ph. having sub-Gaun Gaussian tail implies Ep, [1I1i1]] is bounded.
Lemma E.5 (Modified from Lemma 10 in (Chen et al., 2023c)). For any given >> 0, and I- T-LLipchitz function defined on [0, 11 x [0, 1]d,, there exists continuous function f constructed by trapezoid function that
19 9- J S
Moreover, the Lipschitz continuity of J bounded by
[f(x,v) J(x', v/) < 10doT./la 2'112 (.dy v'll2-
for any x, r E [0, 1] and y. [o, 1]
Proof of Lemma E.S. This proof closely follows Lemma 10 in (Chen et al., 2023c). We divide the proof into two parts: First, we use a collection of Trapezoid function f to approximate the function defined on [0, do [0, 1]dy, Then we establish the Lipschitz continuity of the function f to facilitate the approximation with a transformer.
1. Approximation by Trapezoid Function. Given an integer N > O, we choose (N + 1)do points in the hypereube [o, 1]do and (N 1)dd points the hypercube [0, 1]dy, We denote the index of the hypercubes m [m1, m2, .ma,)" {0.... .N and n [n1, 122 [d,]
(0.-.,,N] Next, we define a univariate trapezoid function (see Figure 3) as follow
1, a <
((a) =
2 |a],
[1, 2 a
(E.5)
0,
|
2
29

---

Published as a conference paper at ICLR 2025
① (3N (ITE
N )
m&/N
31
Figure 3: Trapezoid function.
For any E [0, 1]do and y [o, 1] define partition of unity based product of trapezoid functions indexed by m and n,
(m, {uV E
n -
n
da II
m. I N )))
3N
(E.6)
N
N
For example, the product of trapezoid function (m,a(1,)) * 0 only if y E (" v and
mm2-31133 m+2-1-37 For any target L-Lipschitz function g with respect to x and y, it is more convenient to write its Lipschitz continuity with respect to the loo norm, i.e.,
lo(0, v) o(d','') < L| - 2'l12 + L - 1/112
S LL dol x I'1oo - LV dylly o v'l.
(E.7)
We now define collection of piecewise-constant functions as
Pm, y) = gmm,n) for m E NO. and n E f0. NJ.
We claim that F(x,, y) = Em,n (m, y) Pm (2, y) is an approximation of g, with an approxi- mation error evaluated as
[F(x,)) - g(x,4)
sup == "
sup
Σ Em v) (Pm.n(x.V) g(x. y)) =e[o.1]40 ( mm,.
sup
sup
sup sup
(m,,)(,,)) g(0,
=E[0.1] y
m: ma/NN /NI luy nn/NIE( 2.x.2k11
Σ
la(m,n) g(x, v)1
sup
sup
=E[0.1] ye "
mcla my/
m:|0, nn/NNE( 2x.
LV 20
+ LL d de
(By Lipschitz continuity in (E.7))
3NV
2N
V +1
dg
N
2
where the last inequality follows the Lipschitz continuity in (E.7) and using the fact that there are at most 2do terms in the summation of m and at most 1d terms in the summation of n. By
choosing N [T. (Vig2do+1 + /2y/2) /e],
have ||g S..
30

---

Published as a conference paper at ICLR 2025
2. Lipschitz Continuity. Next compute the Lipschitz of the function f with respect to and y. Suppose the approximation enror e is small enough, then we have
[f(, f(z',v/) ./)l
$ [f(z, v) - g( w) + 1g(2, y) - g(x', y') Ig(a' -
s 2e + Lvdlllz x' l' L dly 1y y/ 1l
x 10L dol - I' + 10L v dg - 8/ 1/11
x 10Ldo||=- z'112 + 10Ld,1ly 5/112-
This completes the proof.
Main Proof of Theorem E.I. Now we are ready to state the main proof.
Proof of Theorem E. From low-dimensional data assumption, the score function pegpe(l)))) de- composes as the on-support and orthogonal component (see Lemma D.1). Recall the on-support score function is given by V log P (fly) = UT s y. f) from (E.7). We use latent score network to approximate the score function (see (K3)). Specifically, the latent score network includes latent encoder and latent decoder. The encoder approximates U E Rdoxdz ,and decoder approximates U E Rd. xdo At its core, we use the transformer 9T (W 2, v, E Th.s. to approximate q y, as defined in (D.1). The expression for q qh,, y. is given by:
(h, y, ofV log (U xay)) +U x= ofU o (h
(E.8)
We proceed as follows:
Step 1. Approximate qT,, y, t) with compact-supported continuous function f(h, y, t).
Step 2. Approximate F(h, y, t) with a one-layer single-head transformer network.
Step 1. Approximate qh,, y. with Compact-Supported Continuous Function f(h, y, f). First, we partition de into a compact subset H₁ == (h |1]|| S rn} and its complement H2, where the choice of тh comes from Lemma E.3. Next, we approximate q(h, t) on the two subsets by using the compact-supported continuous function F(h, y, t). Finally, calculating the continuity of f gives an estimation error of /do + dy between q(h, t) and f(h, tt.. We present the main proof as follows.
Approximation on H2 x |0, 1]x (4,, T). For any > 0, by taking = c(vo do log(do/to) log e). we obtain from Lemma E.3 that
|1 y. t) 1k1 2 rh LP()) Se for te [to. T] and y e [0.1]. 1].
So we set F(h, y. t) = on H₂ J0, 1] x [to. 7].
Approximation on H1 [0, 1] x [to. TJ. On H1 [0, x (to, T). we approximate
gT. 1) = [g (T,, (), qa qa y. (),... qdo (T, y, O),
by approximating each coordinate qu (h, y, t) separately.
We firstly rescale the input by h' = (7 + 1221//r and t' t/T, so that the transformed input space jo, 1]do x [0, 1]d, x [to/ /T 1]. Here we do not need to rescale y. since it is already in [o, 1] by definition. We implement such transformation by single feed-forward layer.
31

---

Published as a conference paper at ICLR 2025
By Assumption E.2, the ou-support score 8+ (h,y,tt) y. ) is Lgg Lipschitz with respect any h R and E Rd,, This implies q(,, y, t) is (1 + Ls_ J-Lipschizz in h and y. When taking the transformed inputs, g(h', t') g(2rah' each coordinate gı g&(h', y: 1) also (1 + -__. J-Lisshhtt
11 Tt)) becomes 2r + Iay -Lipschitzz in h';
h.. Here we denote L, 1 Ls+.
Besides, g(h', y, 1') is - Lipsichitz with respect to t, where
8 Ot
ah, 1)
(rk) = sup se(ta. (e( yEju,1jd
sup
sup
We have a coarse upper bound for ((r)) in Lemma E.4. We restate follows:
T(rN)= = = =0(-)))00 (1+, 33
Be Lo_ + oo vdora)
Th Lo
Since each gb.hh (h',,,, M, 1) is Lipsichitz continuous, we apply Lemma E.5 to construct a collection of coordinate-wise functions, denoted as && (h', v) We concatenate f''s together and construct faol" According to the construction of trapezoid function in Lemma E.5, for any given e, have the following relations:
sup h 11. x(to//
1,W0,,, y,,,,, S
Considering the input rescaling (i.e., h h', y and t + Ł'), we obtain:
The constructed function Lipschitz continuous in h and y, i.e., for any h1 h2 HI, 1ı v22 [0, 1] and t E [to. T), it holds
[h .1 Aih. v2, 11 I 1.d -. | - Till22 I. 11 82ll2-
(F.9)
The function is also Lipschitz in 1, i.e., for any 4, E [4. 77 and |10||2 Smh it holds
[1 (11,,,44 42) s(h, (2)l 10r(r))l(1110)) 12112-
To conclude, the construction of F(h. t) uses a collection of trapezoid functions, as described in Lemma E.5. This ensures that f(h, y, t) 0 for |7/|| > The for all [fo, T] and y [o, 1]. Consequently, the Lipschitz continuity f(h,, y. t) with respect to extends over the entire space dd
Approximation Error Analysis under L Norm. We first decompose the L approximation error of J into two terms (|7ill Ta and ||0]|, < r.):
[aah, v. [(, u. (....)))l (1)))
= -11( - s(,, 1)) [1All] < rn (nbllee(rr)) + (1a(h, ((lllll ra) ralllea(me)
By selecting ги (v log(do/to) log(1/e) (see Lemma E.3), we bound the second term on the RHS of above expression as:
11o(T.
( rul rulllee(pp) Sc.
32

---

Published as a conference paper at ICLR 2025
For the first term, we bound
(1ahh, v. ( < <Th)ll<=(PP)
$ v do + dg
www.veua1foxx000m |[ g(V, v., v,
sup
$ do do + dye.
So we obtain
(lh 1h,, (la(p)) s (Vio dy +1).
Substituting with e/2 gives an approximation error for y, of do + dye.
Step 2. Approximate f(h, y, i) with One-Layer Self-Attention. This step is based on the universal approximation of single-layer single-head transformers for compact-supported continuous function in Theorem G.2.
Recall the reshape layer R(-) from Definition 2.3. We use f() == R-1 Ôr R(-) to approximate f+()) [(, where ÔT(1 (-) Thisr (/2 (FF) f(SA) f(HH) RdxxL L - Iā
We first use (-) == R- o gr R(-) to approximate the function Fe(-) constructed at Step 1 and
denote H = R(h). Using Theorem G.2, we have:
1/2
[((h. - f(. 1..w)
(fzz |1chh.0) F(b,, vi vill(ah))
(E.10)
1/2
(nr Ro RoJooo 1 Re Jr Rt RT- (m)l
1/2
Fe R (() कr (1])l dh
$ c.
(E.11)
Along with Step 1, we obtain
(((h,,.) F(h m2(p?) $ 119 v,t) - (ihh) t) (pp))) + [F - @T(
(1 + do + dy
The approximator sw for the score function V log lo (hly) is define in (D.2) where st (UU "x, v. o/ The approximation error for such an approximator is
1 dd dy c,
V lo sw (- S
for all t [to, T].
of
Finally, the parameter bounds in the transformer network class satisfy
- (log L)+): Wollz = [[xll2 = c
(1+22
|ell2,00 ||x1l2,00 = o o - -G++2T) (log L)2)
33

---

Published as a conference paper at ICLR 2025
Woll2 - (abet) ||Woll2xx = =0
Wv O(d+): |Wvll2,0 O(d);
[Will2 = O de ; 11 2,00 0 d :)
[W2ll2 = O (de 21122222, = -
|E 11,,0 = o d T3))
We refer to Appendix G.2 for the calculation of the hyperparameters configuration of this network. This completes the proof.
E.5 PROOF OF SCORE ESTIMATION (THEOREM E.2)
Lemma E.6 (Lemma 15 of (Chen et al., 2023c)), Let 9 be bounded function class, i.e., there exists constant b such that any function g 9 Rd + [0, 6]. Let ø1, 22. 2n Rd be i.i.d. random variables. For any 8 E (0, 1),a a & 1, and e >> 0, we have
P sup Σg
1
g(z)) (1 a)E [g(=)
(1 (1+3/a)B Ne + +2 s
log
gEg
3n
i=1
P
sup E [g(=)] gEG
1 1+
Σ g(z))
+1+ 6/a)B
log
N1c.G,11-1loo
(2 (2+ a)e 5..
n
6
Main Proof Theorem E.2.
Now we are ready to state the main proof.
Proof of Theorem E.2. Our proof is built (Chen et al., 2023c, Appendix B.2).
Recall that the empirical score-matching loss is
[(sw)
Σ Ela,, yii sw))
(E.12)
with the loss function for data sample (x, 2) is defined as
1' 1
f(x, y, sw)
To T (ca [1s( Ty, V log di (x dt.
We organize the proof into the following three steps:
Step 1. Decomposing L (swp): We first decompose C into three terms (A), (B), and (C).
Step 2. Bounding Each Term: We then bound three terms separately using some helper from
Lemma E.2 and Lemma E.6.
Step 3. Putting All Together: Finally, we combine the above bounds and substitute the covering number of S (Cz) from Lemma J.3.
Step 1. Decomposing L (stp):
Following (Chen al., 2023c, Appendix B.2), for any E (0, 1), have:
[(sw)
34

---

Published as a conference paper at ICLR 2025
C ($p)) (1 + a)
trunc ($w)
(swp)
Ltrune ($
+(1 a)
inf [(sw).
(A)
(B)
(C)
where
grrune (op) = Ba [e(x; Ty, sw) ) [1|a]]2 Srz]],
Tx > B,
We denote
= + Tx = o (Vdn log do + log C + log(n/8))
expl (//0-///07/// to),
Step 2. Bounding Each Term: We bound (A), (B), and (C) term separately using some helper from Lemma E.2 and Lemma E.6.
Bounding term (A). For any O, following (Chen et al., 2023c, Appendix B.2) and applying Lemma E.6, we have the following for term (A) with probability 1- 8,
( (1+3/a) + r log
N ( (T- Ther (1-111)) + (2 + a)c
(CT+r=) [CTttr)loN[//o]
(A) -0
nto to)
where c < 0 is a constant, and E. > 0 is another constant be determined later.
By setting e, log(2/(nlo(T- (n))), then we have
N (((((Cr+ra)) log (T/4))) T hh, .111112)
(A) 0 (1 (1+3/a)(C7+ +
1
log
(T to)
8
(E.13)
with probability 1- 5.
Bounding term (B). Following (Chen et al., 2023c, Appendix B.2) and applying Lemma E.2, we has the following bound for term (B):
CGrdo 2-2/do+2dh do LT to) I(do/2 1)) exp exp (-C2r
(B) o
(E.14)
Bounding term (C). In Theorem E.1, we approximate the score function with the network Sw for any é O. We decompose the term (C) into statistical error (C1) and approximation error (C2):
(C) s [(Sw) (1 ctrune (Sw) +(1 a) Ctrune (Sw)
(Ct)
(C2)
Following (Chen et al., 2023c, Appendix B.2) and applying Lemma E.2 and Lemma E.6, we have the following bound for term (C1);
=0 ( (1++6/1))1))) 6/a)(C7 + + nt to)
!) log 5
(Ct) Ftrunc (Sw) - (1 + a) Ctrune (Sw) (8w)=
with probability
35

---

Published as a conference paper at ICLR 2025
Finally, for the term (C2) we use Theorem E. for score function approximation of ((sw):
(C2) = 0 dg + dy to - to)
2 + (const.).
This give us the bound for term (C) < (Ci) + (1 + (C2) as
(C) (C) so
( ( + r
log
1
to (T to) (
do + dy
(const.).
(E.15)
nT to)
Step 3. Putting All Together: In the final steps, we combine three terms and substitute the covering number get the score estimation bound for latent DiT.
Combining (A), (B) and (C). Following (Chen et al., 2023c, Appendix B.2), we set a = and get the overall bound:
T - to I (,t) Vlogp(())l
(C7 + +r
((n(Cr+ log (T/to))) Syhaar, 11.12)
1
do + dg
=0
log
+
e²
enoo (T' to)
8
n to to)
(E.16)
with probability 33.
Before we move on the covering number of Th.s... we first compute the Lipschitz upper bound LT and model output bound CT.
Lipschitz Upper Bound L, and Model Output Bound Cy. We then compute the Lipschitz upper bound Ly for the transformer. We denote Fr,n(+) Ro ge R-1(-) and (R(h).)) v) We get the Lipschitz upper bound for JT E Thie
Frr (H₂) - Fr (H2) $ ( (H1) J (H1) F + FI. (H2) fr (H2)
+ -1 - F (Hz))
$ 26 + [ (H1) - -J (H2)
(By (E.10)
$ 2e + 10(du + ddy H1 Hell
(By (E.9))
Then we get the upper bound Lipschitzness of Th.s.
Lt = O ((do + dy) Ls.).
(E.17)
Next, we compute the model output bound for T h...r For the output of the constructed transformer
Fr E Thisr, according to (G.17), the output of the network is lower bounded by O(1). Thus with the Lipschitz upper bound I.T O((d + dy) D., ), we have [ O(dd + dy) rk), where H F S Th- With 1 = do log (do/to) + log(1/c)), we obtain
CT 0 day v loggld//// log(1/6)
(E.
Covering Number of hh...r The next step to calculate the covering number of Thee.r In particular, Th.sr consists of two components: (i) Matrix Wo with orthonormal columns; (ii)
36

---

Published as a conference paper at ICLR 2025
Network function gr. Suppose have Wo1, Wuz and 91, 92 such that [Wo1 Woe 51 and S8r vvz log du ve ll (z, y, - g2 v. I 82. where 91 R-
9T1 o R and 92 R-1 972 Then we evaluate
sup |3r|, va, log d ".
swee.ori (x, - (a
1
Wougi Wo202 8,
sup 0i [k:1| ve0
1
[Woig( Wungi (W
sup
of || + da log da 1e0.
1 term
+ Woigh (Wu2 Wo Wo 112
9l v. Wu292 (W N.
2nd term
3rd torm
1 $ of
LT Lob1/do(3ra + v de log de
+
52 2udterm
81
(E.19)
1st term
3rd term
where LT upper bounds the Lipschitz constant of gr (see (E.17)).
For the set (WB E xdd : [BBll S 1), its coverrinn number is (1 + 2//6//1) á.do (Chen et al., 2023c, Lemma 8). The ò2-covering number of f needs further discussion there is reshaping process in our network. For the input reshaped from h Rdo to R we have
TRll S Fx
H S Tz:
Thus we have
[1. v. 92(h, 82.
sup |b|| DrD D. Del
sup | D.pe
(H) - 9T2(H)ll2 S62.
Next we follow the covering number property for sequence-to-sequene transformer hisir i.e. Lemma J.2 and get the following d2-covering number
R
log (ees 1f,.5,11111,.
(E.20)
3
log(nL) a
((C)) + (d d,) (C2 + +d d,) (2(C)) Ckay)
jet term
2nd term
3rd term
(B.21)
where
a = (o + CE).
Recall that from the network configuration in Theorem E.I, we have the following bound:
37

---

Published as a conference paper at ICLR 2025
WWll2 = [Wxll2 1 - 0(ā (1.+2t)) (1gg Ii));
Well2 = | - 0 ( 11222 (log L)}t;=
[Woll2 O (ated) 1| =0 (et):
|Wvll 2 O ); W =0001,00=001)) =Odd);
[Wills - O (āe :) /|Willl, - -0 =0 (dbc
||22l12 o (āe ||22l2,.0 0 (dbc
|| 11 112.0 o (.)
Note that W WoWk and Wo,v WoW.. Combining every component and substitute into (E.20), we have three respective terms bounded as
1st term
2nd term O (do +dy) 2/133333333
3rd term O (do + d,)2/3, (1o8 2/3 -c (1/38113((18)1)
Apparently the 3rd term dominates the other two. For the a² term, we write
a² = 12=0(a²) 2(3/d+44) -(((0)=(0/)) (log (log
where C22 = (Cz + (do + 2
Combining the above bound we get the log-covering number of Tz as
log.N Ecr R (1.0,,111112)) Thesr .11112) z o log (nI) log (i) 22 (do + d2 (1//++1)1))
(E.22)
E
Substituting the log-covering number of Thhs. into (E. we have
T - to L (| v Vlog (1
(C7 + log 1os(2)) (log (nI) log (T)
=0
2 (do + dz) - 02
+ 1 + da + dy
to to) 2 (By (E.16)
ne2to -t) (T' to to?
+ log log(?)) + log (3) log (nI) log (L) p2 (d + dy) 211) 1( C2 de (d + 2.2 (do log
0
to
no 1o)
(T to)
do + dy ?
(By (E.17) and (E.18))
(T lg)
38

---

Published as a conference paper at ICLR 2025
Balancing Error Terms. To balance the error term, we set -3/((4++3/d+11) Also setting ò 1/3n then we have
T to [ || (:,) - Vlogpe())) dt ( 122 do) + dg)²
to n (1(33//d4a)) log L log" n
(E.23)
with probability of 4. This completes the proof.
E.6 PROOF OF DISTRIBUTION ESTIMATION (THEOREM E.3)
Our proof is built on Chen et al. (2023c, Appendix C). The main difference between our work and Chen al. (2023c) is score estimation error from Theorem E.2. This is based on universal approximation of transformers.
Consequently, only the subspace error and the total variation distance differ from Chen et al. (2023c, Theorem 3).
Proof Sketch of (i). We show that if the orthogonal score increases significantly, the mismatch between the column span of and Wy will be greatly amplified. Therefore, an accurate score network estimator forces U and Wo to align with each other.
Proof Sketch of (ii). We conduct the proof via steps:
Step 1: Total Variation Distance Bound. We obtain the discrete result from the continuous-timg generated distribution Pro by adding discretization error (Chen al., 2023c, Lemma 4). It suffices to bound the divergence between the following two stochastic processes:
For the ground-truth backward process, consider h
B3 y/ and the following SDE:
dh
2 hr v VogppTT 1(ht) de dut
Denote the marginal distribution of the ground-truth process Ph.
For the learned process, consider Tı and the following SDE:
dh, d
1 Th + B (h, .T -T -T)) 0) de +UU,
2
where (z, = ((M,,)) = and M an orthogonal matrix. Following the notation in (Chen et al., 2023c), we use (W& Pr, to denote the marginal distribution of Pr.. We first calculate the latent score matching error, i.c., the error between log pt (h, y) and s (h, y, t). Then, we adopt Girsanov's Theorem (Chen et al., 2022) and bound the difference in the KL divergence of the above two processes derive the score-matchine motee-matching error bound.
Proof Sketch of (iii). We derive item (iii) by solving the orthogonal backward process of the diffusion model.
Definition E.I. For later convenience, we define {(n, lo d, L.) = z 433/d=11))
log n.
Here we include few auxiliary lemmas from Chen et al. (2023c) without proofs. Recall the definition of Lipschitz norm: for given function f. (I)(p)) = sup (l) - ffl)ll//lee | vllz).
39

---

Published as a conference paper at ICLR 2025
Lemma E.7 (Lemma of Chen et al. (2023c)). Assume that the following holds
Px V Po < Caho AmmP (1h 2 cus
Bawps |1/112 S Cn.
where Amin denotes the smallest eigenvalue. We denote
[[ /'¹ of
of (o(:;
We set to & (in) and T max (ogo/o/) 1}. Suppose we have
8. Uq(B 2, y. 1))l Se.
Then we have
W.W. - UU 1| O(tto/co).
and there exists orthogonal matrix M Rddaxdo such that:
E B| f(Mh, v. q(h, y. t)
to [T-- log to) max II((; + C.nl cu
max; ((((())li Ch
=6. O
Cp
Lemma E.8 (Lemma 4 of Chen et al. (2023c)), Assume that PA, is sub- -absGaussinn, f(h, t) and log ph (hly) are Lipschitz both h, y and Assume we have the latent score matching error-bound
En-ep | (he, y, Vlogr (helv))l di élatent (T Ln)-
Then we have the following latent distribution estimation error for the undiscretized backward SDE
TV (ph to Ph, S Elatenn (T' (T) + /KL KL (PAIIN (0, la.)) exp(-1').
Furthermore, we have the following latent distribution estimation error for the discretized backward
SDE
TV (ph. Pt. Phialia) S V Eiattent (T to) + /KL (0, Ia,)) exp(-T) + Veas (T -to).
where
max; [1, ys Mleip +
2 ns max [f, y, f))12
Cdis
(to)
max, [[f(-. Ilaip
n7 max {E |looll 2 .o + ndo.
a (lo)
and is the step size in the backward process.
40

---

Published as a conference paper at ICLR 2025
Lemma E.9 (Lemma 6 of Chen et al. (2023c)). Consider the following discretized SDE with step size y satisfying 7 to Kry
dy:
o(T - -kjj) Whydt + dU,, for E [Ky, (k + 1)11,
where Yo (0, I). Then when T and + 1 S 1, have Yr- N (0, with o² S Se + p).
Lemma E.10 (Lemma 10 in Chen et al. (2023c)), Assume that V log Ph (hly) is L, -ipssciti Then we have Ba V log Po v)ll s doLa.
Main Proof Theorem E.3. Now we are ready to state the main proof.
Proof of Theorem E.3. Recall that in (E.23), have
E(n, to d, L.)
1
(1-3/441)))
log L. log n.
Proof of (i). With Lemma E.7, we replace to be c(T to)² and we set Csh Lndo by
Lemma E.10, we have
Woo - UU TUT =o (hg(m, to d,
co
We substitute the score estimation error Theorem E.2 and T = Olog n) into the bound above, deduce
[Www - -u - = ( (
log
We note that log is great enough to make T satisfies T > max(log(Cn/doo + 1),1} where C₁ 2 En-r, [1lll
Proof of (ii). Lemma E.7 and Lemma E.10 imply that
E M f(Mh, t) - a(h, v. b) = OCetssm (T' - to)),
where
la [T (T- logto)do L2, + dgLn
L2 Oh
Clatent 0
Co
Go
Through the algebra calculation, we get
UT JTf(U,,y. y, hh
H(MT y. y(,,
En
-v Flogp/ dt
of
< Etaten (T - tp).
With Caatent and Lemma E.8, we obtain
TV(Ph) (W, Pdis)
41

---

Published as a conference paper at ICLR 2025
2 V Elttett (T - to)
-t) + /KL (PalI (0, Id.)) exp(-TT + Veas (T- to)
1 dg log do
(1+3//6+a1] llog" + n lovco
+
+ do
B
As we choose time step μr = 06///o 1///ov log don (113/d 4b)
=0
we obtain
( (1+3//++1))
TV(PH) (Wu pals) = )
llog' n
lov
By definition, ph. (UWB)) Palls This completes the proof of the total variation distance.
Proof of (iii). We apply Lemma E.9 due to our score decomposition. With the marginal distribution at time T to and observing μ < to. we obtain the last property.
This completes the proof.
42

---

Published as a conference paper at ICLR 2025
F SUPPLEMENTARY THEORETICAL BACKGROUND
In this section, we provide an overview of the conditional diffusion model and classifier guidance in Appendix F.1 and classifier-free guidance in Appendix F.2.
F.1 CONDITIONAL DIFFUSION PROCESS
Conditional diffusion models use the conditional information (guidance) y to generate samples from conditional data distribution P(-ly = guidance). Depending the model's 's objective, the guidance either label for generating categorical images, text prompt for generating images from input sentences, an image region for tasks like image editing and restoration. Throughout this paper, we coin diffusion models with label guidance y as conditional diffusion models (CDMs). Practically, implement conditional diffusion model characterized classifier and classifier-free guidance. The classifier guidance diffusion model combines the unconditional score function with the gradient of an external classifier trained corrupted data. On the other hand, classifier-fres guidance integrates the conditional and unconditional score function by randomly ignoring y with mask signal (see (F.6)). In this paper, focus the latter approach.
Specifically, consider data E Rd and label y 18 with initial conditional distribution P(xly). The diffusion process (forward Ornstein-Uhlenne
process) is characterized by:
1 X,dt + dW, with Xo ~ P(xly),
ax, =
(F.1)
2
where W, is Wiener process. The distribution at any finite time t is denoted by (a(l))), and Xoc follows standard Gaussian distribution. Up to a sufficiently large terminating time T, we generate samples by the reverse process:
ax;
2 x
Vlogpr- w)) dt + dWW with X Pr(xly),
(F.2)
where the term V IX (v) represents the conditional score function. We have X1 N (o, lf with et/ and of = e
We use a score network 3 to estimate the conditional score function V (oly) and the quadratic loss the conditional diffusion model is given by
3 = argmin E, SETH hooe
(((.o.)) [(r [|s(2' v,, f)- - Var logpr(z'|20)) 120)
(F3)
where Unif( T).
With the estimate score network S in (F.3), we generates the conditional sample in the backward process follows:
ax
1
x; +3 v. dt + aw, with N N Ia)-
(F.4)
Classifier guidance (Song al., 2021: Dhariwal and Nichol, 2021) and classifier- free guidance (Ho and Salimans, 2022) are piratical implementations for conditional score estimation. For classifier guidance (Song et al., 2021; Dhariwal and Nichol, 2021), it use the gradient of the classifier to improve the conditional sample quality of the diffusion model. According to Bayes rule, the conditional score function has the relation:
V log
(lgg
V log (egla())
(F.5)
Approximate by 3
Guidance from classifier
43

---

Published as a conference paper at ICLR 2025
It uses the neural network to approximate the unconditional score function V Pl (x, along with external classifier to approximate Pe (3)22)) and compute the gradient of the classifier logits as the guidance V log (y)
F.2 CLASSIFER- GUIDANCE
Classifier-free guidance (Ho and Salimans, 2022) provides widely used approach for training condition diffusion models. It not only simplifies the training pipeline but also improves performance and removes the need for external classifier. Classifier- free guidance diffusion model approximates both conditional and unconditional score functions by neural networks sw, where W the network parameters.
Our primary goal is establish the theoretical guarantee for selecting conditional score estimator ((r, y, 4) chosen from the transformer architecture class and bound the error for such estimation. Based on previous work by Dhariwal and Nichol (2021); Fu et al. (2024b); Sohl-Dickstein al. (2015); Ho and Salimans (2022), adopt the unified setting for the conditional difffusion model. First we define the mask signal as T = {0, id], where denotes the the absence of guidance y and id denotes otherwise. Unites the learning of conditional and unconditional scores by randomly ignoring the guidance y. Therefore we write the function class of the score estimator as
$1 (2, y, t, $2(2,, t),
if y R y= 0.
s(2, v.t)
(F.6)
Both 81 (2r. (x. y. and s2(2, 1) belong to the transformer function class with slight adaption. Following Fu et al. (2024b), we consider P(r = id) = P(r 0) = 2 without loss of generality, and we have the following objective function for score matching:
= argmin E. ((RO.s)) [E [lo V. (x')(r)))))) hua
aw ET
In practice, the loss function is given by
f(ro: yi sw) I" To
1
Br 20 N(
||sw (2e, Ty, - Va, log (ave (av(=0) di,
(F.7)
where To is a small value for stabilize training (Vahdat et al., 2021). To train sw we select n i.i.d. training samples (N0,, W where Ha, Po(-l1s). We utilize the following empirical loss:
"2
[(sw) - 71 Σ
(zo, ye: sw).
(F.8)
With the estimate score function sw y, from minimizing the empirical loss in (F.8), we use sw (z, y, f) to generate new samples. In the classifier-fres guidance setting, we generate new conditional sample by replacing the approximation sw in (F44 with Sw, defined as:
Sw (x, y, 1) = (1 11+10 v) sw (2, y, v sw (x, 0,1),
(F.9)
where the strength of guidance 77 > 0. The proper choice of 7 is crucial for balancing trade-offs between conditional guidance and unconditional ones. The choice directly impacts the performance of the generation process. Wu et al. (2024b) theoretically study the effect of guidance v on Gaussian mixture model. They demonstrate that strong guidance improves classification confidence but reduces sample diversity. For more detailed related work, refer to Appendix C.I.
44

---

Published as a conference paper at ICLR 2025
G UNIVERSAL APPROXIMATION OF TRANSFORMERS
In this section, we discuss the universal approximation theory of transformers.
In Appendix G.I, present the universal approximation results of transformers for score approxi- mation Section 3. We emphasize that most of the material in Appendix G. is not original and is drawn from (Hu et al., 2025; Kajitsuka and Sato, 2024; Yun al., 2020).
In Appendix G.2, we compute the parameter norm bounds of the transformers used for score approximation. These bounds are crucial for calculating the covering number of the transformers and are essential for score and distribution estimation Section 3.3.
G.1 TRANSFORMERS AS UNIVERSAL APPROXIMATOOS
The key idea for demonstrating the transformers' ability to capture the entire sequence lies in the concept of contextual mapping (Hu al., 2025; Kajitsuka and Sato, 2024; Yun et al., 2020). We first restate the background of (7, co-ottetltlll mapping in Definition G.3, using the definition of vocabulary (Definition G.1) and token separation (Definition G.2) in the input sequences.
Contextual Mapping. Let Z, RdxL represent input embeddings and output label sequences, respectively, where Z., E R denotes the k-th token (column) of each Z sequence. The vocabulary corresponding to the i-th sequence at the k-th index defined in Definition G.I.
Background:
Definition G.1 (Vocabulary). We define the i-th vocabulary set for i [N] by D(1 Uketz 2.4 Rd and the whole vocabulary set V is defined by V = U.E[M] v(() с Rd.
In line with prior works (Hu et al., 2025; Kajitsuka and Sato, 2024; Kim et al., 2022; Yun et al., 2020), we assume the embeddings separateness to be (Ymin ('min 5e-spaaatedd, as defined in Definition G.2.
Definition G.2 (Tokenwise Separateness). Let z(1,, z(),... Z(N) E RdxL L be embeddings. Then, ( Z(N) are called tokenwise (Tmin, Ymax. 8) -esparatd if the following three conditions hold.
(i) For any i E [N] and i E [n], 5,6 Ymin holds.
(ii) For any i E [N] and E [n), 2 < Ymax holds.
(ii) For any j.j [N] and k, [n] if - 2 then 1| 2 > holds.
Note that when only conditions (ii) and (iii) hold, we denote this as (7, 8) sseparateesss Moreover, only condition (iii) holds, we denote it as (8) (se-aaarasssssss.
Next, we define (7, (7 ccontextual mapping, building from conditions (ii) and (iii) in the definition of Definition G.2. The contextual mapping extends the concept of token separateness to captures the relationships between tokens across different input sequences effectively. This allows transformers' to utilize self-attention for full context representation.
Definition G.3 (Contextual Mapping). Let z(),.. ..... (N)) E Rdx 1. be embeddings. Then, a map q: RdxL -> dR L is called an (7, coontexxtall mapping the following two conditions hold:
1. For any E [N] and E [7.], ()) )..ell < < holds.
2. For any i.j E [N] and k, E [L] such that v(t) vi) or 2. k +
>
8 holds.
Note that a (2()) for E [N] is called a context ID) of Z
Helper Lemmas. For completeness, we restate existing lemmas before presenting the proof of one-layer single-head attention mechanism as the contextual mapping in Theorem G.I.
45

---

Published as a conference paper at ICLR 2025
Lemma G.1 (Boltz Preserves Distance, Lemma of (Kajitsuka and Sato, 2024)). Given (7, 6)-
tokenwise separated vectors 2(1), (M) E IR" with no duplicate entries in each vector:
2 +
where i E [N] and s, and
[L], s + Also, let
8 > 4 Inn.
Then, the outputs of the Boltzmann operator has the following properties:
Boltz 211) 2
(G.1)
Boltz (2())
Boltz
2 > 8' =In2(n) -22
(G.2)
for all i.j E [N], j.
Lemma G.2 (Lemma 13 of (Park et al., 2021)). For any finite subset X Rdd there exists least one unit vector E IRd such that
2'll S Iu (x -z']l s 1| 2'1l
X
Td
for any x''' E X.
Lemma G.2 provides the existence of a unit vector u E Rd that bounds the inner product of the difference between points finite subset X' c Rd.
We are now ready to restate the construction of rank-p weight matrices self-attention layer following (Hu et al., 2025) in Lemma G.3.
Lemma G.3 (Construction of Weight Matrices, Lemma D.2 of (Hu al., 2025)). Given dataset with a (Tmins Tmax -)ssparrated finite vocabulary V C Rd. There exists rank-p weight matrices W K, Wi Raxd such that
(WKva) (WQve) (WKv)) (Wous) > 6,
for any 8 >>0 0, any min (d, p and any Vay Ub, Ua with va # vh. In addition, the matrices are constructed as
P Σ
WK
piad Bxxd
Wq
Eps9 e Rexd
where for least gi, of Rdd are unit vectors that satisfy Lemma G.2, and pt, pl E R satisfies
po P/l (1V+ 1)a
eYmin
Proof of Lemma G.3. For completeness, we restate the key point from the proof in (Hu et al., 2025).
First, applying Lemma G.2 VU (0), there exists at least unit vector q Rd such that for any Va, Ub E VV and v, + vs the following holds:
(VV + 1)2 20. U. - -bll S 1q (v. va)l S lIva voll-
46

---

Published as a conference paper at ICLR 2025
Following (Hu al., 2025), let = 0 and all unit vector q {9 R" ||q/| = 1), and select some
arbitrary vector pairs Pt. pl E R that satisfy the constraint:
p; p41 = (1V++))
(G.3)
Cunin
By constructing the weight matrices as follows:
WK Σ pig E Ro
Wq =
Σ pog E Raad
where for at least one i, Pt, Pi,P/ satisfies (G.3) and gt. oj E Q we are able to demonstrate that:
(Wrva)" (Wove) (WKu6) (Wovs) d.
This completes the proof.
Any-Rank Attention is Contextual Mapping. Next, we present the generalized result where self-attention mechanisms of any rank serve as contextual mappings, extending the low-rank analysis in (Kajitsuka and Sato, 2024) any-rank attention weights, as shown in (Hu et al., 2025).
Theorem G.1 (Any-Rank Attention is (y, ( C-nttextull Mapping, Lemma 2.2 of (Hu et al., 2025)). Given embeddings ( ((N) E R which are (fimin Ymaax -ottewwwesse separated and Z(N) R I. be embedding sequences
vocabulary set V UNE[N) (() Rd. Also, let (11)
with no duplicate word token in each sequence, that is, 2 # z,4) for any i E [N] and k, [L]. Then, there exists 1-layer single head attention with weight matrices Wo E Rdx and Wv, WK, Wi E Rsxd, that is a (y, o)-contextual mapping for the embeddings ((1) Z(N) with
Tmax + e/4, 8 exp(- " log L).
where k. = 7/max/ 7min-
Theorem G. shows that any-rank self-artention is able to distinguish two identical tokens in distinct
contexts. Specifically, this holds for embeddings 2 z!6 when the vocabulary sets v(i) * ygi)
Since the proof of Theorem G.1 is crucial for subsequent analysis, we restate it for later convenience.
Proof Sketch. Hu et al. (2025) generalize Theorem 2 of (Kajitsuka and Sato, 2024), where all weight matrices have to rank-1. This achieved by constructing the weight matrices as an outer product sum E u.e where E R U4 Rd. The proof in (Hu et al., 2025) is divided into two parts:
Construction of Softmax embeddings by configuring the weight matrices according to Lemma G.3.
Different input tokens are mapped to unique contextual
Use the tokenwise separateness guaranteed by Lemma G.3 to handle identical tokens appearing in different contexts. Additionally, Lemma G. shows Boltz preserves separateness properties.
Handling Identical Tokens in Different Contexts:
With these, prove that self-attention distinguish input embeddings z 2 when the vocabu- lary sets voo) of vis)
Proof of Theorem G. For completeness, restate the key point from the proof in (Hu al., 2025).
47

---

Published as a conference paper at ICLR 2025
The proof consists of two parts: First, show that the attention layer maps different tokens to unique IDs. Second, we show that self-attention distinguishes duplicate input tokens when they appear in different contexts.
For the first part, by utilizing Lemma G.3 and set the weight matrices as follows:
Weight Matrices WK and Wo:
WK- Σ piq; Raxd
i=1 P
W6 Ddjg E Rexd
where Ps, Ps.p/ E I and go of Rd. In addition, 8 lnn and pr E IR" an arbitrary vector pair that satisfies
5
P p1 ((V) 1)*
(G.4)
EYmin
Weight Matrices Wy and Wo: In addition, for the other two weight matrices Wo E Rdxs and Wy E Rexd we set
Wv = Σ P Raxd
(G.5)
where 9" R.d, 4' = and pl RR. is some nonzero vector that satisfies
|Wr''l
(G.6)
4prmmax
This can be accomplished, e.g., Wo Ei= pp" for any vector p!"" which satisfies ||p""| mmx42 "p (2²) for any E [p].
Mapping Condition: With above weights construction, for i [N] and le E [Z]. we have
Wo (w) Softmax [ (wez!!)
e
(G.7)
4
For the second part, we prove that with the weight matrices Wo, Wv, WKY Wa configured above, the attention layer distinguishes duplicate input tokens with different context, z(6) 2,47 with different vocabulary sets 2(0) * DG),
define (1)) a) as
( = (W*z (Woz!!) ER", a0) -0)=(1 (0=(<zu0) T (Woz!!) ER",
where (1) and (1)) are tokenwise (r, 8) seppaatedd Specifically, the following inequality holds
5
la S ((V) 1)ad
2
7max-
exmin
48

---

Published as a conference paper at ICLR 2025
Since pio) * vu) and there is no duplicate token in Z and Z respectively, Lemma G.1 and obtain
Boltz (a()) 100--Bottt
-Boltz (
(ac0) T Softmax [a0]]
- -(abi) Softmax [a(i]
(G.8)
8
(Inn)² 1 2-22
Additionally, using Lemma G.3 and (G.4), and assuming 214 = 2,41 we have
(at)) T Softmax [al0] (a55)
Softmax [a6i]
(G.9)
Tmax ([V) + 1)a² Td 8
(1 )(0) Softmax [ol0]- - (gT2(()) Softmax [oci]]
E/min
By combining (G.8) and (G.9), we have
Σ qi ((4) Softmax [a(o] (4TZuu) Softmax
[ac)]]
8' (IVI
6Ymin
(G.10)
1)* dō
Finally, using (G.6) and (G. we derive the lower bound of the difference between the self-attention outputs of Z(1, 2 as follows:
(SA)
(Z(1)
((SA)
(z(r) 4'max (DV) + 1)a ddx
8' CYmin
(G.11)
where 5 41n and 8' (I)e with y = ((V) 1)a d ( (C/min).
This completes the proof.
With Theorem G.1 establishing that any-rank attention is contextual mapping, restate the universal approximation result for transformers with single self-attention layer from (Hu et al., 2025; Kajitsuka and Sato, 2024).
Theorem G.2 (Transformers with 1-Layer Self-Attentin are Universal Approximators, Modified from Proposition of (Kajitsuka and Sato, 2024)). Let S p oс and f(FF) f(SA) be feed- forward neural network layers and single-head self-attention layer. Then, for any permutation equivariant, continuous function f with compact support and > 0, there exists f' E Theer such that (f, < e, where dp (ʃ [Z) و ()) and || |l is the element-wise wwiiseef-none -orm.
Proof Theorem G.2. We restate the proof from (Kajitsuka and Sato, 2024) for completeness. The proof consists of the following steps:
1. Approximate by Step Function: Given a permutation equivariant continuous function
compact set, there exists a Transformer f' E Thh..r with one self aff-attention layer to approximate f by step function with arbitrary precision in terms of p-norm.
2. Quantization via f The first fced-forwart network f! quantize the input domain, reducing the problem to memorization of finite samples.
49

---

Published as a conference paper at ICLR 2025
3. Contextual Mapping f(SA) and Memorization f2: According Theorem G.I, construct any-rank attention f(SA) to be contextual mapping. Then use the second feed-forward f2 to memorize the context ID with its corresponding label.
The details for the three steps are below.
1. Since f is continuous function on compact set, f has maximum and minimum values on the domain, By scaling with /FF and /5 I assumed be normalized without loss of generality: That is for any Z R 1] 1jux2, we have f(Z) O. For any X E the function f(X) satisfies 1 E f(X) 1.
Let E N be the granularity of grid
Gp (1/,2/ 1) CRaxt.
such that piece-wise constant approximation
F(x) Σ f() 12L)-2EL---/DDO0p
LeGp
satisfies
dp(f. </33.
(G.12)
Such a D always exists because of uniform continuity of f.
2. We use fi to quantize the input domain into Gp-
We first define the following two terms for first feed-forward neural network to approximate.
The quantize term (quant dxL
R -> RdxLy; Quantize [o, I] into (1/D,.... 1}, while projects R \ [0, to 0 by shifting and stacking step function.
D-1 ReL.U (w/6 1/8D) ReLU 10/6- 1-18DD 1/8D)
D
0
x<0
x
1/D
05
quantp(a)
(G.13)
1/D
The penalty term (penalty): Identify whether an input sequence is in [0, 1]4x1, This is defined by
ReLU [0x - 1)/8| - ReLU - 1)/5 - 1] ReLU (=x/8] - ReLU (-x/6 - 1]
-1 5
penalty(x)
0 0<r51.
(G.14)
1<x
Combining these components together, the first feed-forward neural network layer l approxi- mates the following function:
d
(FF) 1
(X) quant L
(X)
+
venalty(Xr,,)
(G.15)
50

---

Published as a conference paper at ICLR 2025
Note that this function quantizes inputs in [0, 1] with granularity D, while every element of the output is non-positive for inputs outside [o, 1]dx1. In partieular, the norm of the output is upper-bounded by
max XEROXL
(x)=
dL Total number elements
Vd
(G.16)
Maximum Fuclidean norm d space
for any & [4].
3. Let GD C GD be sub-grid
G V G. + G.
and consider memorization of GD with its labels given by f(G) for each G GD. Using our modified any-rank attention is contextual mapping in Theorem G.1 allows construct self-attention f(SA) be contextual mapping for such imput sequences, because &o can be regarded tokenwise (L/D, Vd, 1/D) -seppartted input sequences. By taking sufficiently large granularity D of CD the number of cells with duplicate tokens, that is, |Go @ negligible.
From the way the self-attention f(SA) is constructed, have
((SA) -X.0|
1 max |X AvdD kE
for any le E [Z] and E R
If we take large enough D, every element of the output for X E R L \0,, 1]axL is upper-booudde bounded by
1
(SA) H (X),
( [d]. & [4])
4D
while the output for X E [0, lldp is lower-bounded by
3
((SA) f (X),x> 4D
(Vt [d], & e [Z]).
Finally, we construct bump function of scale R 0 to map each input sequence L Go its labels (LL) and for input sequence outside the range X E (-o0, 1/4D)axL to using the second ferd-forward f.F. Precisely, bump function of scale R 0 is given by
bump [(L.) d
L
ReLU [K(X,x - G..)) - 1] ReLU [R(2,, - Gt,a)l
dL
1=
+ ReLU [R(Z,, - Gt,x) + 1]) + ReLu Ze,x)]
(G.17)
for each input sequence E GD and add up these functions to implement f In addition, the value of /
(FF) is always bounded: 0 < L (FF)
< 1. Thus, by taking sufficiently
small quantize the step function, we have
dp (12 (110))) o ((SA) o ((F)
f2 o ((A)) o o x(1F))
(FF)
(G.18)
3
51

---

Published as a conference paper at ICLR 2025
Taking large enough D to make duplicate tokens negligible, have (FF) dp (1/ f(SA) F( .3)
(G.19)
Combining estimation of step function (G.12), estimation of quantization (G.18) and estimatation of duplicate tokens (G.19) together, we get the approximation error of the any-rank Transformer as
dp (1, f(SN) S1 .5)
(1F)) s1m),)) ce.
(G.20)
This completes the proof.
G.2 PARAMETER NORM BOUNDS FOR TRANSFORMER APPROXIMATION
In the analysis of the approximation ability of transformers (Kajitsuka and Sato, 2024), universal approximation is ensured by using a sufficiently large granularity D, a sufficiently small in (( and an appropriate scaling factor I in f(()) Here, we provide detailed discussion on parameter bounds for matrices in T T.. focusing on the choice of granularity and scaling factor.
Lemma G.4 (Order of Granularity and Scaling Factor). Recall universal approximation of trans- former, Theorem G.2. Suppose the target function f is defined on domain f [-1, d for some T E N. Further, suppose the target function is Lipschitz continuous on S2 with respect to the element-wise wise fp-norm. The order for the granularity and the scaling factor follows = O and R O(I)), and the parameter å for the first feed-forward layer in (G.13) follows ō o(D
Proof. We investigate the more precise choice of D, R, and ō respectively. We begin by deriving results for the domain [0, 1 and then extend them to [-1, 1
Bound on Granularity D. Note that there are O(D d GG omitted duplicated input. Clearly,
GD \ GD becomes negligible by taking sufficiently large granularity. However, we aim to evaluate the corresponding order of D. By the extreme value theorem, the continuous function If on [0, 1]axL is bounded by some constant, denoted by CB > 0. Moreover, the total number of omitted points are O(Dd(I--1) and the probability for selecting each point in Gp is 1/Ddr.
Therefore, the corresponding error is bounded by (D-4//)) Since require error to be bounded e/3, setting D o/dd for some constant p guarantees the result. Specifically, let f(-) [0, l]dx + [0, _] be the target function and f(-) be the piece-wise constant approximation of granularity D. The p-norm difference between f() and f(.). (G.12) gives
dr(,J) / /II((=)) [x))
(O(DAL-a) B (LDD)))
= -O(D(UL-)/)) O(D O(D-dL/y)
0(PD
Here, 4(D4//p) = implies D = p(dp)/d) for some constant > 0. For simplicity, we use D (1//)) in our analysis without loss of generality. Furthermore, for Lipchitz continuous target function f, there exist grid Gp on domain [-1, l such that
(,((l)), < L, Zl - L Z' S VaLLg/D.
52

---

Published as a conference paper at ICLR 2025
where Z' GD and / is the Lipchitz constant with respect the matrix 2-norm. Therefore, it suffices take / Altogether, we take D O(E such that Theorem G.2 holds.
Bound on Parameter 5 in f(FF) We ensure the error within region (i/D,i/D+ + 8) does not affect the desired interval (i/ D. (i+ ()++) for i E (D). Thus, we take 0 = o
Bound on Scaling Factor in f2 (PF)) We first ensure that R 0 large enough such that it maps input E (-o0, TD)
4 ddx zero. Since Zt,k Lt.k I aD by taking R ODD), we obtain (G.17) with all (-)U(-) output zero. Then, we ensure that R > 0 is large enough such that it maps L G C (4D dx to the corresponding label f(L). From (G.17), achieve this by:
ΣΣ 1=k==
ReLU RRS - 1| ReLU [SS] ReLU RSS + ReI.U|-RR)) = dl,
where 5 = - -== =
OD- For any S E R, suffices to take = O() such that [RS] S 1 for any uniform continuous target function on [o, 1] L To extend this to [-1, dxx. with the identical approximation error, scale the granularity D 2TD. In sum, have R = O(D) in order to achieve precision on Lipschitz continuous target function on | lJxx1
This completes the proof.
Building upon Lemma G.4, we extend the results to derive explicit parameter bounds for matrices regarding the transformer-basee universal approximation framework. That is, we ensure a more precise quantification of parameter constraints across the architecture.
Lemma G.5 (Transformer Matrices Bounds). Consider an input sequence E [0, 1]dxL Let f(2) [0, 1 d be any permutation equivariant and Lipschitz continuous sequence-to- sequence function. Then, given transformer network f' Tn Th.h.. (Definition 2.4) that approximates within precision, i.c., dp(fff') < e, the following parameter bounds hold for d > 1 and L 2 2
CкQ; 2.00 =
4d Cov. CCCo,cCa Cap = =(();
Cr, C2 Cr.C2.x = OT max (I)())) CE O(1),
where O(-) hides polynomial and logarithmic factors depending on and L.
Proof. We denote the separatedness of the input tokens by (Tmin: 7max, fx) and the separatedness of the output tokens by (7, 6z). We denote parameter taken in f corresponding granularity by dزz-
Bounds for Wq and WK in f(A)
From the universal approximation theorem of transformer Theorem G.2, with Pa p/ E R and Z. % being any unit vectors in Red construct rank p matrix Wq and Wx
WK Σ
P:o RRod i=1
Wq Σ plat e gaxd
with the identity p. p = (IV) 1)= Vmin) With this, we have the bound for pi, p/=
|ps o 0(NP))
8, ex Ymmi
|p(|| = (11p
6_
(G.21)
ExTimi
53

---

Published as a conference paper at ICLR 2025
Summing over the set of p. pl, for i [p]. we obtain the bound for rank p matrix Wq and W
(VMM1
de Touin
|Woll2= = sup ||-||==
|W S SCp=0
V
5g Caminn
|W
max 1sidd
C2 O
dd
|W S CK CK=O (VaIM??
8x
| = sup |=||=1
V
Eo Vanin
(()k) S 22.00 = O Igisd
/
6g
[ max
Emin
where p I and the head size S d. After the first step quantization, we obtain vocabulary bounds [V] (DD)) and output sequences with (1/D, vd, 1/D) tokenwise separatedness, and we take 6- = 4 log in Theorem G.2 to ensure that f(SA) is a contextual mapping.
Furthermore, to extend this to domain [|d,1 we rescale the granularity to D - O(D1). This gives D O(E ") Lastly, recall WKQ = W Wq. the bounds on Wxq follows
|[xxlll= s Ckq Od-- 1(42);;
| S SCk@ O 10-2, 114+2).
Bounds for Wo and Wy in (OA))
Following the construction of WQ and Wx in Theorem G.2, we have
Wv Σ Of e e Wo Σ
11 TT pi Pi Rdxx
with the identity ||r S c Ipp"") from (G.6), and p.¹ E R is any nonzero vector.
Along with the (7min 11/D, 7max vd, 1/D) separateness and taking D O(e 11), have the following bounds for Wv and Wo:
sup [Wvz2l1 - Cv Cv=cc==0 -((P)) = - (vad). |=||==1
WWvll 1|2
[Wvll2,0
max |I ([ 11a S c2.00 = 0 0 (d),
I5i
[Woll== sup || Co 0 (Vp p . .es) 0d
|||| ,=
[olll,,0 max [[ 1l s cgo00 0 (p p-1 Mmax .E.) =0 122))
where we use p < d. Extending bounds to the case where is defined on [-1, l we have
[oorllla WoWlll S Cov O((); IWovl22. WoWWlll S co 0(c)
Bounds for W1 in JFF.
In order to approximate the quantization Theorem G.2, we set up fFr as (G.13) where every entry of W2 in the layer is bounded by O(t/öD) and t O(D). Therefore, we have
vd
|W < ||W S SCr, =
|iill,,0 S C2.00 =0
54

---

Published as a conference paper at ICLR 2025
Furthermore, by Lemma G.4, we have vD-1 for (0, 1). Then,
|1 S SF = 0 ( O || - | s Ca = =
Bounds W2 in fEP.
The bounds for ||W2ll2- || in (G.17) follow the same argument as for W1, with the replace- ment of the largest element with the scaling factor R. So have
| I &OF2 = (VdR). |WWlll2 s =O==O(d))
Furthermore, by Lemma G.4, we have R =O(D) = O(c-)). Then,
2,,0 = ( = max lf(Z)lle)
max (Z) (max)If(2)lle
|| S SOP.
||2ll2 S =0(==(()) 0
Bounds on Positional Encoding Matrix E.
For E 11:
ET 2,00 it suffices to set the positional encoding:
12aax
47max
2m
E
2mmax
4-max
2max
Since the & norm every row is identical, we have
E
11,00
Σ
1.
(2irimax )² = -44 L(L + 1)(21)(2)))))) + 1)
= =0 (Timax LB3
6
Recall that we have the relation 7max
Vd in the self-attention layer. Therefore,
| 12,00 S CC - (2
(G.22)
This completes the proof.
55

---

Published as a conference paper at ICLR 2025
H PROOF OF THEOREM 3.1
Our proof leverages the local smoothness in Hölder spaces alongside the universal approximation property of transformers. Results presented in Appendix G handle uniform continuous functions and do not consider the higher-order regularity of the target function. However, bounding the weight matrices in transformer network function class becomes infeasible without information on the target function regularity. Therefore, we construct a function approximator for the score function that captures Hölder smoothness. Then, we approximate the constructed function using transformers.
Step 1: Function Approximation for the Score Function. Recall that (i) p.(z yy follows the inte- gral form shown in (3.1) (īi) the score function has the expression V Pt (zlvv) -r (xllv)/ PV We perform a -thth order and k2-th order Taylor expansion for po(alyy) and exp(:) to construct two function approximators (I, y, and f2 (z. y. t) for (z(z)) yy) and Vp (zly), respectively. These functions inherit the Hölder smoothness property of the density function class (Definition 3.1).
Step 2: Universal Approximation of Transformers on a Bounded Domain. We utilize the universal approximation of transformers to approximate the constructed function on bounded domain with arbitrary precision. Notably, the constructed functions possess properties stronger than uniform continuity, allowing us bound the weight matrices in the transformers. We reiterate that these bounds are essential for the later analysis of the transformer's estimation capacity.
the Full Domain with Sub-Gaussianity. We extend the approximation results to full space dd by leveraging the sub- Gab-Gaasiinn property of the target density function. Specifically, we apply the Gaussian tail bounds control the error outside the bounded domain.
Step Approximation
Organization. Appendix H.l provides auxiliary lemmas. Appendix H.2 presents the approximation of score function on a bounded domain. Appendix H.3 includes the main proof of Theorem 3.1.
H.1 AUXILIARY LEMMAS
In this section, we introduce auxiliary lemmas for the score approximation. Specifically, Lemma H.1 establishes an upper bound on the score function in the lo -istancee Further, Lemma H.2 presents an integral domain clipping technique that enables the score approximation on a bounded domain.
Bounds the Score Functions. We present theoretical upper bound the score function. We remark that we use this bound to determine the transformer model output bound Cy as well.
Lemma H.1 (Upper Bounds on the Score Function in fx Distance, Lemma A. of (Fu et al., 2024b)). Assume Assumption 3.1. Then, there exists positive constant K such that
[[|==|++1 of
| logg <
Before presenting the clipping technique, we first introduce the multi-index notation for clarity.
Definition H.1 (Multi-Index). Let E Rd,,, We say (a[1),.... add_ ]) is d, -dimension multi- index if every a a non-negative integer and satisfies (i) factorial operation: a! == a[1]!.- ... a !dz]!
(ii) derivative operation: d = @-(1) go(4)) (iii) power operation: x = x[1]a[1] x (dy [o(ds)
Furthermore, when contexts are clear, we use (o(z)y), p(zoly) and (u( to denote the true (target) density function throughout this section interchangeably.
Clipping Integral Domain. Let (0, 1) be precision parameter and B,, be bounded domain dependent on e and x E R. Considering the integral form of pe (xly):
Pr w) = Sr of (2 po(zolv) exp
dr
(-
||aaoo 2z
207
the next lemma shows that the integral outside of the BN is bounded by
56

---

Published as a conference paper at ICLR 2025
Lemma H.2 (Approximating Clipped Multi-Index Gaussian Integral, Lemma A.8 of (Fu et al., 2024b)). Assume Assumption 3.1. Let R E Z4 be an integer vector with ||<||| S n. Then, there exists constant C(n, dæ) 2 1, such that for any E Rd and 0 < E S 1/e, it holds
J Rdr BB
04,30 xr - p(zoly) (2 exp
|aa,o - xrl2
doo S
(H.I)
Tt
20
where
onC(nn dz) log (1/e) x + dz log (1/e)
B₂
04
n|
C(n, de) (1/e), C(n, da) / (1/e)
de
In Appendix H.2, we approximate the score function on B, and align the approximation error with error introduced by the integral domain clipping. We specify on approximation error in Lemma H.3 and Lemma H.4. For now, with hindsight,
N-8 for some N N. This gives:
Ba, O0, )o Bog N
+ t,, Blogg N
O2.
Ot
n Cl, C(O, d.))
Blog de
(H.2)
(II)
and the clipping error follows Lemma H.2:
II x/12
Pe(xly)
p(æolu) of (27)////2 exp
)ado dao S sc=
20
Next, we present the the score approximation on bounded domain.
H.2 SCORE APPROXIMATION ON A BOUNDED DOMAIN
In this section, we approximate components the score function and incorporate the Hölder smoothness index (Definition 3.1) 3 > O. Specifically, Lemma H.3 approximates Pc (zly),, and Lemma H.4 approximates (xly). Then, we approximate the constructed function on a bounded domain by leveraging the universal approximation of transformers (Appendix G.1).
Step Function Approximator for the Score Function. Recall that the score function has the form V log Pa (xly) = (p.(cll))/p((((()) First, we write (xly) into the product of p(æu (y) and exp(-):
Pe(xly) { p(zoly) ((aolv)aaallee)dr Jrp. of2 (2x)dd/12 p( w)xpp
x0l2 [_lla,a
dro-
207
Let Br,N be the bounded domain defined in Lemma H.2. Then, approximate p(zuly) and
exp | =all" with -order Taylor polynomial and o2-order Taylor polynomial within Ba,
207
respectively. Altogether, we approximate pr(zly) with the following diffused local polynomial:
llllll gr
fi(x.y,t)=
Σ
Σ
Pn t,
WE(N)A=,WE(NN dida [i +1 Ski Bn dra dy
R= Ral - #
(H.3)
57

---

Published as a conference paper at ICLR 2025
where
n.lll, 0Iv27 JB.,. Bugini
En R
+
vid
n=(+]
|||
dro-
2077
Bug
mld Ci0.
N
n [
PB. ( -
Pa (I, U, := ) o o x)) I M(<li), ne] oft), u).
Remark H.1 (Diffused Local Polynomial). The expression of the diffuueed local polynomial given (H.3) arises from the Taylor expansion applied on each grid point within [0, 1] where we use [N] and [N to denote the specific grid point undergoing the Taylor approximation. Furthermore, Assumption 3.1 imposed on p(roly) allows us to leverage the Hölder smooth property establish upper bound the error arising from the remainder term in the Taylor expansion.
The next lemma specifies the approximation error for pa (z/yy using [1x (.
Lemma H.3 (Approximation of (x by fa,, y. Lemma A.4 of (Fu al., 2024b)). Assume Assumption 3.1. For any x E IR E [0, 1] t > 0, and a sufficiently large N > 0, there exists diffused local polynomial (I, y) with at most a td,
+d= (d, + dyy monomials such that
to N.
Li (r (=(lv)) & BN
log
To avoid the score from exploding, we need to set a threshold for fi (z, y, t):
Definition H.2 (Truncated Approximator of pe(xly)). Let étow be positive real number. Let f1 y, be the diffused local polynomial defined in (H.3). Then, we define:
f1.elip (z, y, t) = max(flow, fi y,t).
Similarly, have the approximation for Vp (aly) based on the diffused local polynomial:
Lemma H.4 (Approximation of (p((x y) by fe(2, y, t). Lemma A.6 of (Fu et al., 2024b)). Assume Assumption 3.1. Let f2 (f2 [1) ... fa (dd Rdd where fa is diffused local polynomial for all i E [dz]- Then, for any x E Rd= y E [0, 1] 1,d, t > O, and sufficiently large N >> 0, holds
de +
(f2 y, +) o, Vp. [i] S BN log
N
where each [2 contains at most Nd. (d. + ya monomials.
To this end, we complete the approximation of p. and Vp, with diffused local polynomial fi and f2-
Step 2. Score Approximation with Transformers on Bounded Domain. We use transformers to approximate a function approximator for the score constructed with fi (x, y, t) and f2(1, y, t).
Lemma H.5 (Approximate the Score Approximator with Transformers). Assume Assumption 3.1.
Let (dz, 3, C1, C2) be a positive constant. Let fa y, == f2/(o, f1,cti) be the target function. Then, for any E (0, 1), any [-Cavlog N. C. Vlog Njd=, any y [0, 1]d, and any i [N CC- Ca log N],, there exists a transformer g(x, y, t) E T h.s.r
such that
JrIIs Vogn || v) () fa(a, v. v.. 2d < e. v
Furthermore, the parameter bounds in transformer network function class satisfy:
CKQ, c.oe = O(log 42 2- 4d 4d-2 Cov. C22 = 0(e);
Cp. 02000 O log -1 max |l f); CF = 0(1),
58

---

Published as a conference paper at ICLR 2025
where O()) hides all polynomial factors depending on dz. d, L, C2, 3,
Proof. Since the diffused local polynomials integrates over some polynomial functions and or is smooth function, the target (x, y. is Lipschitz continuous. Therefore, for any E (0, 1),
d2(g. = (f 11g(2, (1((2.,,)) y,t) fa( y, t)l
1/2 S.
(By Theorem G.2)
The parameter bounds in the transformer network class follow Lemma G.5, This completes the proof.
Next lemma incorporates previous approximation results into an unified transformer architecture.
Lemma H.6 (Approximation Score Function with Transformer on Supported Domain). As- sume Assumption 3.1. Consider 1, [N- C, log NJ, for constant Co. Co. and (x, [-C₂ Vlg N, C2 / Vlg N] x [0, 1jd,, where E and C₂ depends on d, 3, B, C1. C2- Then, there exist transformer network Tscore (2, v,,) 1) Thh such that
/ (p.(=ly)) logpi(zlv) Tswore (z, y, t) x oi
B² N - (log 20 N) 4 +b N)
The parameter bounds in the transformer network class satisfy
CkQ. C2
(log N))d+2); Cop = +): Cp, C2.0 ( (log N) ) Cg = 0(1): CT O( log N /of)
where (-)) hides all polynomial factors depending on d, d, L, 3, C₁, Cz.
Proof. Recall Lemma H.1, Lemma H.3, Definition H.2 and Lemma H.4. We define:
fs(2, = min (
f2
K (C.
(c
के
orJi,cip
where set ficpii (f, [ow) } to prevent the score explosion. We specify the coice of clw later.
We proceed with the following two steps:
Step A. Approximate Score Function with fa. For any i E [dz]], it holds
V log Pc] fs[i]
fa[1]
V log Pr]
fi.fi.
(V
(Vp.)
(Vpc)
fa
Pt
S1,tlip
S1,etip
ar.ii.atip
From Lemma H.1, the bound on the score implies (Vp.) < K dr log N + / Therefore,
(V log fali]l
K
1
(Vorp:)[]] [2
(V log N + 1)pt
Pr.
S1,clip
S1,clip
1 1 f1,llip
[VoePe)[[] fall]
Vlog log N lp- fi,tip
(By dropping Constant Terms)
Or
59

---

Published as a conference paper at ICLR 2025
From Lemma H.3, we have
If1 pil $ BN-BBll log dx N.
We set Clow = C3N - log
N S Pt such that fi 2 pt/2 by the choice of constant C3. Then,
I(V [T] [a[
1
[Vospe)[]] - f2(1]]
<
log Nlp: fi.lipl
(By the choice of clow)
Pr
oi
Ot
B
8 (log N) d
N
(By Lemma H.3 and Lemma H.4)
P
By the symmetry of each coordinate, it holds
B oip.
de t4
V log Pt fsl I V Voggtt fall2 S
N (log N)
(H.4)
Step B: Approximate fa with Transformer Teooe- We use transformers to approximate fa to an accuracy of order - such that aligns with the error order in (H.4). By Lemma H.5, we have
f [1 1) 1, e.
By setting N-0, it holds
pł I Tscore v.t. v.t) V log (xl)))))
< / p Tscores v,t) fa(ø, y. t)llrr + pfl.f((z,).t) v log Pt (xl))
(By triangle incquality)
/ (T, v. fs(x, 12 + pillfa(z,,,t) Vog pa (xl)))
(x [o, 1])
/ [x fa(a, y, 1)2d +
132
N)).+ki++1 f (log N028(log
N-28
(By (H.4)
/ (2, (x,, fa(z, y. 122 +
B²
N
28
(log (log.V)
-4 that1
(By ||o]|loe S C. N)
ot
B N N 29 (log N) +k1+1
< of
(By Lemma H.5)
Next, by Lemma H.l and x E 1-Covlog N, C2 Vlo Njd, we have
Vologtlll> = Oogoo N), Therefore we have Cf O(Vlog N), and by (H.4) we have
V log
fald] = O(N (log N)
This implies
||al|2 = O(Vlog N + - N) d+
60

---

Published as a conference paper at ICLR 2025
We take looser bound on fs such that holds for all dz:
fl - da O(loo N) dxtpt))
where use 1 S B.
Then, the parameter bounds follow Lemma H.5 with N-P. Therefore, have
CKQ. Cke KQ = O(N4 (log N) 2)) Cov, cor = O(N-9);
CF 22000 0 (N - ) Cg=0(1) CT 0 Olylog N N/o)).
This completes the proof.
H.3 MAIN PROOF OF THEOREM 3.1
In Lemma H.6, we establish the score approximation with transformer that incorporates every essential components and encodes the Hölder smoothness in the final result.
However, it is only valid within the input domain iCz Vlg N, Coylog Nj [0, 1]d,, and we also excludes region Pt élow where the problem of score explosion remains unaddressed.
To combat this, we introduce two additional lemmas.
Lemma H.7 (Truncate x for Score Function, Lemma A.1 of (Fu et al., 2024b)), Assume Assump- tion 3.1. For any R₁ 1, y, t we have
1 1111.20 (o(zlu)dz s Rí exp( exp(-C,
|> 2n1
Ra <
exp Cg
21
d
where C2 = C2/(2 max(C2, 1))-
Remark H.2. Because we only impose assumption on the light tail property of the conditional distribution in Assumption 3.1, the unboundedness of necessitates truncation for integrals regarding z, or else the result would diverge.
Furthermore, we address the explosion of score function with the second lemma.
Lemma H.8 (Lemma A.2 of (Fu et al., 2024b)). Assume Assumption 3.1. For any R2, y/s flow have
f M1 SRa
1(pp((11)1 flow) (=(l))d & R Clow.
J [1 (1lp( < Gow V (o S of
Rd-+ +2 clow-
Remark H.3. Recall that the score function has the form logr:(<ly) = Vp (2ll)))) It essential to set threshold for p. (xly) prevents the explosion of the score function.
We begin the proof of Theorem 3.1.
Proof Sketch of Theorem I.I. In the following proof, we give error bound for the three terms:
(A.1): The approximation for ||=]] Ri.
This step controls the error from truncation of d with radius R, in & distance. We approximate the error with Lemma H.7
61

---

Published as a conference paper at ICLR 2025
(A.2): The approximation for (zp()) < dow) and f||=||>> I R1}-
This step controls the error from setting threshold to prevent score explosion within the bounded domain || & Ri We approximate the error with Lemma H.8.
(A.3) The approximation for {pr (arly) 2 flow) and [| S R1)-
With previous two steps ensuring the bounded domain and preventing the divergence of score function, approximate with Lemma H.6.
Proof of Theorem 3.1. First, we set Ri R2 2 log N/C² in Lemma H.7 and Lemma H.8. Next, expand the target into three parts (A1). (A2), and (As):
V Pl pr(
() v (og (r(l)))
28 log N >
(A1)
(1pe(rlu)l < <low 1,(2,,0,1) V lo (all)dd
|x|| x S log
(Ag)
l(lp((rl))]
Clow [1 V log Po (x(l))ddz.
| s0 S cq
log
(As)
derive the bound for (A1), (A2), (Ag) and combine these results.
Bounding (A1). We apply Lemma H.7. Note that we have || y. 2) S Vlgg N /o? construction of the score estimator in Lemma H.6.
from the
28 log f [1 V
y, log g)(r)))d
(By expanding the & norm)
22
(s(2,,,)) (c()))) +2
V logp((xl)))1 - (=ly)de
/28 Cz
loRN
|| > / log
(By 11 < 1-11)
|1,(,,,,111 pe(xl))d +2 f
< 2dz
V ( Pe (a(y)dd
||=1|
/ 28 lox
||ll >
28 log N c7
(By the Eoo bound on the score function)
vlog.V N (vogg) of
/ ||=]|
S 2dz
pe pr(xl))dr +2
[oogp( pe(l)))d
2 log
|| >
log N
(By Lemma H.7 dropping constant)
log N ol
23
2 2,3
S 2dg
1
log N.
20
20
+
o C%
log
2-
(By dropping constant and lower order term)
1 N - 20 (log 2
01
62

---

Published as a conference paper at ICLR 2025
Bounding (A2). We apply Lemma H.8. Recall that we set clow (Lemma H.6).
C3N
1gNV~((lop "loo N) t
S [10) an / / loR
(xlp(())))1 elow) fow))1((z,8,t)) V Plogi(<ll))l pr(zly)dz
(By expanding the & norm)
21 (x(1]( Clow] (11(2, v. V log (zlv)))) p (x(l)))
|0ll on S. log
(By 11 = d
(x < clow} (d + | [ogg)()
Pa (zly)drr
20 log (e s
(By the foo bound on the score function)
VVog N oi
[xpp( {on)
dx
(Vogoi(=))) Pi (æly))
log
2
By Lemma H.8 and dropping constant)
Vlog N o,
2 2 C2
4
2,3 C2
d 2
foow
S dr
log N
Clow +
log N
of
By dropping constant and lower order term)
12
(log N)
clow-
Bounding (Aa). We apply Lemma H.6.
1 ((pe(c)))) 2 Clww Holsss,,, v, - V logg)(<z)))1 (x(l))ddr
23 log N S
-J 11
1 (1pp(v)
2 élow d=l1s(x,,8.)) V Vlog (xlv)d
(x(l)))
2 log N
(Multiply with p./p.)
(1pe 2 <low
d=l1s((,,,. v. - V Plog
||=1|000 S
Gr log N
Clow
(By l/p: 1/(lo))
B2 N N-28 of
28 (log N)
2 k1+1
(Lemma H.6)
Clow
Combining Three Upper-Bounds.
Combining (A1), (A2) and (Aa), we have
[}((,,,))
- V (zg
Rd
N- (log 2
flow (log N) 2ل1
B²
N 29 (log N)
34x +h
of
of
(At)
(Aa)
(Aa)
63

---

Published as a conference paper at ICLR 2025
By replacing flow with CaN (log N) Ny and using the relation k1 B, we obtain
J 11((,,,t) (og = O
B² N (
3 (log (log.N) N))
Last, the transformer parameter norm bounds follow Lemma H.6. This completes the proof.
'Recall the definition of the Hölder smoothness, Definition 3.1.
64

---

Published as a conference paper at ICLR 2025
I PROOF OF THEOREM 3.2
We provide the formal version of Theorem 3.2 at the end of Appendix 1.2.
Organization. Appendix I.I includes auxiliary lemmas for supporting our proof. Appendix I.2 includes the formal version and main proof of Theorem 3.2.
1.1 AUXILIARY LEMMAS
We utilize the condition assumed in Assumption 3.2 achieve the decomposition.
Lemma 1.1 (Lemma B.1 1. of Fu et al. (2024b)), Assume Assumption 3.2. The conditional distribution at time has the following expression:
C2 C211=12 2( + C207)
(x(ly) - (a? + C207)dl/2 exp
M1,, ().
Moreover, the score function has the following expression:
-C2x
+ Vh(r,,,1)
V Vlogp(
07 C207
h(x,v.t)
where M(x, v) = f(za.2)) exp( (2x))
Ga 2
a dro, θ, (a (a)+C20f)u7z and â of+0207
Proof. Let Ty denote the initial data distribution.
By Assumption 3.2, we have p(zly) = exp (-C2ll11B/2) f(2, v). Therefore,
pr(zlv) /
|| -|x- a021?
of (2 pl (27))a72P22lll
dz,
20 [(2,8) exp
C211=112
(-1le [_1ks-axi a12l1
o (2
exp
20
dz.
(I.1)
We rearrange the two exponential terms in (I.1) into
C2 C211=1/
| a,2l
Σ -
af2111+
+0000(+Co2
exp
exp
exp
20?
20
To simplify the summation in the exponents, we first rewrite the term for single coordinate. Without loss of generality, we derive the first coordinate of the fucntion:
[x[1]
20, [1][[1] +f] C2o
exp
20?
cxp
?
(a C20,)
[2[1}=
af + C2of
20z
[[1]=[1]
a
x11]2
C20 )000))
1
(a +207f)
x/[]] a + C2o
ax1(1]
of
+ +1
x[1]2
cxp
20²
20
a
Czo?
exp
20
1
(a
C20
[211]
af C20?
a+r[1]
exp
2( + C2
C2
The other d₂ coordinates abide by the same derivation.
65

---

Published as a conference paper at ICLR 2025
Next, we rewrite the the exponential with product of all coordinates by
C211=112
| a1=1l
exp
exp
2
20
exp
(a + C20
ax2 a + C20?
(2)
exp
C2 C207) [m(b
(I.2)
20
2(a
Define āg ar/az C2az and o of/0i + C2of. Next, we plug (1.2) into (I.1) and write
po(xly)
1
C211a112
+ C207)
/ f(z,))
2 (aî + C207t
a+
l²) dz,
(2m) cxp
2(a +
as + C207
C2l1ø112
/ f(2,)) exp
||2 0r2l1
(2 exp
+ C207)
dz.
(1.3)
2(a +
2
Last, we define h(,, y. J (2 f(, exp
||2 -|==
dz and plug it back to (I.3).
20;
This completes the proof.
Next, we provide lemma that provides bound on h(a, y, () and Vh(x, v,4) in Lemma I.1
Lemma L.2 (Lemma B.8 of (Fu et al., 2024b)). Under Assumption 3.2, we have the following bounds for hx, v,4) and ds Vh(x, v, the
C S b. SB,
Vh(z, y.t)
S VV++
where C₁ and the hyperparameters H [0. 11 B) in Assumption 3.2.
Remark 1.1 (Bound on and Vh). We reiterate that Lemma 1.2 drives the key distinction between the analyses in Theorem 3.1 and Theorem 3.2. Specifically, in Appendix H.3, the decomposed term containing the threshold Elow results in lower approximation rate, while bounds on and V eliminate the need of the threshold with h's lower bound C₁, rendering faster approximation rate.
This step parallels Lemma H.2; however, the discretization differs due to the structure of h.
Lemma L,3 (Clipping Integral, Lemma B.10 Fu al. (2024b)). Assume Assumption 3.2. Consider any integer vector x E z with ||<|]] < n. There exists a constant C(n,, dz), such that for any x E dd and < e S 0.99, holds
Jen i.B
arzo - x )1 0z
| x=|² 2
:((0ol)) () v) (2m) exp
doo Se,
(I.4)
where @
6.50 x12] -(2)) (( (0م00\1)) - 6z
( -x(del ) and oe
6e
B = fa C(n, djō, co ōix + C(n, djō, / loge-11 -d
Step 2: Approximate h and Vh with Polynomials. Similar to the construction of the diffused local polynomials, the following two lemmas render the first step approximation for h(x, y, and Vh(r, v, that captures the local smoothness.
66

---

Published as a conference paper at ICLR 2025
Lemma 1,4 (Approximation with Diffused Local Polynomials, Lemma B,4 of (Fu et al., 2024b)). Assume Assumption 3.2. For sufficiently larger N > 0 and constant C2, there exists diffused local polynomial f1 y. with at most Nd+dy (d + dy) monomials such that
(I, ys b(,, y, t)l - BN log N,
for any E V N, Vog N7 y [0, 1j and 0.
Lemma 1,5 (Counterpart of Lemma I.4, Lemma B.6 of (Fu et al., 2024b)). Assume Assumption 3.2. For sufficiently larger > 0 and constant C2, there exists a diffused local polynomial f2(2, y, t) E Thear with at most Nd,+d, (d.z + d monomials [ar y. such that
(2 Vi a
0
[8] s BN log - N.
(x.
for any E 1d4, y [0, 1]4 and t > 0.
Approximation of the Score Approximator with Transformers We apply the universal approxi- mation of transformers to approxiamte score approximator constructed with fi and f2.
Lemma L66 (Approximate the Score Approximator with Transformers). Assume Assumption 3.1. Let Cz(d2. 3, C1, C2) be positive constant. Let fa be the target function:
6. f2(2,,,1) fs ō. fi(
Cza
C20?
Then, for any E (0, 1), any x E [-CoVlog N, C. Coglog any y E [0, 1jd and any t E [N-Ca, C. log N), there exists a transformer g(r,y,1) E Thhis"" such that
N 10( ys v, = 2
Furthermore, the parameter bounds in transformer network function class follow Lemma H.5. We introduce the counterpart of Lemma H.6. It is the of the proof for Theorem I.1.
Lemma I.7 (Score Approximation with Transformer). Assume Assumption 3.2. For sufficiently large integer N, there exists a mapping from transformer Tacore E &Th Thh.s.r such that
C2x 12 S o? / ||Teooe V Vlogh(z,y,t) + + +2o?
B² N N-28 (log N
for any x E [-Cz, Tog N, C. Tog N [0, 1] and E [N-C, C2 log NJ. Furthermore, the parameter bounds in transformer network function class follow Lemma H.6.
Proof. Our proof follows the proof structure of (Fu et al., 2024b, Proposition B.3).
We establish the the first-step approximator fa with the form:
a, fe(x,y,t) o. fI af C2a7
Cz
fo(
We derive the error bound the approximation of the first term containing Taylor polynomials in fa- We incorporate second term containing the linear function in x into the the transformer architecture.
We proceed with the following two steps.
Step A. Approximate Scroe Function with fa-
67

---

Published as a conference paper at ICLR 2025
We first construct fi fi y, and f2 t) from Lemma 1.4 and Lemma 1.5 approximate (x,, y, t) and Vh(x, y, t) respectively.
From Lemma I.2, we have C I S B and Lemma I.5, we select a sufficiently large N such that Without loss of generality, we begin by bounding the first coordinate of Vh, denoted as Vh(1):
Vh C2
S v B. Next, by Lemma I.4 and
20 S
f S 2B and f2 S B.
Vh[1 ār Vh[1]
12
0t fi
4h[1
ãe
f2[1]] fi
f1
fi
6,
[h[li]
ā
6
[i hl
12
[
h fi
24
[/1 M|
5/
(By bounds on Vh, f1. f2)
24
. 7 + BN 8 (log N =
(By Lemma 14 and Lemma L.5)
0,
1
(BN (BN- N )
VA
at
Note that in the last line, we utilize
ãt at
1
o, oi Vot + C2o7
/ (ar/an)2 σf V C2 I-0
By the symmetry of each coordinate in Vh, obtain the foc bounds:
Vh(z,y,t) â fe(.. 1( o, fi
B L - 3 at
<
(1.5)
Step B. Approximate fa with Transformer Tsewre We use transformers to approximate fas to an accuracy of order N such that it aligns with the error order in (H.4). By Lemma 1.6, we have
f [ITTaanddr. (x, v. fa(x, v. 1)]ddr s 2
By setting = N-9, it holds
Pi (2, t) - V lo (xlp))
2
$ / pi ITToooe (z, v, fa y, 1) pellfs(z.110t) v log (o
(By triangle inequality)
/ Vaccoro v) - fala, u. 1) + / pillfa(a,,,)) - v (xlv) [gde
(x [0, 1])
B²
N 28 NN (log Ny Idv
[ (5, Ja(x, v, 1)
(By (L.5)
o7
/ [|Tacoro (x. v1 fa(x, v. B)
B²
N- 20 (log N) 4 thit1
(By |o]|oo S C. Co
ot
68

---

Published as a conference paper at ICLR 2025
B² of
23 (log N) 2/4 +h++1
By Lemma 166
Furthermore, the parameter bounds in the transformer network class follows Lemma H.6.
This completes the proof.
1.2 MAIN PROOF OF THEBRRM 3.2
Similar to the proof of Theorem 3.1, we implement the truncation due to the unboundedness of æ.
Lemma 188 (Lemma B.2 of (Fu al., 2024b)). Assume Assumption 3.2. For any R3 1, it holds
J 11
(r(r)))dr S R3 exp(-CC2
Moreover, it holds
{ 21g
R3 expp( (xxp((CC $ of
where C2 = (2 (2))
Unlike results under Assumption 3.1, the explicit form of p. (xly) (I.1) along with the upper and lower bound on the joint distribution Lemma I.2 allow us to skip the threshold clow in Lemma H.8.
We state the formal version of Theorem 3.2.
Theorem L.1 (Score Approximation, Formal Version of Theorem 3.2). Assume Assumption 3.2. For any precision parameter < e < and smoothness parameter > O, let e S O(N for some N E N. Let C. and C. be some positive absolute constants. For any y [0, 1] and t E [N- Ca log N), there exists Tacoro (z. y,t) T Thiss. such that
| f) V 1oly(() pr(2l))de -
=
-
B2 N
2-
28
(log NY3+1)
Rd
0f
Notably, for O(N the approximation error has the upper bound O(E//o?). The parameter bounds in the transformer network class satisfy
CRQ KQ ((N18d+28 (log.) 1d) Cov. Ca 0(N-9); Cp, = ON' N) detsttt) Ce 0(1); CT OlVlog N/o,),
where 0()) hides all polynomial factors depending on dz, d, L,, 3, C1 Cz.
Proof of Theorem 3.2. We apply Lemma 1.7 and Lemma I.8.
Specifiaclly, we take C₂ and Rs Cz Vlog N.
28 T
Next, recall transformer parameter bounds in Lemma 1.7. We have | l S vlog N/ / Therefore,
| V log P.dx.
(By expanding & norm)
Jd
69

---

Published as a conference paper at ICLR 2025
T 1 2 log (b2) p.dx
(ITosore V log polpp pedr,
Ing
S 2/
Ing
C22
(A.1)
(A.2)
(By l bound on Thcoc and Lemma 1.7)
2 log N
B²
+ 2| V p.dx +
N-
N-23
23 (log N)
11 100 >
24 log N
0
o
(By Lemma L8)
Vlog N
2B
2
2/
3/
B
< 2d of
log N
N-28 + of C2
log N
N-23
+
oo
N
23 (log Ny
C2
σi
(By dropping lower order term)
B²
of -.
N
28 (log N
This completes the proof.
70

---

Published as a conference paper at ICLR 2025
J PROOF OF THE ESTIMATION RESULTS FOR CONDITIONAL DrTs
Overview of Proof Strategy of Theorem 3.3.
Step Preliminaries. We introduce the mixed risk that accounts for risk with the distribution the mask signal in Definition J.1. We restate the loss function and the score matching technique in Definition J.2.
Step Truncate the Domain of the Risk. We truncate the domain of the loss function in order to obtain finite covering number of transformer network class. Precise definition of the truncated loss function class is in Definition J.4. We bound the error from the truncation from the assumed light tail condition in Lemma J.1.
Step 2.
Derive the Covering Number of Transformer Network. We introduce the covering number of given function class Definition J.5. We provide lemma detailing the calculation of the covering number for transformer architecture in Lemma J.2. We derive the covering numbers under the respective parameter configurations for our two previous main results in Lemma J.3.
Step 3. Bound the True Risk on Truncated Domain. With the previous steps, we present the upper-bound of the mixed risk in Lemma J.4.
Overview of Proof Strategy of Theorem 3.4. We decompose the total variation into three compo- nents and we bound the separately.
Step 1. We bound the total variation distance between the true distributions evaluated at = 0 and carly-stopping time f to-
Step 2. We bound the total variation between the true distribution at to and the reverse process distribution using the true score function.
Step We bound the total variation between the reverse process distributions using the true and estimated score functions at to.
Organization. Appendix J.1 includes auxiliary lemmas for supporting our proof of Theorem 3.3. Appendix J.2 includes the main proof of Theorem 3.3. Appendix J.3 includes auxiliary lemmas for supporting our proof of Theorem 3.4. Appendix J.4 includes the main proof of Theorem 3.4.
J.1 AUXILIARY LEMMAS FOR THEOREM 3.3
Step O: Preliminary Framework. We evaluate the quality of the estimator sw through the risk:
R(sw)= = /"² T- to
Er Em.. ||s (/wwwZz (Z,, t) - V log pr (2r
(J.1)
Definition J.1 (Mixed Risk). The risk (J.1) considers label y throughout whole the diffusion process.
We refer it the conditional score risk. In contrast, have the mixed risk R., that accounts for the distribution of the mask signal r = (0, id} with P(r = 0) = P(r = id) = 0.5:
Rm (sw) = ''
T 1
E [[s (2,, Ty, t) - log (x dt,
(J.2)
T
Remark J.1. Given score estimator trained from empirical loss (F.8), the conditional score risk is upper-bounded by twice of the mixed risk. That is, have R(3) < 2R, (3) by observing
T
1 to
Ry
E [| 0, t) Vlogp( dt = R(5).
71

---

Published as a conference paper at ICLR 2025
Definition J.2 (Loss Function and Score Matching), Let æ = Tz|20g denote the random variable following Gaussian distribution N(OTTO, of we define loss function and score matching loss:
((x, (3,,)88) [(-
J10 T To
1
Br [| TV, V log Pt (Irakro)llā] dt,
T 1 T 10
[(sw) =
Ero.y
Er, [[ow TW, - V (o de.
Remark J.2. Given i.d samples {zo, 3: M= we write (( y/s sw) with the understanding that
34 = Ze To,5- When context is clear, we use ((ri, Usi sw) and ((Tr, yes sw): (Fo y: Ji=1 and {2x, Vi 1 interchangeably.
Remark J.3. By (Vincent, 201 L(sw) and R. (Sw) differ by constant that is inconsequential to the minimization. Therefore, minimizing the mixed risk equivalent to minimizing the score matching loss
Definition J.3 (Empirical Risk). Consider score estimator sw E TR Recall the definition of
empirical loss: [(sw) Ei [x Mii sw). Let 8a V Po (xly), we define empirical risk:
(m = [(sw) ) - [(s²) =
[x 11 Usi (w) n
Σ
(1 Ms: n
Remark J.4. The key distinction between R.n. and L lies in their formulations. Specifically, R. measures the expected difference between sw and the ground truth V log Pa (xly) with respect to (Ir, D, r). In contrast, the score matching loss provides an explicit calculation based on the sample {2o, 94 H=z. With the tower property of conditional expectation, L measures the expected difference between Sw and V log Pt ( (20(2))) first with respect to (In T), and then with respect to Tu.
Remark J.5. Observe (I): 8 V log (x(l)) is the ground truth of score function with R., (88) = and (II): By (Vincent, 2011), R.m and L differ by a constant. Based on (I) and (II), we define the empirical risk Rm, using the score matching loss as an intermediary: (m((5)) Rm (sw) R., (8*) = L(sw) C()"). This leads to the definition of the empirical risk R.,. as a practical approximation of the true risk difference RRm (sw) Rmm(""
Remark J.6. For any score estimator sw E T h.sr obtained from the training with i.idd samples {2i, y4 12 it holds [(,,,, [Rm( (sw)) (sw). This follows from direct calculation with Definition J.3 and the i..dd assumption.
Step 1: Domain Truncation of the Risk. We define the loss function with truncated domain. This essential for obtaining finite covering number for transformer network class.
Definition J.4 (Truncated Loss). We define the truncated domain of the score function by D = [- RT, R dd [0, 11dd U 9. Given loss function f(x, yi sw). we define the truncated loss:
girmme (x, Vi sw) E(x, y: sw 11 f11=11 S RT).
(J.3)
We define Lunnn = ((sss((sW)) [||=]lx 5 RT Ritunec (sw) Rm (sw) S RT} and Rimino (sw) = (sw(s)) ( S R. }. We define the function class of the truncated loss by
S(RT) = [E(,,33w) D R sw E T The.r.
(J.4)
Next, we introduce the following lemma dealing with the error bound for the truncation of the loss.
Lemma J.1 (Truncation Error, Lemma D.I of (Fu et al., 2024b)). Consider the truncated loss girane (x, yr )w) and & E [n -(1), O(log n)). Under Assumption 3. we have [f(x, y, sw)l X 1/to-
72

---

Published as a conference paper at ICLR 2025
Consider the parameter configuration Theorem 3.1, holds:
(1)
Ez,y [| v. (xrnn v. s) 2 exp(- exp(-C2Rt) Rr to
Moreover, under Assumption 3.2, have (1 x (gg(1/ta)- Consider the parameter configuration in Theorem I.1, it holds:
-
Bx, [1 v, etrune (Ir, y. 8]] S Eexp(-C222t Ro log
10
Step 2: Covering Number of Transformer Network Class. We begin with the definition.
Definition J.5 (Covering Number). Given function class F and a data distribution P. Sample data points {Xi)== from P, then the covering number ((e. F, {X, J Il-II) is the smallest size of collection cover) C F such that for any F, there exist C satisfying
((x.) s.
max
Further, we define the covering number with respect to the data distribution
M(e, F, I1-I1)
sup
I1-11)
Next, we introduce the following lemma that provides results for the calculation of the covering number for transformer networks.
Lemma J.2 (Modified from Theorem A.17 of (Edelman et al., 2022)).
Let Th.s. T (CT. C3 Cq. C2.00 CK. C2 Cv, C2 Co. C2, ,C,, C/22 CCsy LT)
represent the class of functions of one transformer block satisfying the norm bound for matrix and Lipsichitz property for heddffowwrrd layers. Then for all data point || I Ry we have
log ( T log(nLT) a (8 (c222)) + +d
Th.s.r .1.112)
2z0) +2 (C(()))))))) 3
2
where (Cr) (o CE).
Then, we derive the covering number under transformer weights configuration in Theorem 3.1 and Theorem I.I.
Lemma J.3 (Covering Number for S(RT)). Consider 6, and || Ry. Given sample {zi, He the e--ooveiing number for S(RT) with respect to |1-Ilr.. under Theorem 3.1 and Theorem 1.1 satisfy
log N (log N) (RT)²
log N ((R)) S
2
where v1 163d + 123 and V12 20d, 43 + 18
Proof. Applying Lemma J.2, we have
(Ec, Th.s. .l:112)
73

---

Published as a conference paper at ICLR 2025
log c
a² 2
(CFC))
2,00
(dă
(c2=0)
+
2,00 (2(
22.00
kQ
(I.5)
(
(II)
(11)
where a =
Cg).
Note that we drop LT because it is inconsequential under either Assumption 3.1 or Assumption 3.2.
Step A: Covering Number of Transformer Network Configuration.
Recall that the parameter bounds in Theorem 3.1 follows
CKQ, CkQ = 0 (log N) N) 1d+2) ); Cov, COv = O(N-0);
CF, 22000 = (N (log N) d+t
O O(Vlog N/f))
Among three terms, it obvious that (III) dominates the (II) and (I). Therefore,
Cov 2,00 KQ 0 (log N) 2dz 22344 N-28
N884443
(log N) 122d
(Cp)
(Cov)²
(Okg
= O (log N) 10d2+28+8)
Therefore,
a llog(nLT) (N884463 (log N) +24x44))
logNN(es, This. :11-112) S
By Lemma J.2, we have
a
CCr) 4CkQ) CCF
2 229 (2og NJd.+842 N N480+28 (log ad CF)
(By the definition of a)
(Ce)²
(Cov)
(Oxq)
= N4, (log N) Nod.
Altogether, we have
log (nLT)
(og.N(Ee, Thisk .1l112) S
RZ 22-11166 166d44129
(log N) 20d +49416
2²
Further, by |1- 1ll0 $ $11-11. we have
log(nI(T)
log.N(c, (Tee Thee Ill (1ao)
R
16844 (log 248-1
(J.6)
2
Furthermore, the same bounds hold for results under Theorem I.1.
Step B: Covering Number under Domain Truncation.
We extend the result to the covering number for S S(Rr) defined in (J.4).
74

---

Published as a conference paper at ICLR 2025
For score estimators $1 (2, y, t, 82 The" such that ||sı s2 LOo, I by lemma D.3 in Fu al. (2024b), the difference between the loss 1,)1,88) and f(., 82) in Too is bounded by
(((),,81) E ( s2)l & elog
(J.7)
By replacing with ea/ log N in (J.6), we obtain the log-covering number
log.V (Cer (1 log(nLT)
N (log NY
2
where V1 163d 123 and V2 20d2 + +43 18.
This completes the proof.
Step 3: Bound the True Risk on Truncated Domain. We begin with the definition.
Definition J.6. Let 8 = V log (xly) denote the ground truth of score function for simplicity. Given i..d samples (x,, Ui - and a score estimator sw E T Th.s."" we define the difference function:
(sw, 8²)
E (m....).
Rimm (sw) R (sw)
Remark J.7. Note that the difference function (sw, s*) measures the expected difference between the truncated empirical risk and the truncated mixed risk with respect to the training sample. Since construct A,, (sw, s*) serving
the true risk is unattainable, derive the upper-bound on the mixed risk. Surprisingly, we are able to handle the upper-bound of the difference function, presented in Lemma J.4.
intermediate that allows to
Definition J.7. Given the truncated loss function class S(Ry), we define its e--covering with the minimum cardinality in the L metric as Lx = {L1, . }. Moreover, we define & LN with random variable J. By definition, there exist , Lx such that || (( ys sw Il S c..
Note that Lemma J.3 provides the upper-bound on the ec-covering number of S(Rr) for score estimator trained from transformer network class. Next, we bound the difference function.
Lemma J.4 (Bound on Difference Function). Consider i.i.d training samples (No,, y: |i=1 and score estimator á from (2.1). Under Assumption 3.1 and parameter configuration Theorem 3.1, holds:
An S E(E.NEE Rm (3)
1 (RTexp(CC2R)) + log.N) + 7Ec: to
+
where (Ne,, hhs (1-11)) is the covering number transformer network class. Moreover, Under Assumption 3.2 and parameter configuration in Theorem I.1, it holds:
An s0) S E (2c.0171 Rum (3) + log
1 to
[TT (RTexp(-C
1 log.N) + 7Ec.
n
Proof. this proof, we let z; := (wo vi). [(z;) == ( 8) and (z:) = garune 82);8)). For simplicity, use 1/to for the case in Theorem 3.1 and log(1/to) for the case in Theorem I.I.
Step A: Rewrite the true risk.
To derive the upper-bound bound of the true risk, we introduce a different set of i.i.d samples fa'o v/ V/ independent of the training data drawn from the same distribution.
75

---

Published as a conference paper at ICLR 2025
This allows to rewrite the true risk as:
Rm (3) Rm) = L(3) - ((s)) = Big
(( (J.8)
With (I.8), we rewrite the difference function:
Σ (((t)))(E))) (()))((())
(5,
(ene
E
(I.9)
n
Step B: Introduce the c.-ooveinng.
Before further decomposing (J.9), we introduce three definitions.
w)(z) = E)(z) - (2) and (2)) = ((2) - (2).
Ω = max
Σ
(r(z))
ISJSN
hJ max A, Eg (e)(z)) (2')l with constant be chosen later.
With hj. wy and Ω we start bounding (I.9) by writing
a (((z) ( (
(,)
E1 (x
- (()))
71
E (x
Σ - -v(=))
+ 2ec
(By Replacing Z with E.)
n
E
(a ( [krS2]] + DEa
(By introducing Ω and hJ)
71
E (= [13]] 113] (2),)) [S2a] + 2Ec
(By Cauchy- Bauchy-Schwarz inequality
E
(
[13]
E
(ane (029)
+ 2
(By AM-GM inequality)
2
2n
Bla
[13]
+
E
[022] +2Ec.
(J.10)
2n
(I))
Step B.1: Bounding (I).
By the definition of h
E E [13] < A² [ [E (x)] (=)]]
< A² + E. [W0(2') ]] + 2
= 2 + E 1 [Rimine trunc (8) + 2
(J.11)
Step B.2: Bounding (II).
By Lemma J.I, we have [e(2; sw) ) S K, and by the definition of ? we write
w)((x)) - wx(ر+) ) 27
[ (1))))
)
(r(((t) 27
E (anat
h
h
(By the independence between 2, and
76

---

Published as a conference paper at ICLR 2025
w+(
w3(21))
hJ
hJ
< 2nx.
We use the Bernstein's inequality and the following two facts (1)
( K/A.
hj
- hJ
(2)
is centered.
For any J and w 2 0, it holds
2
wr(Zi) wr(z() hj
wJ(Zi) wr(xf)
P
2
2P
Σ
v
hj
1=1
w/2
< 2xxp
v/ 2n
3A
Therefore,
2V
wr(Z)) - wr((() hJ
w/2
P ( 2 w
Σ P J=1
2 22Nexp exp
K 2n +
10
3d
For some wp > O, we bound !2 by
E (x..a)))
[2_]] = / Wa
PS2² ww) dw + /
PS2 2 w) dw,
(By integral identity)
w/2
< <wo +
2Nexp
dw,
you 2n
wo
3A
3AAW
< wo + 2N
exp
+ exp
dw,
8nK
4x
< wo + 2V
|Sn exp
Wo
18k, /wn/Ww
+
32x
exp
3.A/
8mK
3A
9.A2
4x
Taking A /wo/6nnoo and wp 8nx log N, we have
E [(
[92 S nx log...
(J.12)
Step C: Combine (I) and (II).
Combining (J.11) and (J.12), holds:
A (3, <
(=
[h]]
2n E (= [922] + 2ec
2
1
R (8)] + "
7
(z))
log N +
2 Cc.
2
2n
77

---

Published as a conference paper at ICLR 2025
Recall Definition J.6 and multiply the above inequality with 2, have
E (x.)
Rm
[$) 2 2B (=)
R
(5) +
log + T
n
Therefore,
(5, S B (ondiin
Rimun (3)
+ n
log.N~ + 7co
(By Lemma J.1)
Run (8)
+ x. (RT
Ry (R (xpp-C2RR))
1
log.N) + Tccs
n
This completes the proof.
J.2 PROOF OF THEOREM 3.3
Proof of Theorem 3.3. Recall Definition J.1. We bound the mixed risk with following three steps.
Step A: Decompose the mixed risk.
Let s" v. V logp:(xly) be the ground truth. If 0, we set s y,t) V Por).
Recall Definition J.3 and Lemma J.4.
Let (a% v (f y) be different set of i..dd samples independent of the training samples. Next, we rewrite the mixed risk
Ru 8) (a/
((14.1).3) l1do 1/. )0))
E (n
Rin (3)
72
where we use R/m (3) to denote the empirical risk of the score estimator trained from {a/, 11 Next, the decomposition of E( RRm (8) follows
E E(z.,1 1i Rm (3) E (2..01
B
Rim (3)
trunc (3)
m
E (at (m..)))
Ritrunc 33)
Perunc (8)
(II)
E (maa))
Rtrunc(3) (3)
Rm
+ E (mann 1
Ru (3)
(IIT)
(TV)
Step B: Derive Upper Bounds.
Recall Lemma J.4.
Let 1/to for results Theorem 3.1 and log(L/to) for results in Theorem I.I.
By Lemma J.1, we have both (I), (III) S nexp(- lt.,
By Lemma J.4, we have (II) S (IV) + k (Ry exp( C2RR)) + 1 " log.N + Tec.
Next, we we bound (IV) by
(TV) = E
R(3)
S B
(m)(s) Rm (8).
Therefore, (IV) < min
ew ohr (s)
78

---

Published as a conference paper at ICLR 2025
The inequality holds because $ is the minimizer of the empirical risk.
Combining these bounds, it holds
[ T to
E (m [Rm (8) S min
ER....
(u((z,TT,.) V log Vlogne dt
sw ETR to
Ki log log + O(exp( (exxp(-C2ttts)) (aeoN((OOxp(-pPPPee + (Ec).
+ o
(I.13)
Next, we take Ry=
(Co (G++29) log
C2
Since S log(1/4o) 2 1/10
NCs it holds
Rm (3)] 5
min
ET, [1s(E, - lo de
RETR
t0 T- 1o
+ O
log. + O (N 28) + (ccc..
(J.14)
To apply the previous approximation theorems (Theorem 3.1 and Theorem I.1) to the first term on the RHS of (J.13), we rewrite the expectation as
2. [|s((ee:T,.) V log (oog
(J.15)
- [s((,,0,) (o E, Rd=
S/e |s(0, v, (1gg) pd=
Since (x) satisfies the subgaussian property as well, the previous result the conditional score estimation applies to its unconditional counterpart by removing the label throughout the process.
Step C: Combine All Bounds.
First, we derive final bounds for results under Assumption 3.1.
Result under Assumption 3.1
Let 71. 72 (0,1) be arbitrary real numbers.
By taking = n and & = n ?2, we rewrite (J.14)
E [x)) Rm (3)
x O - (log
(log N) N2)++411111111 +0 (N-20)
+o nn No (log N) E-2
+ + (iv)
to
(1))
(ii))
(By Theorem 3.1)
8mm1 (log n) ga+ ( + 0
Cem
5 0 n
n n (logn)*² 2 2T2
O (n 72)
min( Pal Co 222/12)
(logn)"²
where 2 163d + 123 and 1/2 = 20d, + 4,3 + 18.
We find the optimal upper-bound by the following choice of 7 and 72.
For any 21, E (0, 1) satisfying,
7 + 27 + Co .
< 1, 11
79

---

Published as a conference paper at ICLR 2025
consider
min 871
Ca
71
22
9/2
1/1
To simplify, we set
871
Co71 Y2:
272
and hence
811
Co
1 71
272
72
V1
11
Rearranging and solving for Y1, 72, obtain
33
71
C. +3
72
Mi Cr 33
so that the three arguments in the min(.) all coincide.
Therefore,
Ca
B
min
71
222
72
= =
V1
11 + C, + 33
Next, we ensure condition 71 + 27/2 + Caom < 1.
Since
11 +
28
Co
V1 C. + 23
+
<1,
+ C. + 33 i + C. + 33 v +C, + 33
+ C. +
the condition holds for all da, and C..
Next, we derive final bounds for results under Assumption 3.2. Result under Assumption 3.2
Let Ya 74 E (0, 1) be arbitrary real numbers.
By taking N 78/// and e, n-74, 74, we rewrite (J.14)
[x Rm [$)
(By Theorem I.1)
1
(N 28 (log (log 3+1) + 0 (N-22)
23
+0
-1
No (log 22 _==
+ O (Ec)
to
(
(n)
(i0) (By dropping lower order term (ii)
50
28-73
(log 8+1 ( + O (log n n1 na (logn)'2 2244 + O (n 74)
n
50
min n
2873 1-78 27a.34) (log n) max(13,842)
where 03
4(1284, +3194468)
126, (12d, -124, + 72C
We find the optimal upper-bound by the following choice of 71 and 72-
80

---

Published as a conference paper at ICLR 2025
For any n1, 722 E (0. satisfying,
Na + 274 < 1,
we consider
2973
min
1 73 27/4 74
1/1
To simplify,
2,3 73
1
733
27/ 7/1-
V1
and get
1/1
2,3
74
V1 633
+ 6,3
with 73 + 274 < 1.
This completes the proof.
81

---

Published as a conference paper at ICLR 2025
J.3 AUXILIARY LEMMAS FOR THEOREM 3.4
We begin with following lemmas that serves the key components for the proof of Theorem 3.4.
Lemma J.5 (Lemma D.4 of Fu al. (2024b), Proposition D.I of Oko et al. (2023) and Chen al. (2022)). Consider probability distribution Po and two stochastic processes Y = {Y) (et))(( and y" = (Y/heto.1) with distribution Pt and Pú respectively, satisfying the following SDE:
dy, b(Y,, t)dt dW, Yo Po.
dY; b' (Y t)dt dW, Yo Pa-
the condition Je (x (x, l) & C. holds for any / E [o, 7] and constant C, holds
Pr 11 Pt) S 1 2. P((x)) (z, t) dzdd.
2
Moreover, need the following lemma to bound to total variation.
Lemma J.6 (Lemma D.5 of Fu et al. (2024b)). Assume Assumption 3.1. 3.1.For For any y E [o, 1]4h it holds
( log sلي2 (t) to )
TV (Po((l)).P,, =0 (vTo
J.4 MAIN PROOF OF THEOREM 3.4
Proof of Theorem 3.4. Given label y, let Peo (-19) denote the data distribution with early-stopped time to generated by the reverse process with the score estimator 3 from transformer network class. We define the reverse process starting with standard Gaussian.
1 axx Vloger w) di + dW,, N N(O, 2
ax;
(J.16)
We denote the distribution of X conditioned the label y P--l(- Next, we decompose the total variation into three parts
TV P(-1)) P.o (-18)
s TV Peo (.1v)) + TV (1o),, Pro (-1)) + TV (Pro(ll)), Pro (l)))
(III
where we introduce distribution P.o at time to intermediary defined in (J.16).
Step A: Bounding (I).
By Lemma J.6, we bound term (I) by
TV (P(-))), (-v)) = (ool))=o(Vi)) (Vo اليe (1)
)
Step B: Bounding (II).
By Data Processing Inequality and Pinsker's Inequality (Canonne, 2022, Lemma 2), we write
TV (-lv), Pro (-11) SV KL(P,, (-b) II Pa, (+b)),
82

---

Published as a conference paper at ICLR 2025
SVKL(Pr(1)) (N..,d.)))
SVKL(P(-1o) [0, Ia_)) exp(-
(1.17)
Next, bound term (II) by
TV Pho S (KL(P(.1)) |I N Ia,)) exp(-IT)
(By (1.17)
exp(-T).
Step C: Bounding (III).
By Lemma J.5 and (J.17), bound term (III) by
TV (Po) & V V.o 2 J. p((
v log (x(lv) | dadt.
By incorporating three steps and taking expectation with respect to we obtain the upper-bound
Fyy
TV
(P(1)). Pro (111))
3 V
to log
1
( 1
+ exp( + S. 2 _
p((2lv) - V log (xl)))) (xlp))l dædt,
to
(By Jensen's incquality)
d
voo log $
+ exp( -T)
+
R(
2
(ii)
(i)
(iii)
Result under Assumption 3.1.
Recall Step C.1 in the proof of Theorem 3.3. We set N = n// and e = n for all m, E (0 satisfying 71 + 2x2 + Ca m
< 1.
Therefore,
Ey TV (P(-)), Peo(-11))
N log dett (N (N-Co) C.
(By Theorem 3.3)
ii
(ii
Ca log N min d:1 11 272
Ca m22
+
(log n us+2
(iii)
4p m
du (n)
Guen
<
201
log
(By de +1)
(i)
(ii)
min 8-71
m 27
(log n) 7
+ n
(ii)
S n
(log n) 7
83

---

Published as a conference paper at ICLR 2025
where w min
C. 2v1
8-71
1 7/1 272
C.
72
Moreover, recall that the following choice of y1 and 7/2 leads to the tightest upper-bound
V1
3
71
72
+ C. + +38"
1 + Co 33
Therefore,
C
Co
8
min
V1 + C. + 33 2(1 + Co + 33) vs Co + 33"
This completes the first part of the proof.
Result under Assumption 3.2.
Recall Step C.2 in the proof of Theorem 3.3. We N 7//// and &g 7 for all 73. Ya (0, satisfying 73 + 274 1.
Therefore, by Theorem 3.3,
By TV P(-lu), Pro (-wl))
Aut1 (N Cn (N-Co) + N-C. Co
2 N log
min 20 ,1-79 274.74 (log n max (13,6++)
+
Co log N n
C 2r1
Axtt (n)
llog
(ii)
min
20
73 24) (logn)*
+ n2
(m)
S n 0 (log 1)
1 7+
(J.18)
where Φ min
Coina 2n1
Cuina
28
73
-244, 74
P1
Moreover, recall that the following choice of yts and 7/4 leads to the tightest upper-bound
28 V1
73
7/
V1 6,9
63
Therefore,
C.
C.
28
= min
2( + 63)
+ 63 v1 + 63
This completes the second part of the proof.
84

---

Published as a conference paper at ICLR 2025
J.5 PROOF OF THEOREM 3.5
Recall Lemma 3.1, we have the minimax optimal rate
inf sup B( [TV(2, 2 Q(n 2= ((nt) pEP
Further, recall Theorem 3.4, we have
By TV (Poc = O 0m (log (n=0(1ogm) (++)
E (e
My+ +1+ where min 2[1 +63) Mi +pp minimax optimality under the setting C. = 2Ca 2,3 and +65 d. +23
8
}. Therefore, unconditional diffusion transformers achieve
This completes the proof.
85


Reviews:
[{"review_id": "R6JbW80gMP", "reviewer": "Reviewer_WTrU", "summary": "This paper studies the statistical rates of approximation and estimation for conditional diffusion transformers (DiTs) with classifier-free guidance. It provides a comprehensive analysis under four common data assumptions, showing that both conditional and latent variants of DiTs can achieve minimax optimality. It uses a modified universal approximation of the single-layer self-attention transformer model to circumvent the need for dense layers, which results in tighter error bounds for both score and distribution estimation.", "questions": "**1.** On the top of page 3, should the denominator of $\\nu_3$ be $d_0+d_y$ instead of $d_x + d_y$?\n\n**2.** The bound in Theorem 3.1, the first bound of Theorem 4.1, and the first bound of Theorem 4.2 seem to depend exponentially on $d_x$ or $d_0$, which seems to suffer from CoD. Is the bound tight here?\n\n**3.** Is there any practical example that matches the assumption 3.2? If so, I’d suggest adding a brief paragraph to discuss this.\n\n**4.** In the first bound of Theorem 3.3 and Theorem 3.4, the bound seems to be decaying in a rate of $exp(-d_x)$, is this a typo?\n\n**5.** The second bound of Theorem 3.3 seems to be not sensitive to $d_y$. The second bound of Theorem 3.4 depends on $d_x$ and $d_y$ only through $d_x+d_y$, and is not sensitive to it. Similarly, the second bound of Theorem 4.2 only depends on $d_0$ and $d_y$ only through $d_0+d_y$. Is there any intuition behind this? \n\n**6.** There aren't any empirical results in the current version. I’d suggest adding a few practical examples to illustrate the validation of the bounds.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is original in its theoretical approach to studying conditional diffusion transformers, offering new insights into their performance and limitations. The quality of the theoretical analysis is high, with clear outlines and complete proofs. The clear and thorough exploration of statistical limits and estimation procedures is a key strength.", "weaknesses": "1. Some results seem to suffer from the curse of dimensionality, which seems not a practical bound.\n2. There is an absence of empirical results to complement the theoretical analyses.\n3. More practical implications of the assumptions could be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YKhvBWaE6T", "reviewer": "Reviewer_mSoC", "summary": "The paper studies the statistical understanding of the conditional diffusion transformers (DiTs) with classifier-free guidance. The authors focus on understanding the approximation and estimation rates of these models under various data conditions, particularly in \"in-context\" settings. They show that both conditional DiTs and their latent variants can achieve minimax optimality, similar to unconditional DiTs, in certain scenarios. The analysis is based on discretizing input domains into small grids and applying Taylor expansions to the conditional diffusion score function, assuming data follows Hölder smoothness. This approach allows for a more refined piecewise approximation, leveraging transformers' universal approximation capacity. The study also extends to latent conditional DiTs, showing that these models offer better approximation and estimation bounds than standard conditional DiTs, especially under linear latent subspace assumptions.", "questions": "In addition to the weaknesses outlined, please consider the following:\n\n- Can the work be extended to a manifold setup where the rates depend on the intrinsic dimension of the data, similar to the approach taken by Oko et al. (2023)?\n\n- Regarding the assumptions made, if one has a function with significantly less restrictive conditions than those presented in this study, what advantage does using a transformer provide over traditional statistical methods for achieving a minimax rate? This question is particularly relevant from a theoretical perspective, which is the primary focus of this paper.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a detailed theoretical framework for analyzing the approximation and estimation rates of conditional diffusion transformers (DiTs) under various data assumptions.\n- The work integrates classifier-free guidance into the analysis of conditional DiTs.\n-  The paper provides rigorous results on score approximation and distribution estimation, including sample complexity bounds for score estimation which lead to minimax optimal estimation results.\n- The work integrates classifier-free guidance into the analysis of conditional DiTs.\n- Beyond theoretical contributions, the findings offer practical guidance for configuring transformer-based diffusion models to achieve optimal performance.", "weaknesses": "A general weakness of this paper is its reliance on a lengthy and detailed appendix, which is not mandatory for reviewers to read. While the results are strong and represent a valuable contribution to the field, the extensive appendix makes it challenging to validate the findings within a limited review timeframe.\n\n- The analysis relies significantly on Hölder smoothness assumptions, which may not apply to all types of data. This limitation restricts the generalizability of the theoretical results to datasets that satisfy these specific conditions. Additionally, the assumption of isotropic smoothness across input variables does not always hold, particularly in cases where the condition is discrete or irregular.\n\n- The study does not address the high-dimensional structural dependence of the data. It is overly restrictive to assume that the ambient dimensions of practical data distributions are small and that the rates of dependence relative to the ambient dimension are sufficiently slow.\n\n- The paper lacks empirical validation that connects the theoretical rates and dimensionality to practical datasets. For instance, it would be beneficial to clarify what smoothness refers to in the context of the given dataset. Furthermore, it is unclear how one might establish practical relevance for the proposed rates and validate them empirically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ofl6rX5UxU", "reviewer": "Reviewer_GBgr", "summary": "This paper provide a theoretical analysis for statistical rates of conditional diffusion transformer, mainly focusing on the score approximation, score estimation and distribution estimation.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, which provides detailed theoretical analysis and clear explanations. Additionally, the main theorems are presented concisely and effectively, and the discussions in remarks are particularly helpful for interpreting the results.\n2. Comparing with the previous work ([1]), the theoretical analysis alleviates the double exponential factor and achieves minimax optimal statistical rates for DiTs under Hölder smoothness data assumption.\n\n[1]. Hu J Y C, Wu W, Li Z, et al. On statistical rates and provably efficient criteria of latent diffusion transformers (dits)[J]. arXiv preprint arXiv:2407.01079, 2024.", "weaknesses": "1. As a follow-up work of [1], maybe the contribution mainly focuses on polishing and extending the previous results, which may lead to a lack of novelty. Could you provide some new insight points about it?\n2. With the good rates theoretically,  could this work provide some implementations in practice?\n\n[1]. Hu J Y C, Wu W, Li Z, et al. On statistical rates and provably efficient criteria of latent diffusion transformers (dits)[J]. arXiv preprint arXiv:2407.01079, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fdek1DnkHt", "reviewer": "Reviewer_1NAW", "summary": "This paper mathematically investigates the approximation and estimation properties of conditional diffusion Transformers (DiTs) with classifier-free guidance. Theoretical connections between conditional DiTs and their unconditional latent variants are established for data distributions with certain smoothness.", "questions": "1. For the score approximation: \n- $\\sigma_t^2 \\to 0$ when $t \\to 0$, leading to infinitely large upper bounds (Table 1).\n- $\\epsilon^{1/d} \\to 1$ when $d \\to \\infty$, leading to $O(1)$ upper bounds (Table 1) that are vacuous in high-dimensional settings. \n- The upper bound is *exponentially* large given high-dimensional inputs, due to the multiplicative term $(\\log N)^{d_x}$ (Theorem 3.1). \n- The parameter bounds of Transformers are also *exponentially* large given long input sequences (with fixed dimensions), since $\\||W_{\\{Q,K\\}}\\||=O(N^{L/d})$ (Theorem 3.1). \n\n2. For the score estimation: \n- In Theorem 3.3, why does the upper bound deriving from Assumption 3.1 (weaker) have a much better dependence of $\\log n$ ($(\\log n)^{-dL}$ v.s. $(\\log n)^{\\Omega(1)}$) over the upper bound deriving from Assumption 3.2 (stronger)? Also, the former error term $(\\log n)^{-dL}$ decreases exponentially fast as the data dimension or sequence length increases, which seems unreasonable. \n- In Theorem 3.3, for the upper bound deriving from Assumption 3.2, $\\nu_3=O(L/d)$, which gives $N=n^{\\Omega(d/L)}$, $t_0 \\sim O(n^{-d/L})$ and $\\log (1/t_0) = \\Omega(d/L\\cdot\\log n)$. This term can be merged into the subsequent polynomial dependences in data dimensions and sequence lengths (i.e. $d^{14}L^4$), and hence introduces no singularity. However, the power of $1/n$ is approximately $1/(\\nu_3\\cdot d)=\\Omega(1/L)$, leading to the vacuous sample complexity estimate given long input sequences. \n\n3. For the distribution estimation: \n- In Theorem 3.4, there are the same issues raised in the second point of the score estimation, since the upper bounds in Theorem 3.4 and Theorem 3.3 are in similar forms. \n\n4. It seems that there is no characterization of the model capacity dependence. That is, all the provided estimates do not contain (at least, explicitly) error terms related to $h, s, r$. \n\n5. Minor issues:\n- The notation $s$ denotes both the head size and Hölder smoothness (Definition 3.1). \n- In the caption of Table 1, \"... where $L$, $\\tilde{L}$ are sequence length of transformer inputs\", length ->lengths.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper seems mathematically sound. \n2. The settings studied in this work are abundant and quite complex. \n3. The roadmaps of proofs are clear.", "weaknesses": "1. It seems that some estimates scale poorly in the asymptotic sense, particularly in high-dimensional and long sequences settings (see details in the \"Questions\" section). \n2. Without verifications by numerical simulations, one cannot guarantee that the derived approximation and estimation estimates can be achieved in practice, i.e. whether these theoretical constructions can be found through concrete training dynamics. Is it possible to numerically verify (partial of) the dependence in these mathematically derived upper bounds?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)