How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Authors: Jing Qiao
Yu Liu
YUAN YUAN
Xiao Zhang
Zhipeng Cai
Dongxiao Yu
Venue: ICML 2025 poster (2025)
Keywords: distributed diffusion models
generation error bound
Primary area: theory->everything_else

Abstract: This paper examines the theoretical performance of distributed diffusion models in environments where computational resources and data availability vary significantly among workers. Traditional models centered on single-worker scenarios fall short in such distributed settings, particularly when some workers are resource-constrained. This discrepancy in resources and data diversity challenges the assumption of accurate score function estimation foundational to single-worker models. We establish the inaugural generation error bound for distributed diffusion models in resource-limited settings, establishing a linear relationship with the data dimension $d$ and consistency with established single-worker results. Our analysis highlights the critical role of hyperparameter selection in influencing the training dynamics, which are key to the performance of model generation. This study provides a streamlined theoretical approach to optimizing distributed diffusion models, paving the way for future research in this area.

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Jing Qiao
Yu Liu
Yuan Yuan 34
Xiao Zhang' Zhipeng Cai Dongxiao Yu'
Abstract
subsequently learning to reverse this perturbation to gen- erate new samples (Yang et al., 2023; Song et al., 2020), leading to notable advancements across fields like computer vision (Harvey et al., 2022; Saharia et al., 2022), natural language processing (Austin et al., 2021; Li et al., 2022), and temporal data modeling (Lopez Alcaraz & Strodthoff, 2023; Tashiro al., 2021).
This paper examines the theoretical performance of distributed diffusion models in environments where computational resources and data avail- ability vary significantly among workers. Tradi- tional models centered on single-worker scenarios fall short in such distributed settings, particularly when some workers are resource-constraine. This discrepancy
Theoretical research on diffusion models has traditionally been framed within the single-worker paradigm, yielding substantial progress in defining polynomial error bounds. These studies generally use measures such as total variation distance or KL divergence to quantify the discrepancies between true and approximate paths. Notable advance- ments include using Girsanov's methodology to achieve error bounds that scale linearly with data dimensions under early-stopping conditions (Benton et al., 2024) and adopting stochastic control perspectives to derive comparable bounds under specific smoothness conditions (Conforti et al., 2023). These insights significantly aid single-worker training of diffusion models.
resources and data diver- sity challenges the assumption of accurate score function estimation foundational to single-worker models. We establish the inaugural generation error bound for distributed diffusion models in lomotee settings, establishing a linear re- lationship with the data dimension d and consis- tency with established single-worker results. Our analysis highlights the critical role of hyperparam- eter selection in influencing the training dynamics, which are key the performance of model gener- ation. This study provides a streamlined theoreti- cal approach to optimizing distributed diffusion models, paving the way for future research in this area.
However, the trend towards geographically dispersed data sources makes centralizing data for processing not only costly but also privacy-invasive prompting shift towards distributed diffusion model training (Tun et al., 2023). De- spite this practical shift, thcoretical exploration remains largely anchored in the single-worker setup, leaving the impacts of distributed training on diffusion models largely unexplored and not well understood (Li et al., 2024).
1. Introduction
Diffusion models have significantly challenged the promi- nence of Generative Adversarial Networks (GANs) (Good- fellow et al., 2014) and have become a central focus in the realm of data generation (Yang et al., 2023; Song & Ermon, 2019; Dhariwal & Nichol, 2021). These models operate by introducing noise to data through forward process and
To effectively develop theoretical insights for distributed dif- fusion models, several critical questions must be addressed: (1) How to facilitate distributed collaboration in diffu- sion model training? Privacy preservation is paramount, as diffusion models often handle sensitive data like sound and images. Traditional methods that distribute fragments of data for collaborative training do not adequately safe- guard privacy, rendering them unsuitable (Li et al., 2024), Furthermore, the resource variance among workers com- plicates participation; low-resource workers may struggle with full model training, potentially leading to an unfair model. Simultaneously, accommodating slower workers could introduce delays, known as the straggler problem, thus hindering overall training efficiency. Balancing inclu-
!School of Computer Science and Technology, Shandong Uni- versity, Qingdao, China 'Zhongtai Securities Institute for Financial Studies, Shandong University, Jinan, China SScoool of Software, Shandong University, Jinan, China 4 "ointt SDU-NTU Centre for Artificial Intelligence Research (C-FAIR). Shandong University, Jinan, China 'Deparmment of Computer Science, Georgia State University, Atlanta, GA, USA. Correspondence to: Dongxiao Yu <dxyu@ sdu.cdu.cn>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
sivity and efficiency is therefore essential for successful distributed collaboration. (2) How to evaluate the genera- tion quality of collaboratively trained diffusion models? The collaboration of multiple workers, each with differ- ent resource constraints and data heterogeneity, introduces unique challenges in assessing the theoretical performance of diffusion models. Limited resources might force some workers to engage in sparse training, potentially accumu- lating errors that degrade model performance. Moreover, variations in the original data distributions among work- ers can cause inconsistencies in the quality of data gen- erated by the unified model, complicating the application of traditional single-worker evaluation metrics like accu- rate score function estimation (Benton et al., 2024; Chen et al.. 2023b:a). (3) Can appropriate hyperparameters be selected to optimize generation quality? Certain hy- perparameters, including learning rate and noise scheduling, critically influence the generation quality of diffusion mod- els. For example, noise scheduling plays a pivotal role in the denoising phase, directly affecting the quality and di- versity of the generated outputs. Therefore, identifying and adjusting these hyperparameters is crucial to maximize the efficiency and effectiveness of the diffusion model.
noise scheduling, while adapting varying resource constraints via local sparse training.
To the best of our knowledge, this is the first time a generation error bound has been established for dis- tributed diffusion models under resource limitations, which matches the best-known results of the single- worker paradigm (Benton et al., 2024) and shows a linear relationship with the data dimension d.
We detail strategic hyperparameter adjustments, such as noise scheduling and learning rate optimization, to ensure that the generation error bound is chiefly in- fluenced by the dynamics of distributed training, thus enhancing overall model performance.
2. Related Work
Diffusion models have recently risen to prominence due to their exceptional performance in various domains such as computer vision (Harvey et al., 2022), natural language pro- cessing (Li et al., 2022), and multi-modal learning (Ramesh et al., 2022), challenging the once-dominant Generative Ad- versarial Networks (GANs) in terms of both stability and efficiency of generation (Dhariwal & Nichol, 2021).
In this paper, we present the first theoretical performance evaluation for distributed diffusion models under resource- constrained scenarios. Initially, to maintain data privacy, each worker independently processes the noising and de- noising stages using locally scheduled discrete-time noise, learning a parameterized score function that is shared with central server. Specially, we allow for sparse training tailored to each worker's available resources, enhancing computational efficiency significantly. To evaluate the gen- eration quality of the models, we utilize Girsanov's 's theorem to link discrepancies between ideal and actual data distri- butions to factors such as time discretization, distributed training dynamics, and equivalent loss substitution. For distributed training dynamics, we manage sparse training er- rors through coordinate-wist model aggregation and bound the impact of data distribution heterogeneity with auxiliary functions. Additionally, we calculate the expected devia- tion in the drift terms caused by time discretization and demonstrate that score matching during the denoising phase incurs consistent numerical error. Lastly, by carefully adjusting hyperparameters, including the learning rate and the intervals of discrete time, ensure that the generation error bound is predominantly dictated by the dynamics of distributed training, thereby elevating the overall efficacy of the diffusion models.
Theoretical investigations into diffusion models have deep- ened our understanding of their mechanics, particularly re- garding convergence rates, stability, and data generation quality. Initial theoretical explorations required stringent assumptions about data distributions, such as compliance with log- Sobolev inequalities (Yang & Wibisono, 2022), and often resulted in either non non-quantitative bounds (Pid- strigach, 2022) or exponential dependency on problem pa- rameters (Block et al., 2020). Recent advancements have mitigated these limitations, with studies like those by Chen et al. (2023b) achieving polynomial error bounds in total variation distance without restrictive data distribution as- sumptions, using the Girsanov change of measure to quan- tify discrepancies between true and approximated reverse processes. Further progress by Chen et al. (2023a) refines this approach by expanding the Girsanov methodology and introducing pivotal theorems that address the behavior of KL divergence in relation to data dimensions and the impacts of carly-stopping on егтor magnitudes. Moreover, Benton et al. (2024) have significantly advanced this line of work by establishing state-of-the-ar error bounds that are linear in the data dimension, even in the absence of smoothness assumptions on the data distribution.
Despite these theoretical advances, the focus remains pre- dominantly on single-worker setups. The shift towards dis- tributed training, as seen in works like those by Zhou et al. (2024) and Lian et al. (2017), highlights the evolving land- scape in response to the scalability demands of big data. Tun et al. (2023) explore the FL strategy to train diffusion
The core contributions can be summarized as follows:
We propose an innovative training mechanism for dis- tributed diffusion models that prioritizes data privacy by keeping private data localized through synchronized
2

---

How Distributed Collaboration Influences the Diffusson Model Training? A Theoretical Perspective
models, paving the way for the development of federated diffusion models. Stanley Jothiraj & Mashhadi (2024) intro- duce Phoenix that integrates various strategies to enhance the diversity of generated samples, even when the training data exhibit statistical heterogeneity. DistriFusion (Li et al., 2024) exemplifies this trend, facilitating the parallel process- ing of diffusion models across multiple devices to reduce lateney in sample generation without sacrificing quality.
where (B,, (1 is the standard Brownian motion on R. Ideally, the process (Nn,))e(o. can thus generate samples from the distribution In,0 by sampling Yn,0 9n,7.
For each worker-n (n 1, 1, ,N, the score function v log ( described in equation (3) is not directly accessible, necessitating the use of an estimated function se (Y T - t) to approximate V Vloggn, Tr.TT--Yy through- out the interval E [0, T7.
However, a theoretical understanding of how distributed dif- fusion models perform under resource constraints remains underexplored, indicating a crucial gap in the literature that future research needs to address. This presents an oppor- tunity to extend the robust theoretical groundwork laid by single-worker studies to more complex, distributed environ- ments, potentially unlocking new efficiencies and capabili- ties in diffusion model applications.
Given that equation (3) outlines continuous- process, practical implementation requires discretization of the time variable. This is achieved by segmenting the continuous timeline into a series of discrete intervals 0 = to < t1 < (/ <-... .... < l S T. The process begins with sampling Yn, from the distribution 9n,7, followed by sequentially solving the SDE, also referred as the exponential integra- tor (Zhang & Chen, 2023; Bortoli, 2022; Chen et al., 2023a), for each interval [|x, [x+1] where k = 0, ......... K - 1.:
3. Training Description
dyn,, = (Yn, + 250 (Ynsto T [x)] + V2dBa,t
In our framework for training distributed diffusion models, we employ a setup that includes N workers and a central server. For each worker-n (n = 1,.... .N) the initial phase is to progressively transform the given data distribution 4n,0. into known prior distribution. This is referred to as the forward process, and it can be described using the Ornstein- Uhblenbekk (OU) process via the stochastic differ- ential equation (SDE) (Pedrotti et al., 2024):
where (Bm,))e(ooTT is a standard Brownian motion. We denote the length of the k-th discretized time interval by 7k = - tk-
Considering the potential constraints the quality and quantity of training data available to each worker, our objec- tive is to leverage the collective capabilities of N workers to collaboratively train a score function estimator (. Typ- ically parameterized as a neural network, this function is defined by a parameter vector 0 E IR.. Our primary goal to optimize the performance of this model by minimizing the following loss function:
dX,,tt = -X dt + 2dd Xn,o ~ 4n,0
where (Bn,))efoor7 denotes a standard Brownian motion on Rd. Equation (1) aligns with a methodology known as Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020), and is also referred to as Variance Preserving SDE in (Song et al., 2020). The OU process is favored for its analytically tractable transition densities, and it holds that Xn,e| Xn,0 ~ N',,n, -f, 1 e - La)).
(0)
(4)
n=1
where the local objective (.) is denoted as (,)(
=
K Bll V Vlog - (Yn,aa) so Sa T - (x) [1a).
We use In, (Xn,), E [0. to denote the marginals of the forward process for each worker-n (n = 1,... NN) and then the ideal reverse process satisfies the SDE:
To address the unavailability of the score function V log (Yn.te) we replace (0) with an equivalent objective (,)))) using denoising score matching (Vincent, 2011). The equivalence of the two is given by Lemma 4.8.
[Xx,,= = (XXn, + 2V +VV 4x, + VddBn,t
X 9n.0
And (0) is expressed as follows:
where (Bn,)teja is another standard Brownian motion on Rdd. Aligning with the idea of reconstructing the data distribution from noise, the reverse process (2) can be trans- formed to a forward one by inverting the time direction L with T - t and setting Xn,t = Yn.-ii
K-1 Σ
Blll so (Y T - tx) - V
[|2]
(5)
Due to the inherent randomness in sampling during train- ing, we introduce a stochastic local loss function, denoted as fn (0, E.). Specifically, we assume that fn &..) is un- biased, which is a common assumption in distributed set- tings (Lian et al., 2017), meaning that (0, = (0).
dy = (Yn, + 2V log ( + V2aB'r.
Yn, ~ Tn,T
3

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Worker-N
...
Worker-2
Server
Obtain O for data generation
Worker-1
FORWARD
1. Receive the global model perform local prosing
0,100
Eg
5. Coordinate-nise update
t1
Ps.0
41. t
R.Rndomly sample
training with pruned model
O,,55
T-8 {K
Local pruned model transmission
Pite REVERSE
0ss
O,5s
Figure I. The illustration of distributed diffusion model training with pruning.
With this formulation, our objective can be rewritten as the following optimization problem:
Model Update and Synchronization: After S steps of local training, each worker-n obtains the final pruned model parameters f, s in the round r. These param- eters are then sent back to the server to update and synchronize the global model. For each coordinate 1,2. D, the parameter is updated
min F(0) BERD
Σ (0,
(6)
FF,00)
0
Given the potential constraints and variability in available resources across workers, enable each worker to train only subset of the full model parameters. More specifically, the learning process in round for achieving (6) can be described in detail as follows:
(N
mmn
(8)
DEN
where N²(1)
{n m 1 and we denote I
min,, (N(() 2 1.
Mask Generation and Model Pruning: Each worker- n generates mask m, {0, 13" based on its own resources. Once worker-n receives the latest global model parameter 0, E RD from the server, it performs pruning operations based on its local mask, resulting in the initial pruned model parameters 0,,n, 0, O Mr,a in this round r.
After completing R rounds of distributed training (each with S steps), we obtain the score function estimation sæ,, (-). We can then solve the following process to approximate the initial data distribution Lz.0i
dYn (Yn, 2808 (Vn,te (k)]dt + V2d
E [tt, t++1]
Yn,o qn.T
Local Training with Pruned Model: Each worker- n performs steps local training on the pruned model. The model parameters are updated in each step according the gradient of the local loss fn and the local mask my,n:
(9)
Since the initial distribution q,. is not directly accessible, we instead sample Y,,0 from the standard Gaussian distri- bution næ leveraging the fact that the Ornstein-Uhlenbent (OU) process converges exponentially fast the standard Gaussian (Bakry al., 2014; Chen et al., 2023a).
oyn,, = 0,, - 17
O myn (7)
Furthermore, rather than running (9) to approximate the initial data distribution 9n.0 we opt to approximate 4n as an early-stopping measure (Song et al., 2020), setting tk T - 8. This approach is justified by the fact that for sufficiently small 8, the difference between gn.0 and fn.6
Here, represents the current step, 0, the model parameter from the previous step, and 77 is the local learning rate.

---

How Distributed Collaboration Influences the Diffusson Model Training? A Theoretical Perspective
is negligible. Additionally, this strategy mitigates potential issues where V log f may grow rapidly or even "explode" as t approaches zero, particularly in cases involving non- smooth data distributions.
Assumption 4.2 (Qiao et al., 2023) ensures that the pruning operation, which typically involves using a binary mask, does not degrade the model's performance beyond certain threshold.
As a result, starting from pure noise state Yn,0 ~ Td. the noise is gradually transformed to approximate the data distri- bution 9n.6 for each worker-n. And (E(,7) can be de- fined by the following SDE for each interval & [tR: tx+1]:
Assumption 4.3 (Bounded Variance). For any model @ and sample &, there exist > 0 and d2 > 0:
IE || (0,E) (F) l| of.
dY n. =Y,,, + 2son 190n(Ynne (Ynde T ts)]dt + v2dB...
E || VF. (F ll' o².
(10)
Assumption 4.3 (Lian al., 2017) imposes constraints on the influence of randomness, such as stochastic gradients, and regulates the divergence between local and global gra- dients. This ensures that these variations remain controlled and do not substantially disrupt the optimization process.
where (B is standard Brownian motion, and the marginals of this process can be denoted by pn,es. For
= -,,... K - 1, the above (10) can be solved explicitly by
Y Ynda+t1 =m, + +((e² - 11)se (Yn,te T tx)
Assumption 4.4 (Data Distribution). The data distribution In.0 of each worker-n has finite second moments Mn
+
- - 1,,
Yn, ~ Ad
Assumption 4.4 ensures that the data distribution of each worker has finite second moments, which is necessary for the convergence of the forward process in the diffusion model (Benton et al., 2024; Chen et al., 2023b;a).
where 6,,k ~ NO, Ia). For additional details, please refer to Appendix B. An illustration of distributed diffusion model training with pruning is provided in Figure 1.
To provide a theoretical foundation for deploying diffu- sion models on geographically distributed and resource- constrained devices, we analyze the effects of incorporating pruning techniques into distributed diffusion model training in Section 4.
Based on the above assumptions, our main results are estab- lished through the following steps:
Step 1: In Lemma 4.5, we establish an error bound for the discretization of the reverse SDE (3). This result extends the findings from previous work (Section 3.1 of (Benton et al., 2024)), which originally analyzed diffusion model training in a single-worker framework.
4. Main Results
In this section, we quamtify the difference between the ideal- ized data distribution In.6 generated by process (3) and the actual data distribution Pa,tæ obtained from process (10), using KL divergence as the measurement metric.
We evaluate the impact of distributed collabo- ration on diffusion model training through three-step analysis. First, we characterize the convergence behav- ior of distributed score function estimation in Lemma 4.6, capturing the effects of iterative updates, pruning opera- tions, and stochastic errors. Next, leveraging this con- vergence behavior and the construction of auxiliary func- tions, we derive an upper bound the local loss for each worker, as presented in Lemma 4.7. Finally, we ex- amine the discrepancy between the ideal loss, given by
Step 2:
Before presenting the formal results, we introduce the as- sumptions required for our theoretical analysis.
Assumption 4.1 (Lipschitzian Gradient). Loss function (-)s )s are with Lipschitzian gradients. i.e., For VO, و E IR,, holds that
K-1
(F) VF, (Ø) 2 L 1 0- 111
Σ TxE || sug (Yn,te, T (x) V log In,T (Yn, 112, k=
and the practical loss F, (0n). which arises due to the ap- plication of the denoising score matching technique. This inconsistency is formally analyzed Lemma 4.8.
Assumption 4.1 (Lian et al., 2017) is commonly adopted to guarantee the stability and solvability of optimization prob- lems. It ensures that the gradients of the loss functions vary smoothly, avoiding abrupt fluctuations, which is crucial for the convergence of gradient- optimization algorithms.
Step 3: In Theorem 1.11, we quantify the discrepancy be- tween the idealized data distribution 9n,ó and the actual data distribution Po.tx using KL divergence. This discrepancy can be decomposed into two key components: the differ- ence between reverse path measures and the impact of early stopping. The first component is derived by Steps and 2 using Girsanov's theorem, as detailed in Lemma 4.9. The
Assumption 4.2 (Pruning-induced Error). For an arbitrary mask m,,r E {0, d and a model 0, E IR (r = 1.... R and n = 1, ..... N), we assume that there exists " E [0, 1):
|| 0, 0r ④ mr,3 1P- n
5

---

How Distributed Collaboration Influences the Diffusson Model Training? A Theoretical Perspective
second component captures the distance between In.T and Wd following the findings of previous studies (Chen al., 2023a;b).
Based on the local update (7) and the global model aggrega- tion (8), B(()) and BB") can be bounded as follows:
L r
N
s
E || 6 0r., - 0, 12+ N
For the time discretization enror, we provide the expected difference between the drift terms at different time.
B
r
n=1
Lemma 4.5 (Time Discretization Error). (Benton et al., 2024) Suppose r is the solution to the SDE (3), and there exists some K. > O such that for each dis- cretized time point to.... 1kk we have "Ik- = 1k+1 - 4x S min(1,T I. Then it holds that
n
E F 11
2
2n² 2n SLNO
2n SL N 8
Brrr _
E || 0 - 0r 11?
(T+)²
r
==1
2x
LN
r + 2 VF(0.) 1
K
Ell A1 - A2 111 S dddK + KM + kdT
To further derive E || 0r 0, 1| we need to explore the cumulative entanglement of arbitrary pruning operations and local multistep training. In other words, it holds that
where we write "x S y² to mean I S Cy for an ab- solute constant C > 0. Additionally, the terms A and A2 are defined as A = V log (n.n-=(Ynnt) and A2 = V log Tt. (Yn,z),, respectively.
E 0r,n,--1 0r 11==B || 0r.n,s-1 0r,n,0 + 0r,n,0 0, 11
22E || 0r.me-1 - 0r 112 +2w2
This proof, originally presented in Section 3.1 of (Benton al., 2024), was developed in the context of single-worker diffusion model training. The core approach relies on novel Itô calculus argument, which establishes differential inequality for E log (n,,e)((mn.)) - V log (Y..-s(Yn,.))).
B5"
(12)
where BBx) can be bounded
Based on Assumptions 4.1.4.3, we can derive the follow- ing result for the distributed training of the score function estimation.
BRR =2E ||
71
V
my.n I|
Lemma 4.6 (Distributed Learning Dynamic), the local learning rate y satisfies < S min( / 6205-12 1), the following convergence result holds for the distributed learning of the above model On:
22n 1)
E V Sn (1
j=1
(F, + (0mms-)))) - -Fa(
(F, -F((- + VF(0 VF(0,) 112
8 8(F(O)) - F(OR))
E || (0.) 12-
+ +01 + 0z))
<8n - 1)1. Σ
E || 11.m.,111 -0 112+
R
RSR.
160w LN
40No2
161
- 8n (s - 1) || F(O,) 112 ( + +02
(T*)²
By summing (12) from s = 1 to S, from n = 1 to N, and from r = 1 to R, we can obtain the following inequality:
N Σ Fa) and (2 is defined in (5).
where F(O)= = N.
n=1
R-1 (1-81
N
|| - 112
Proof Sketch. We provide a brief outline here, and the de- tailed proof can be found in Appendix C.
r=0
R=1
==1
87/2 N
8ny² 5
R-1
Utilizing the Lipschitzian gradient assumption, we start the proof by analyzing the change in the global loss function during one round the model transitions from 0, to 0
(0) -
+
E F(0,) 112
r
r
2w RSN
(13)
[(( ((F())))
Then summing (12) from r = 1 to R, and substituting (13) to it, we can further control the appropriate learning rate !) to obtain the final result for distributed learning dynamic. This completes the proof outline.
L E || 0r+1 0, 1|?
S SE(F 0,41 0.) 2
(11)
B PP)
6

---

How Distributed Collaboration Influences the Diffusson Model Training? A Theoretical Perspective
Lemma 4.6 describes the rate at which the average gra- dient norm converges over all training rounds. The term ((F(0))-F(0a)) reflects the impact of iterative updates on the
where C1 is a constant independent of 0g-
Proof.
nSR
convergence behavior, while the remaining terms capture the combined effects of pruning operations, randomness, and local errors.
B|ll As A2 |19]
=E || s0x (Yn T - -tx) 112 +E V log T (Yn,))) 112
Based on Lemma 4.6, we can further derive the following local loss bound: Lemma 4.7 (Local Loss Bound). If the local learning rate
g. Eyn. 1210 (sen (Yn,t T tx), V log q n, (x,,))
=E || s0x (Yntu: T (x)) 11 +E V log fn T to (Ynts) 11
+ 2E ns n (Yn.e T -tt), n,,,, c (T-
Xn.0 X
7) satisfis 0 < 1) S minly 6405-1,2N 13, each local loss Fn (O)) can be bounded as
r.
- ((x
E || 80g (Ya,u T (x) + Yn c -T-- (T.D
|?
(OR) S (0o) - F(6) || +02 || 1 00 (0b)
- E 0-((1--aa
20nSRu LN 5n SNSNN
2n SRLN
+ || V log -n (Yn,te) 112 1i--1- e-2(T-re)
r
(T^)²
Y nntt c (T- Xn.
S + +
E || san (Yn T tx) +
1 +0
2-2(T-tx)
8
=E || s0x (Yn.s T (x) V log 10gg((Ynnn, (X 1 +CT
Proof: Based on Lemma 4.6, we can directly obtain the fol- lowing inequality based on the fact that the average gradient norm is non-negative:
where we use the fact that Xn,t = -n, and X
N(X,, -e 21 Ia) in step 3. Oft is constant
K-
F(OR) <F((0)+ + S +0²) 20nSRuu LN
independent of 0R. Let C1 2
Σ
7h Ch. it holds that
r
k= 0
5nSRNo3 SRLNO
(T+)²
B|l As- - A2 1121
We now need to bound the discrepancy between local and
global errors || F(OR) - (0) ||. Consider the auxiliary
7 || 80n
(Yn,ts: (0 T tx)) V (Kno) 1|
function h(t) = 0o + t(AR - Bo), then it holds that
F(OR) F(Oo)= ! (F(M()))(O 0o)dl
= + Fa(RR)+ +4
(0R) - (0)) L'' (F)((())))(RR= - -o)ddt
which completes the proof.
Subtract the above two equations and take the norm to get
Specially, as measures of learning loss, (4) and (6) are equiv- alent because the only difference between them is constant.
|| (..())) F(OR) I|
< || (Oo) - F(6)) +oz || 0 6 II
Then we can bound the distance between path measures by Girsanov's 's theorem, which is the first part of Step 3.
Then the proof can be completed by using the fact that (O)) < (OR) - ((RR) +F(RR).
O
Lemma 4.9 (Distance between Path Measures). Let Qn Pla.T be the path measure of the solutions to the process (3) and (9). They both start from Ya,0 ~ qu.T and run from 0 to tg. Then we establish the following result:
Lemma 4.7 describes the local loss bound of the score function estimation after It rounds of collaboration. How- ever, this result is based on the denoising score matching technique due to the unavailability of the score function V log - (. We explore its impact in Lemma 4.8: Lemma 4.8 (Equivalent Denoising Score Matching). If A₂ = V log In.T-t. (Yrote) as defined in Lemma 4.5, and Ag = San (Yn,te) T - 1x),, it holds that
K-1
KL(Qn) 1| 1M
<
EE|
A1 AA 11
0=
K 1
Tu Bll Ag - A2 [12]
k=
K-
EE
Ag- As 11 S (F((R)) +
where A₁, Ag, and Ag are defined in Lemma 4.5 and Lemma 4.8.
0
7

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Proof. Based on Lemma 4.5 and 4.8, it can be directly
Theorem 4.11 (Generation Error Bound). Suppose Assump- tions 4. 14.44 hold, T 1, and there exists constant C1, and some > 0 such that 7 S K min(1, -a -tx+1). Then under the same settings of as in Lemma 4.7. for each worker-n, using the collaboratively learned model On it yields the following result when approximating the data distribution qn,5i
derived that
Blll A1 As df]
K-
K- Σ k=
Blll A - A2 dd +
Bll
A3 - A2 [1a]
<00
[g 11 Pr.Xx)
< (0o) - F(Bo) +02 On - 0o +F((0) + C1+
Then according to Girsanov's theorem, it holds that
7,SRu RSR2 LN nSRN (o) + 0z) 1 ?
r +
+
A11
r
(r-)²
KL(Qn || plaT)
A1 - A3 dr
Rd + KMn,2 + xdT + (d + Mn,2)e 27
k=
A/
This argument is essentially identical to Proposition 3 of the work (Benton et al., 2024) and Section 5.2 of the work (Chen et al., 2023b).
In Theorem 4.11, the term || (o)) F(6) || +02 || OR
On captures the local-global error discrepancy. The term C1 arises from denoising score matching, while x²dK + RMM,22 + kdT is due to time discretization approximations, and (d -22' governs the convergence of the forward process. The remaining terms are interpreted as the local loss associated with OR which results from the distributed learning of score estimation with arbitrary pruning.
The following Lemma 4.10 indicates that the generation error bound with early stopping can be decomposed into two parts: the distance between path measures and the distance between T and πd.
Lemma 4.10 (Influence of Early Stopping). Let Qu P4a.r be the path measure of the solutions to the process (3) and (10) respectively. Under Assumption 4.4, the following result holds
We next show that we can choose suitable hyperparameters to control the generation error bound based solely on the distributed training dynamics.
KL Il Po.tx) S KL(Q,, P...T) - + (d Mn.2)e-2
Remark 4.12 (Hyperparameter Selection). For T > 1,
< 1, 2 log(1/6), we set k = (T' (1+bgg(1/6)
K T = 1 log (d+MM..2) ). K = O (d+ 2 (Tooo//()))))))
Proof. Since 9n and Pr,tx are the pushfowards of the path measures Qn and Pin.7, implies that KL(gn,6 Pr.tx) I KL(Qn] || pTa).
2
((0)
and further control the learning rate to satisfy
S min! (SNNlafta) SRNN"" SRNLOT }. we
F( F(06) F(B.)(T-)
Based on Lemmas 4.9, 4.5 and 4.8, we see that On is ab- solutely continuous with respect Pla. for each n 1...., N. For path y (y:)c=to.ex)- we write apan.T (H) = dxd (yn), which holds according to the fact dP Pan.7 and pad differ only by a change of starting distribu- tion. Therefore,
have KL(qn,5 || PP,AK) XII (0)) F(O,) || +02 OR - 0o l| +F((n) + C1.
d
Specially, if the initial samples of all workers are identically distributed, i.e., In,0 9m.u (for all then all local target loss function (.) are identical. In this context, each local loss F2 (0R) can be rewritten as
= 1,... NN,
KL(Qu/ || pmd) =Eq. log
dP
dP (Y)
dgn.T drd
(V)))
(nn( $F((0) + S|S LN
(7
I SRNOT (T+)²
=Q || pl..T) + KT.(Pa..r pua.T || Ta)
T
where the first term can be bounded by Lemmas 4.9, 4.5 and 4.8, and for the second one, it can be controlled by the following inequality (Chen et al., 2023b):
At a result, for the generation error bound, it holds that KL Po,tx) ) & F(Oo) Ct.
(L.(P || TT) S (d + Mn,2)e -T
which completes the proof.
8 Proof. For 2 1, 8 < 1, K 2 log(1/8). if we set
K k= θ T T+ (1gg( ) then there obviously exists sequence
According to the above lemmas, the formal generation error bound can be summarized as follows.
K such that 7 S 6 minn11 14+1). Then, if we (1
8

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
set K- = 0
a) ) it holds
References
F(66)
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14.11991-1799, 2021.
dF dFo)
x² dk = O
S F(Oo)
d Mn.2
((.)
KMnn2 O
- F(0o)
Bakry, D., Gentil, I., Ledoux, M., et al. Analysis and geome- try of Markov diffusion operators, volume 103. Springer, 2014.
(d+ 2n.2) (T' log(1/0))
dTF (Fo)
RdT = O
[d+ (TT log(1/0)) S F(0o)
Benton, J., De Bortoli, V., Doucet, A., and Deligiannidis, G. Nearly d-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations, 2024.
If we set T 1 log F(oo)
(4 it holds that (d +
2
Mn. Mn.2)e 27 F(On). Then have x2dK + KM + kdT + (d + Mn.2)e 2T S F(6o).
Block, A., Mroueh, Y., and Rakhlin, A. Generative model- ing with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
Similarly, if we further control the learning rate to satisfy 7 S min F(Og) F( (F(0))(')2)
SRN(af+o3)) SRN L V SRNLof }, we have nSRN(of
LN
1" + nRSRLN & F(6))-
Bortoli, V. D. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research. 2022. ISSN 2835-8856.
1
(1")²
These results complete the proof of Remark 4.12.
5. Conclusion
Chen, H., Lee, H.,, and Lu, J. Improved analysis of score- based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Con- ference on Machine Learning. pp. 4735-4763. PMLR, 2023a.
In this paper, we introduced an efficient distributed diffusion model training mechanism that adapts to varying resource constraints via local sparse training. Theoretically, we estab- lished the first generation error bound for distributed diffu- sion models, which scales linearly with the data dimension and aligns with state-of-the art results in the single-worker setting. Moreover, we demonstrated that with carefully se- lected hyperparameters, the generation performance of col- laboratively trained diffusion models is primarily governed by the dynamics of distributed training.
Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. R. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations, 2023b.
Conforti, G., Durmus, A., and Silveri, M. G. Score diffusion models without early stopping: finite fisher information is all you need. arXiv preprint arXiv:2308. 2tl::2338.22444444 /2240, 2023.
Acknowledgements
This work was supported in part by Joint Key Funds of National Natural Science Foundation of China under Grant U23A20302 and U24B20149, in part by the Na- tional Natural Science Foundation of China under Grant 62202273 and 62302247, in part by the National Key Re- search and Development Program of China under Grant No. 2022YFF07 100007121000 12100, in part by the Postdoctoral Fellow- ship Program of CPSF under Grant GZC20231 146,,,23460,,1 in part by the China Postdoctoral Science Foundation under Grant
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Goodfellow, I., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
2024M761806.
Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., and Wood, F. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35: 27953-27965, 2022.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851 2020.

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Li, M., Cai, T., Cao, J.. Zhang. Q.. Cai, H., Bai, J. Jia, Li, K., and Han, S. Distrifusion: Distributed parallel inference for nigh-resolution diffusion models. In 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7183-7193. IEEE Computer Society, 2024.
Song, Y., Sohl-Dickstein, J., Kingma, P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
Stanley Jothiraj, F. V. and Mashhadi, A. Phoenix: feder- ated generative diffusion model. In Companion Proceed- ings of the ACM Web Conference 2024, pp. 1568-1577, 2024.
Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328-4344 2022.
Tashiro, Y., Song, J., Song, Y., and Ermon, Csdi: Con- ditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-2488 2021.
Lian, X., Zhang, C., Zhang, H., Hsieh, C.-.,, Zhang, W., and Liu, J. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
Tun, Y. L., Thwal, C. M., Yoon, J. S., Kang, S. M., Zhang, C., and Hong. C. S. Federated learning with diffusion models for privacy-sensitive vision tasks. In 2023 In- ternational Conference Advanced Technologies for Communications (ATC), pp. 305-310. IEEE, 2023.
Lopez Alcaraz, M. and Strodthoff, N. Diffusion- based time series imputation and forecasting with structured atate apace models. Transactions machine learning research, pp. 1-36, 2023.
Vincent, P. connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661- 1674, 2011.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised featur learning, pp. 4.
Vora, J., Bouacida, N., Krishnan, A., and Mohapatra, P. Feddm: Enhancing communication efficiency and han- dling data heterogeneity in federated diffusion models. arXiv preprint arXiv:2407.14730 2024.
Granada, 2011.
Pedrotti, F., Maas, J., and Mondelli, M. Improved con- vergence of basee diffusion models via prediction- correction. Transactions on Machine Learning Research, 2024.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mmin a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 0rXiv:1708.077: 2017.
Pidstrigach, J. Score-based generative models detect mani- folds. Advances in Neural Information Processing Sys- tems, 35:35852-35865, 2022.
Yang, K. Y. and Wibisono, A. Convergence of the inexact langevin algorithm and score-based generative models in kl divergence. arXiv preprint arXiv:221 1.015 2022.
Qiao, J., Shen, S., Chen, S., Zhang, X., Lan, T., Cheng, X., and Yu, D. Communication resources limited decentral- ized learning with privacy guarantee through over-the-air computation. In Proceedings of the Twenty-fourth Inter- national Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, pp. 201-210, 2023.
Yang, I.., Zhang, 7., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., and Yang, M.-H. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1 2023.
Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. Proceedings of Machine Learning Research, 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2):3, 2022.
Zhou, H., Lan, T., Venkataramani, G. P., and Ding, W. Every parameter matters: Ensuring the convergence of federated learning with dynamic heterogeneous models reduction. Advances in Neural Information Processing Systems, 36, 2024.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713-477 2022.
10

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
A. Notation Table
Table I, we summarize main notations this paper.
Table 1. Some key notations and their descriptions.
Notations
Descriptions
T
The total time of noise scheduling
The current time of noise scheduling
K
The total number of discretized time interval of noise scheduling
l
The k-th discretized time point of noise scheduling, and it holds 0 = l0 < 4 < l
<... tK < T
Xn,
The data of worker-n time noise scheduling, such as image data The data of worker-n, which satisfies Yn,: Xn,r-t
Yn,
qu,, [0,
The marginals of the forward process (2) for each worker-n
d
The dimension of data
(Bn,t)eEjo.T)
The standard Brownian motion on R
(Bn,))EEOo.T
The standard Brownian motion on d The standard Brownian motion IR The standard Brownian motion on IR
(BALDEEiO
(Ba.)eEjo
(X,,
The score approximation which can be parameterized by a neural network with parameter vector 0 RDD
D
The dimension of model parameter 0, 8 E R
R
The total communication rounds for training the score approximation sg(-) The current communication round for training the score approximation sa(-) The number of local steps during two communication rounds The total number workers
S
N,(()
The set of workers for which the value of coordinate-i in the mask is non-zero, and NZC = {n: : m = 1}
The minimum occurrences of any dimension parameter the local model, and I min,, | 2
In fn En.s)
The loss of worker-n on a (a batch of) data sample En,s in the step of round r The loss function of worker-n, and (0) = [ (0,E.)
(0)
My,
The local mask of worker-n generated based on its own resources, and myn E (0,1) The learning rate for training the score approximation sg(+)
B. Solution to (10)
Consider (10):
dYn,: = (Yn, + 280 (Yn,te T - tx))dt + v2dBn,.
We multiply both sides of (10) by -t to get
d(e "Yna) = 288 (Yn,t, T - tx tx)d(e
'd
For each time interval [4x, 11,, we perform an integration operation to derive the following result:
1x+1 Yadeis =e - YYyne + 280g (Y T (x)(E - -(+1)) + V2 datz e
And then the following can be derived by multiplying both sides of the above equation by ctR12;
Ynsees = 2 1)san (Yn tx) + Ve 16,
where 7% 1++1 1s and 6n,k ~ (N,,

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
C. Detailed Proof of Lemma 4.6
Utilizing the Lipschitzian gradient assumption, we start the proof by analyzing the change in the loss function during one round as the model transitions from 8, to 8,+1:
L
- - ((VF0))) - -0,) + 2
0,41 0r 11?
(14)
B(
m
Based on the local update (7) and the global model aggregation (8), it holds that
0
ol0
0.50)
r+
IN
REN
Σ
17 Σ
1:: -mm -0(0)
|N
REN
S
Σ
(0())
(
Jn 11 mm(.)) 0/0
(5)
1N
m 7,
1)
REN!
Σ
fl0)
IN
REN
where the last step follows from the fact that for all n E N Then B(r) and Blr can be bounded as
(3)
Mr.. 1.
B
B(V (VE)) (0.), 0
D
S
E(VE (0,), -7 |N!' !EN
1
Σ
(00 (0 (0n,.))))
8=1
E(VE( (0,), -n |N
1
Σ
V
F (Brm..-1) -VF() - nSE VF((.) ||?
1=1
REN! 8=1
D
8
= - =- (F,) 12 S Σ
E
(VV
Σ Σ
(0 ((((0)))))
SIN!01
REN!)) ==1
DSS 2
S - S (F() 1|2
E 11 VF 1|2
+
S
E|
1
Σ
Σ
S
V F
(0rm,.-)) (0) (2)) (O)) 1|
2 i=1
NEN
s=l
I S
|| VF(O.)
12
7
Σ
V n1 -1) + V (-) - 1?
2
2
[N
nEN
< S E VF(O.)
1| +
17
N
E || (F.(Cr..s-)) (F,) 1 + r ==1==1
ΣVΣ E (..) VF((,) 112
N
2
r
n=1 ==1
TT N
N
S E VF(0.) 2
Σ Σ
E ||
0,,,,-1- 0, 112
T
1
1=
12

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
L
E
(6) x+1
0(00
||
2
1=1
L Σ
E
Σ
Sn, 1|
|N700
NEN s=1
D
Σ
Σ
(6) (0 (n V (0 + (0 VV (
2
n
1=1
EENO
==1
+
VFL (0,)
(F. + F (0.)) ||
<2L
Σ
( ( (n (0m.-)))))) 1|++
1=1
NEN, (8) s=1
2L
||
VF (0rn..-1) VFF 1
NEN
s
2L
||
T
Σ
(0,) -VF((()) 1 +2L
||
7
Σ
Σ
(B,) 1?
N
NN
1=1
NEN (8) s=1
1=1
NEN:
s=1
2
N
(0.,).,,))))) 11 - 11 21
N ΣΣ
E
|| F -Fa(0mm-))--FF. VV,,(0.) 1
r
SL
N
S
(F, VF(O,) 112 52 VF 1|
F
ST,
N
2 LNog
+
ΣΣ
0 0r 112
+ 2 VF(0,) 1|?
(T+)
r
r
Subtituting ( and B(T)) to (14), we can obtain
E[F(0,+1)] [[F(
$ - n - 52 L)) VF(0,) 112 +1 n22
2 2T,
N
E || 0r.,,-1 - 0, 112 +
r
USN
277² 5² LN
SLNo?
1 +
J0
(15)
1"
()
Next, to further derive E || O - 0, 11 we need to explore the cumulative entanglement of arbitrary pruning operations and local multistep training. In other words, it holds that
E || 0 0p 0, 2| || O - Or,0, + fr,n,o - 0, 112
2 0ym,a-1 0rn00 1|2 - 6, O my,n 0, 12
2E
Or,,,-1 Or.n. +2w2
(16)
B²²²)
where BSx) can be bounded as
BOd) -2E 1
V (n
Oyyn
1=1
<2n2(s 1) Σ
E || V (.n.
(Ommn)) V., VF,(0,) (F, (B. VF(B,) + VF(O.) 1|
0=0
13

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
<81 1)
E || VF, (Fr(0rn..-))-- (0,( +8n 1) F(O,) 1| 18)
1=1
a-1
<8n 1)
Σ
0-.j-1 -r 112 +81 - 1) F(O)) 112 +8n - -))
By summing (16) from = to S, from n 1 to N. and from r = 1 to R. we can obtain the following inequality if the learning rate satisfies
85
S
0r 1 S
8 S N
R-1
2w RSN
(1- L
E || 0r,n,-11
0i
02)
+
Σ
E || F(O,) 112
(17)
Summing (14) from 0 R I, we can obtain
[[[[(R)]] - E[F(())]
S- - n -2 L)
R-
12 nL + +n
R-1 1
E VF()))
N + +0))+++)++++++ N Σ
E || F(O,) 12
[0(1 -1 T.)
r=0
MSN
21/5 SN
Ro + 2 Ro?
+ 2w RSN) +
T
(Tt)²
Specially, we can further set the learning rate
1
1
7 S
+
S2 $8
nSL S 8
8SL
to get
S
5 < S1 2n15
4
SL2 < SLL 2nL2 nI?
n + SL3
S 8(1112 nI² 1 S22
4 =1022
4
nSN 4
2n S SSLL 2nSN
Then
[[F(OR) -
< S Σ E (F(),) 2|
R-1
10n
( + Σ E F(O.) 11 +2w*RSN) + 5 Ro
41
2ny² S
+
(1
S
80n 3 $3 N
R-1
L + +
20mL2Sw2NR
5nSN
2n² SLN Ro?
E
(F,) 12
+
r
r
r
47
(T+)2
(18)
Let
S
80n $3
118
S2
<
2
4
r
8
r+
640.5
It holds that
VI 1s 8(F(8)) F(BR))
160 L LN 40No3
16nLNot
E
+ (a + -33) +
+
S
(T+)²
D=0
which completes the proof.
14

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Cifur-10
SVIIN
Fushion-MNIST
FedDM-F
FoIDM-F
FoIDMFF
Trainin Loss
0.2
R-FodIDM-T.
Trainin Loss
0.2
R-FodDMM
Training Loss
0.2
R-FodTM44
B-FodDMM-
R-FedDM
R-FedlMM
R FedDM-S 5
R. F-FiDDM
R-FedliM
0.1
0.1
Communication
Communication
Communication
Figure 2. Training loss of FedDM under the random pruning with different pruning levels.
D. Experimental Details
D.1. Experimental Setup
We conduct experiments using the Cifar-10 (Krizhevsky et al., 2009) SVHN (Netzer et al., 2011), and Fashion-MNIST (Xiao al., 2017) datasets. To simulate distributed learning scenario, partition the training data among 10 workers. As described in Section 3, DDPM (Ho et al., 2020) can be viewed as special case of our work, version (known as FedDM (Vora et al.. 2024)) under resource-consiee resource-constrained conditions. In the experiments, we mainly consider two pruning techniques: Random Pruning (R) and Top-k Pruning (T) based on model weight. In particular, in order to explore the heterogeneity of pruning policy caused by resource differences among workers, we set for different pruning levels named F (Full), (Large), M (Medium) and (Small):
consider its distributed
F: All workers with full model;
L: 80% workers with full model, and 20% workers with 75% model parameters;
M: 60% workers with full model, 20% workers with 80% model parameters, and 20% workers with 75% model parameters;
S: 60% workers with full model, and 40% workers with 75% model parameters.
utilize multiple metries to evaluate the performance distributed training diffuusin models with different pruning levels: Training loss used assess the convergence for distributed learning of score estimation. Additionally, the Inception Score (IS) and Fréchet Inception Distance (FID) are employed evaluate the quality of data generation.
In the training stage of obtaining score estimation, we use the U-Net backbone containing residual blocks (Tun et al., 2023). And we use the following settings unless otherwise stated: The number of communication rounds Q is set as 300, the local training steps S are configured as 5 epochs for Cifar-10 10 and 2 epochs for both SVHN and Fashion-MNIST, and the step size 1) is 0.0001.
All the experiments are implemented in Py PyTorch 2.5.1, Python 3.12, Cuda 12.1. And we run them on Cloud Server with Intel(R) Xeon(R) Platinum 8358P CPU and total 10 RTX 3090 GPUs in Ubuntu 22.04.
D.2. Model convergence for distributed learning of score estimation
We assess the convergence for distributed learning of score estimation on the above three datasets, using Random (R) and Top-k (T) pruning techniques. Specifically, we establish four pruning levels (F. L, M, and to observe the effects on convergence behavior. This series of experiments is designed systematically evaluate how various levels of model sparsity influence the training dynamics.
Figures 2 and 3 illustrate the impact of different pruning strategies and pruning levels on the convergence rate of the distributed training diffusion model across three datasets. Overall, the training loss all settings is effectively reduced as the number of communication rounds increases, verifying the effectiveness of the coordinate- wise wise aggregation method. Under both pruning strategies, as the degree of pruning increases (denoted by F, L,, M, S), the training loss requires more
15

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Cifar-10
SVIIN
Fushion-MNIST
FedDM-F
FedDM-FF
FedDMeF
Traiiiy Lows
0.6
T-Fodf9M---
Training Lows
0.4
Training Lows
0.5
F
T-FalIDM-M
T.FoIIMMM
I-FedlM
0.4
T-FedDM-S
0.4
04
I-FeilMM
Communication
Communication
Communicatiinn
Figure 3. Training loss of FedDM under the Top-k pruning with different pruning levels.
Table 2. IS and FID comparison of FedDM with different pruning levels.
Cifar- 10
SVHN
Fashion-MNIST
Method
IS (f)
FID (4)
IS (f)
FID (L)
IS (t)
FID (C)
FedDM-F
4.59 0.13
73.73
2.79
163.36
3.58 # 0.08
87.59
R- R-FdDM--L
3.95
103.59
2.76
93.78
3.47 0.004
53.70
R- R-FedDM-M
4.01 0.08
104.53
2.00
127.47
3.32+
52.31
R- R-FedDM-S
3.00
111.21
0.05
120.57
3.06
49.94
T-FedDM-L
4.39
83.75
2.72
157.19
3.00
87.85
T-FedDM-M
0.10
80.42
2.55 + 0.05
146.27
0.06
100.69
T-FedDM-S
4.31 0.13
84.98
2.51 0.06
193.84
3.63 0.07
109.83
communication rounds to decrease effectively, and the total reduction diminishes. This is because the reduced model introduces additional errors, which slows the convergence rate to certain extent.
D.3. Data Generation Quality
We assess the performance of distributed training DDPM (known FedDM) with different pruning levels the above three datasets. Specifically, we establish four pruning levels (F, L, M, and S) and utilize two indicators, IS and FID, to observe and compare the average data generation quality.
6
2
21
6
S
&
Ar
3
6
54
(a) FedDM-F
(b) R-FedDM-L
(c) R-FedDM-M
(d) R-FedDM-S
(e) T-FedDM-L
(I) T-FedDM-M
(g) T-FedDM-S
Figure 5. Generated samples on SVHN for each pruning setting of the two pruning techniques.
16

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
(a) FedDM-F
(b) R-FedDM-L
(c) R-FedDM-M
(d) R R-FedDM-S
(c) T-FedDM-L
(f) T-FedDM-M
(g) T-FedDM-S
Figure 4. Generated samples on CIFAR-10 for each pruning setting of the two pruning techniques.
As shown in Table 2, the experimental results demonstrate that pruning significantly impacts the performance of diffusion models in distributed learning, with the effects closely related to the pruning strategy, dataset complexity, and model heterogeneity. On complex datasets such as CIFAR-IO and SVHN, the full model (FedDM-F) achieves the best performance, while increased pruning levels lead substantial decline in the quality of random pruning (R-FedDM), as indicated by decreased IS scores and increased FID values, particularly at high pruning levels (e.g., S). In contrast, Top-k pruning (T-FedDMM) better preserves model performance by retaining critical parameters, resulting in smaller increases in FID and performance closer to the full model, especially at moderate pruning levels (c.g.. M). For simpler datasets like MNISS, where the data distribution is less complex, pruning has a relatively smaller impact, and the performance difference between random pruning and Top-k pruning is minimal. Additionally, on Fashion-MNIs higher pruning levels unexpectedly improve FID values. This phenomenon can be attributed the lower capacity requirements of simple data distributions, where high pruning reduces redundant parameters, acting as regularization effect prevent overfitting, thus smoothing the generated distribution and making it closer to the real distribution. Model heterogeneity introduced by pruning is another critical factor affecting global performance, with random pruning more likely to cause aggregation errors, while Top-k pruning alleviates this issue to some extent. Overall, T pruning proves more advantageous for complex datasets, while random pruning is better suited for resource-constrained scenarios involving simpler tasks.
The generated samples shown in Figures 4-6 further reveal that pruning influences generation quality, particularly on complex datasets. For CIFAR-10 and SVHN, the full model produces images with more details and consistent color distribution, whereas aggressive pruning leads noticeable degradations-succh as blurred features, increased noise, and color distortions. Top-k pruning preserves critical parameters more effectively, yielding images that are more closed to those produced from the full model, particularly under moderate pruning conditions. For simpler datasets like MNNSS. the impact of pruning is less severe; in fact, higher pruning levels sometimes result in better images due to regularization effect that eliminates redundant details and prevents overfitting. Future work can focus on optimizing pruning strategies and aggregation algorithms to further balance model efficiency and performance across various data distributions and task requirements.
E. Discussion
Our analysis relies on several standard assumptions commonly adopted in the distributed learning literature, such as bounded gradient variance. While these assumptions facilitate tractable theoretical analysis, they may not fully capture the complexities of real-world distributed systems.
In our theoretical results, the term 2 reflects the extent of pruning applied the distributed model. As pruning becomes
17

---

How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
(a) FedDM-F
(b) R-FedDM-L
(c) R-FodDM-M
(d) R- R-FecDM-S
(c) T-FedDM-L
(f) T-FedDM-M
(g) T-FedDM-S
Figure Generated samples on
each pruning setting the two pruning techniques.
more aggressive (i.e., fewer parameters are retained), this term increases, leading to a looser error bound. The parameter I* captures the frequency with which each model parameter is updated across the distributed workers. small I indicates that some parameters are rarely trained, which may lead to suboptimal or imbalanced updates and thus larger bound. This motivates seek more effective pruning strategies that achieve balance between resource availability and generation quality, which we leave for the future.
While this work provides a theoretical perspective for distributed diffusion models under resource constraints, real- world constraints are often more complex. These constraints include heterogeneity in computational power, memory availability, communication latency, and the frequency of parameter updates. Such variability can pose significant challenges for maintaining model quality. Extending our framework explicitly model these factors-such as by incorporating asynchronous optimization or adaptive, resource-aware pruning- also represents an important direction for future research.
18


Reviews:
[{"review_id": "YvlPjnjIAO", "reviewer": "Reviewer_hsmD", "summary": "This theoretical paper analyzes the possibilities of distributed training of diffusion models The authors propose a new, privacy-preserving approach to distributed training of diffusion models and present a proof of the error bound. They further analyze hyperparameter adjustments to improve performance in this setting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer hsmD for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**In response to the concern about the outputs of diffusion models**, we have provided additional Figures 4-6 in Appendix E.3, which can also be found at the anonymous link: https://anonymous.4open.science/r/Diffusion-0D86/\nSpecifically, we randomly selected 30 noise instances following Gaussian distribution, and then generated 30 samples from them. This procedure was applied to each pruning setting of both pruning techniques across all datasets.\n\nThe generated samples shown in Figures 4-6 further reveal that pruning influences generation quality, particularly on complex datasets. For CIFAR-10 and SVHN, the full model produces images with more details and consistent color distribution, whereas aggressive pruning leads to noticeable degradations—such as blurred features, increased noise, and color distortions. Top‑k pruning preserves critical parameters more effectively, yielding images that are more closed to those produced from the full model, particularly under moderate pruning conditions. For simpler datasets like Fashion‑MNIST, the impact of pruning is less severe; in fact, higher pruning levels sometimes result in better images due to a regularization effect that eliminates redundant details and prevents overfitting.\n\n**For the concern about the Figure 1**, it serves as an illustration of distributed diffusion model training with pruning. Through Figure 1, we aim to help readers comprehend the training process as well as key aspects of the theoretical analysis, such as time discretization error, distributed training dynamics, and the impact of early stopping. \n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.", "claims_and_evidence": "The authors claims are primarily backed up via proofs, which is appropriate given the theoretical focus of the paper.", "methods_and_evaluation_criteria": "The method and evaluation criteria are appropriate for the paper's claims. I did appreciate the inclusion of experimental results in the supplementary materials. But it would have been beneficial to also include example model outputs to further characterize the generation error bounds.", "theoretical_claims": "I checked the correctness of the proofs to the best of my ability. All proofs appeared to be well-structured.", "experimental_designs_or_analyses": "Outside of the supplementary material there are no traditional experimental designs. The supplementary material includes what is essentially an ablation study that further supports the authors' theoretical contributions.", "relation_to_broader_scientific_literature": "The authors do a good job of positioning this work in terms of practical diffusion model research, theoretical diffusion model research, and distributed learning and privacy-preserving research. I have no complaints about this aspect of the paper.", "essential_references_not_discussed": "To the best of my knowledge the authors did not miss any essential references.", "other_strengths_and_weaknesses": "The paper is very well written and argued. I appreciate the inclusion of Figure 1 to help intuitively explain the approach despite the authors theoretical focus.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "1. Can the authors share outputs of the approach and its ablation? If not, is Figure 1's illustration representative or generation or just the training process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uj3kXTewMq", "reviewer": "Reviewer_grcd", "summary": "In this work, the authors investigate the impact of distributed collaboration on diffusion model training in environments with heterogeneous computational resources and data availability. It establishes the first theoretical generation error bound for distributed diffusion models, demonstrating a linear relationship with the data dimension d and consistency with single-worker results. A novel privacy-preserving training mechanism is proposed, incorporating synchronized noise scheduling and local sparse training to enhance computational efficiency. The study further highlights the critical role of hyperparameter selection, such as learning rate and noise scheduling, in optimizing generation quality. The findings provide a theoretical foundation for deploying diffusion models in distributed and resource-constrained settings while ensuring robust performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer grcd for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to “Other Weaknesses 1” ＆ “Other Comments Or Suggestions 1”:**\nWe thank the reviewer for this constructive feedback. We agree that the proofs in Lemma 4.6 and Theorem 4.11 could benefit from additional intuition and visual aids. In response, we have enhanced the manuscript by adding more detailed explanations that clarify the logical steps. These revisions are intended to make the technical sections more accessible to a broader audience while preserving the rigor of our arguments. We believe these improvements will significantly enhance the readability and overall presentation of our work.\n\n**Response to “Other Weaknesses 2”:**\nWe appreciate the reviewer’s observation. Indeed, while assumptions like synchronized updates and bounded variance are common in theoretical analyses, we recognize that these assumptions may be overly idealized when compared to the complexities of practical distributed learning scenarios. We adopted these assumptions to align our work with the established literature on distributed learning. In the revised version, we have included an expanded discussion on potential limitations arising from these assumptions, along with directions for future work to extend our framework to settings with asynchronous updates and more realistic variance conditions. We believe these clarifications strengthen the paper and provide a balanced view of both the theoretical guarantees and practical applicability.\n\n**Response to “Other Comments Or Suggestions 2-3”:**\nWe appreciate the reviewer’s detailed suggestions. In response, we have reworded several sentences in the introduction and conclusion for improved clarity and conciseness. For example, we revised the phrase to: \"The disparity in resources and data diversity undermines the accurate score function estimation assumed in single-worker models.\" Additionally, we conducted a thorough proofreading pass to address minor grammatical errors and ensure consistency in notation, particularly within the equations. We believe these refinements enhance the clarity and overall quality of the manuscript while maintaining its theoretical rigor.\n\n**Response to “Questions For Authors”:**\n**Regarding your first question**, we would like to thank you for your carefulness. There is indeed a citation error here, that is, (15) should be (16), which we have corrected in the revised version. We next explain how to get (17) from (16). \nBy summing (16) from $s=1$ to $S$, from $n=1$ to $N$, and from $r=1$ to $R$, we can obtain the following inequality:\n$$\n\\begin{align}\n\\sum_{r=0}^{R-1}\\sum_{n=1}^N \\sum_{s=1}^S \\mathbb{E} \\parallel  \\theta_{r,n,s-1}-\\theta_{r} \\parallel^2\n\\le& 8\\eta^2 S^2 L^2\\sum_{r=0}^{R-1}\\sum_{n=1}^N \\sum_{s=1}^S \\mathbb{E} \\parallel  \\theta_{r,n,s-1}-\\theta_{r} \\parallel^2+ 8\\eta^2 S^3 N R(\\sigma_1^2+\\sigma_2^2)+8\\eta^2 S^3 N\\sum_{r=0}^{R-1}\\mathbb{E} \\parallel F(\\theta_{r})\\parallel^2+ 2w^2 RSN  \\notag\n\\end{align}\n$$\nBy shifting term and dividing both sides of the inequality by $\\Gamma^*$, we can obtain (17).\n\n**For your second question**, if the initial samples across all workers are identically distributed, then $\\sigma_2^2$—which quantifies the discrepancy among workers—would be zero. Consequently, the bound on $F_n(\\theta_R)$ simplifies by eliminating the $\\sigma_2^2$ term. Thank you for your careful consideration, we have simplified this representation in the revised version. **For your concern on the third question**, we avoid using Lipschitz continuity assumption on the data due to its inherent limitations. A uniform Lipschitz condition can be overly restrictive, as the associated constant may scale with the data dimension—particularly when the distribution is approximately supported on a sub-manifold. Moreover, even employing a time-varying Lipschitz constant does not fully mitigate this issue. For example, Chen et al. (2023) assume Lipschitz smoothness at $t=0$ but still obtain a quadratic dependence on the data dimension $d$. Consequently, we choose not to rely on the Lipschitz assumption with respect to the data.\n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.", "claims_and_evidence": "This paper makes several key claims about the theoretical analysis of distributed diffusion model training, supported by rigorous theoretical derivations. The following is an assessment of the main claims and their supporting evidence:\n1. The paper derives a generation error bound for distributed diffusion models under resource-constrained scenarios. The analysis builds upon Girsanov’s theorem to measure the difference between ideal and actual data distributions. It demonstrates that the error scales linearly with the data dimension d, consistent with single-worker results. The theoretical results are formulated through KL divergence-based analysis and validated with convergence proofs.\n2. The error decomposition in Theorem 4.11 shows that the main contributor to generation errors stems from distributed training constraints rather than local model accuracy alone. \n3. The paper systematically analyzes how hyperparameters (learning rate, noise scheduling, pruning strategies) affect model convergence and performance.", "methods_and_evaluation_criteria": "The methods and evaluation criteria in this paper are well-aligned with its theoretical nature. The study develops a rigorous mathematical framework for analyzing distributed diffusion model training, leveraging KL divergence, Girsanov’s theorem, and stochastic gradient methods to derive a generation error bound. These methods are appropriate for assessing theoretical performance under resource-constrained and heterogeneous environments.\nThe key evaluation metric, KL divergence, effectively quantifies the difference between the idealized and actual data distributions in distributed training settings. Additionally, the study considers error decomposition due to factors like time discretization, sparse training, and distributed model aggregation, ensuring a comprehensive theoretical assessment.", "theoretical_claims": "The paper presents theoretical claims regarding the generation error bound for distributed diffusion models. These claims are primarily supported by mathematical proofs leveraging KL divergence, Girsanov’s theorem, and stochastic gradient analysis. Below is an assessment of the key proofs:\n1. Time Discretization Error (Lemma 4.5): The proof applies Itô calculus to analyze the impact of discretized time steps on the reverse stochastic differential equation (SDE). The result extends previous single-worker error bounds to the distributed setting, ensuring consistency with established theoretical findings.\n2. Distributed Learning Dynamics (Lemma 4.6): This lemma establishes the convergence rate of the gradient norm in a distributed training framework with pruning. The proof correctly derives an upper bound on the expected gradient norm over multiple training rounds, accounting for pruning-induced errors and stochastic noise. While the proof relies on the bounded variance assumption (Assumption 4.3), which may be restrictive under extreme data heterogeneity, this assumption is widely accepted in distributed learning and does not significantly impact the theoretical validity.\n3. Distance Between Path Measures (Lemma 4.9): This proof uses Girsanov’s theorem to derive an upper bound on KL divergence, linking the idealized and practical training processes in a distributed setting. It extends the analysis from single-worker diffusion models to multi-worker collaboration while maintaining theoretical rigor.\n4. Generation Error Bound (Theorem 4.11): The theorem quantifies the gap between the idealized data distribution and the actual learned distribution using KL divergence. The proof decomposes the overall generation error into distinct contributions from time discretization, distributed training dynamics, and early stopping effects, providing a comprehensive theoretical foundation for analyzing distributed diffusion models.\nIn summary, the proofs are well-structured, mathematically sound, and extend existing theoretical results to the distributed setting. While certain assumptions—such as bounded variance and synchronized updates—may not fully capture real-world constraints, they are reasonable within standard theoretical frameworks for distributed learning. Overall, the claims made in the paper are strongly supported by rigorous mathematical analysis.", "experimental_designs_or_analyses": "Since this is a theoretical paper, the experimental design and analyses are included only in the appendix and primarily serve to illustrate the theoretical findings. Below is an evaluation of the soundness of these experiments:\nThe experimental design effectively illustrates the theoretical findings by simulating a distributed training scenario with different pruning strategies on CIFAR-10, SVHN, and Fashion-MNIST. The use of training loss, IS, and FID as evaluation metrics provides a reasonable assessment of model behavior under resource constraints. \nWhile primarily a theoretical study, the experiments in the appendix support key claims and offer insights into the practical implications of distributed diffusion model training.", "relation_to_broader_scientific_literature": "This work builds on existing research in diffusion models, distributed learning, and theoretical generative modeling, extending key findings to a distributed setting. It connects to prior work on diffusion model error bounds (Chen et al., 2022; Benton et al., 2024) by establishing the first known generation error bound for distributed diffusion models under resource constraints, aligning with single-worker results. Additionally, it draws from federated learning by incorporating sparse training and pruning strategies to optimize model performance in heterogeneous environments. The study also reinforces prior findings on hyperparameter selection in generative models (Song et al., 2020) by demonstrating that training dynamics, rather than local accuracy alone, drive generation quality in distributed training. By bridging the gap between diffusion model theory and distributed optimization, this paper provides a theoretical foundation for scalable, resource-efficient generative modeling.", "essential_references_not_discussed": "In this paper, the authors provide a thorough discussion of prior work, covering key contributions in diffusion model theory, distributed learning, and generative model optimization. It appropriately cites foundational studies on diffusion model error bounds (e.g., Benton et al., 2024; Chen et al., 2022), as well as relevant work on federated and distributed training (e.g., Li et al., 2024 on DistriFusion). Given the scope of the paper, there do not appear to be major missing references that are essential for understanding its contributions. The cited works sufficiently frame the theoretical advancements and position the study within the broader landscape of distributed diffusion models.", "other_strengths_and_weaknesses": "The paper demonstrates strong originality by extending single-worker diffusion model theory to the distributed setting, providing the first known theoretical generation error bound under resource constraints. This is a significant contribution, as most prior theoretical analyses of diffusion models have focused on centralized training. The study also introduces a novel distributed training mechanism that incorporates privacy-preserving synchronized noise scheduling and sparse training via pruning, which aligns well with real-world constraints in federated and distributed learning.\nThe theoretical rigor is a key strength, as the paper provides clear derivations using KL divergence, Girsanov’s theorem, and stochastic gradient analysis to quantify the impact of distributed training on generation quality. The mathematical results are well-structured and contribute to a deeper understanding of error propagation in distributed diffusion models.\nIn terms of clarity, the paper is generally well-written, with precise definitions and clear explanations of theoretical results. However, some technical sections—particularly the proofs—could benefit from additional intuition or visual explanations to improve accessibility for a broader audience.\nA potential consideration is that some assumptions like synchronized updates and bounded variance, though standard in theory, may not fully capture practical distributed learning challenges.", "other_comments_or_suggestions": "The paper is well-structured and clearly written, but here are a few minor suggestions for improvement:\n1. Some proofs, particularly in Lemma 4.6 and Theorem 4.11, could benefit from additional intuition or visual explanations to improve readability for a broader audience.\n2. A few sentences in the introduction and conclusion could be slightly reworded for clarity and conciseness. For example, the phrase \"This discrepancy in resources and data diversity challenges the assumption of accurate score function estimation foundational to single-worker models\" could be made more direct. \n3. Conduct a final proofreading pass to catch any minor grammatical errors or inconsistencies in notation, particularly in equations.\nThese are minor refinements, and the paper is already well-organized and rigorous in its theoretical contributions.", "questions_for_authors": "1. Could the authors explain in detail how (15) is summed to obtain (17) in L.721? Is there a formula citation error here, i.e. (15) should be (16)?\n2. If the initial samples of all workers are identically distributed, does that mean sigma_2=0 in remark 4.12? If so, then the corresponding bound of F_n(theta_R) can eliminate sigma_2.\n3. As far as I know, a lot of work on single-node diffusion models requires the score function to be Lipschitz continuous with respect to the data, such as [1]. Does this paper use the same assumption?\n[1] Chen H, Lee H, Lu J. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions[C]//International Conference on Machine Learning. PMLR, 2023: 4735-4763.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSYzLpWThb", "reviewer": "Reviewer_ySb9", "summary": "This paper presents a theoretical analysis of distributed diffusion model training in scenarios where computational resources and data availability vary across workers. Traditional single-worker diffusion models assume uniform resources and centralized data, which are impractical in distributed settings. To address this, the authors introduce a novel distributed training mechanism that preserves data privacy through synchronized noise scheduling and accommodates resource heterogeneity via local sparse training. They establish the first generation error bound for distributed diffusion models, demonstrating a linear relationship with the data dimension d and consistency with state-of-the-art single-worker results. Using Girsanov’s theorem, the paper quantifies the impact of time discretization, training sparsity, and data heterogeneity on model generation quality. Additionally, the authors analyze coordinate-wise model aggregation to mitigate sparse training errors and show that hyperparameter selection, particularly learning rate and noise scheduling, plays a crucial role in optimizing distributed training dynamics. The findings provide a solid theoretical foundation for extending diffusion models to decentralized environments while maintaining competitive performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer ySb9 for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to “Essential References Not Discussed”:**\nWe appreciate the reviewer’s insightful suggestion. While our work primarily focuses on theoretical contributions, we agree that incorporating discussions of experimental works in distributed diffusion models can further underscore the practical relevance of our analysis. In response, we have cited some works on the empirical performance of distributed diffusion models, such as Phoenix and FedDM. We believe these additions enhance the manuscript.\n\n\n**Response to “Other Weaknesses 1” ＆ “Other Comments Or Suggestions 1”:**\nWe appreciate the reviewer’s comment on the balance between mathematical rigor and intuitive clarity. In response, we provide a few brief intuitive explanations. For the reverse SDE derivation, we now clarify that reversing the SDE involves “rewinding” the forward process by inverting the drift, which conceptually aligns with the idea of reconstructing the data distribution from noise. For the local loss bound (Lemma 4.7), it is directly derived based on the fact that the average gradient norm is non-negative. We hope these explanations address your concerns.\n\n\n**Response to “Other Weaknesses 2” ＆ “Other Comments Or Suggestions 2” ＆ “Questions For Authors 3”:**\nWe appreciate the reviewer’s insightful suggestion. To make it clearer how the results in Remark 4.12 are established, we have added a detailed derivation in the Appendix. Here are the details:\nFor $T\\ge 1$, $\\delta < 1$, $K \\ge \\log(1/\\delta)$, if we set $\\kappa = \\Theta \\left(\\frac{T + \\log(1/\\delta)}{K}\\right)$, then there obviously exists a sequence $\\\\{t _ k\\\\} _ {k=0}^K$ such that $\\gamma_k \\le \\kappa \\min \\\\{1, T-t_{k+1}\\\\}$.\nThen, if we set $K=\\Theta \\big(\\frac{(d+M_{n,2})(T+\\log (1/\\delta))^2}{F(\\theta_0)}\\big)$, it holds\n$$\n\\begin{equation}\n\\\\left\\\\{\n\\begin{aligned}\n&\\kappa^2 d K=\\Theta \\Big( \\frac{d F(\\theta_0)}{d+M_{n,2}} \\Big)\\lesssim F(\\theta_0)\\\\\\\\\n&\\kappa M_{n,2}=\\Theta \\Big( \\frac{M_{n,2} F(\\theta_0)}{(d+M_{n,2})(T+\\log (1/\\delta))} \\Big)\\lesssim F(\\theta_0)\\\\\\\\\n&\\kappa dT =\\Theta \\Big( \\frac{dT F(\\theta_0)}{(d+M_{n,2})(T+\\log (1/\\delta))} \\Big)\\lesssim F(\\theta_0)\n\\end{aligned}\\\\right.\\notag\n\\end{equation}\n$$\nIf we set $T=\\frac{1}{2}\\log \\big(\\frac{d+M_{n,2}}{F(\\theta_0)}\\big)$, it holds that $(d+M_{n,2})e^{-2T}=F(\\theta_0)$. Then we have $\\kappa^2 d K+\\kappa M_{n,2}+\\kappa dT+(d+M_{n,2})e^{-2T}\\lesssim F(\\theta_0)$. \n\nSimilarly, if we further control the learning rate to satisfy $\\eta \\le\\\\{\\frac{F(\\theta_0)\\Gamma^*}{SRN(\\sigma_1^2+\\sigma_2^2)},\\frac{F(\\theta_0)\\Gamma^*}{SRNw^2 L},\\sqrt{\\frac{F(\\theta_0)(\\Gamma^*)^2}{SRNL\\sigma_1^2}}\\\\}$, we have $\\frac{\\eta SR w^2 LN}{\\Gamma^*}+\\frac{\\eta SRN(\\sigma_1^2+\\sigma_2^2)}{\\Gamma^*}+\\frac{\\eta^2 SRLN\\sigma_1^2}{(\\Gamma^*)^2}\\lesssim F(\\theta_0)$.\n\nThese results complete the proof. And we hope these explanations address your concerns.\n\n\n**Response to “Questions For Authors 1-2”:** Thank you for these thoughtful questions. **For your concern about the constant $C_1$**, we believe that the current version is correct. In fact, $C_1$ is generated due to the equivalent denoising score matching, which we have discussed in Lemma 4.8. Therefore, this constant is not included in the bound of local loss $F_n(\\theta_R)$. **For your concern about $\\sigma_2$**, it is indeed equal to zero and can be omitted. Thank you for your careful consideration, we have simplified it in the revised version. **For your concern about $w^2$ and  $\\\\Gamma^{*}$**, the term $w^2$ reflects the extent of pruning applied to the distributed model. As pruning becomes more aggressive (i.e., fewer parameters are retained), this term increases, leading to a looser error bound. The parameter  $\\\\Gamma^*$ captures the frequency with which each model parameter is updated across the distributed workers. A small  $\\\\Gamma^*$ indicates that some parameters are rarely trained, which may lead to suboptimal updates and thus a larger bound. This motivates us to seek more effective pruning strategies that achieve a balance between resource availability and generation quality, which we leave for the future.\n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.", "claims_and_evidence": "The claims are supported by rigorous proofs, and the key claims and their corresponding evidence are as follows:\nClaim1 : Distributed training of diffusion models introduces unique challenges due to resource heterogeneity and data privacy concerns.\nEvidence 1: The paper discusses how centralized data processing is impractical in real-world scenarios and introduces a distributed training mechanism that allows for local sparse training while preserving privacy. This is mathematically formalized in the distributed learning dynamics (Section 3), where the authors account for the effects of worker variability and introduce coordinate-wise model aggregation.\nClaim 2: A generation error bound for distributed diffusion models is derived, which scales linearly with the data dimension d and aligns with single-worker results.\nEvidence 2: The theoretical generation error bound is rigorously derived in Theorem 4.11, which decomposes the total error into components from time discretization, distributed training errors, and early stopping. The proof relies on Girsanov’s theorem and previous results from single-worker diffusion models, showing that the bound remains consistent with known results while extending to the distributed setting.\nClaim 3: Hyperparameter selection, particularly noise scheduling and learning rate, significantly impacts the generation quality in distributed settings.\nEvidence3: The paper provides a hyperparameter selection strategy (Remark 4.12) to control the generation error bound. By carefully tuning these parameters, the dominant factor influencing model quality remains the distributed training dynamics. This is supported by analytical expressions that show how different choices affect convergence.", "methods_and_evaluation_criteria": "The authors present a rigorous theoretical framework for analyzing distributed diffusion model training, leveraging coordinate-wise aggregation, sparse training, and hyperparameter optimization to address key challenges in distributed settings. The use of Girsanov’s theorem and score matching techniques ensures a solid mathematical foundation, and the derivation of the generation error bound provides valuable insights into the effects of time discretization, distributed training dynamics, and early stopping. While the theoretical analysis is comprehensive, the assumptions of bounded variance may not always hold in real-world settings, and discussing their relaxation would strengthen the generalizability of the results. Overall, the proposed methods and evaluation criteria are well-suited for a theoretical study, and the findings provide a solid foundation for future research on distributed generative models.", "theoretical_claims": "The theoretical claims in the paper are supported by rigorous mathematical derivations, particularly in establishing the generation error bound for distributed diffusion models. I examined the key proofs, including:\n1. Theorem 4.11 is the core theoretical result, providing an upper bound on the difference between the ideal and actual data distributions in the KL divergence sense. The proof systematically decomposes the error into contributions from time discretization (Lemma 4.5), distributed learning dynamics (Lemma 4.6), local loss bounds (Lemma 4.7), and denoising score matching equivalence (Lemma 4.8). The use of Girsanov’s theorem (Lemma 4.9) to measure the discrepancy between the true and learned distributions appears correct and follows standard techniques in diffusion model analysis.\n2. The proof of Lemma 4.6 follows a standard optimization analysis approach, bounding the expected gradient norm using assumptions on Lipschitz continuity and bounded variance. The decomposition of pruning-induced errors and stochastic gradient noise is logical, and the derivation steps are consistent with related works in distributed learning.\n3. Lemma 4.8 shows that the local loss function used in training is equivalent to the theoretical loss function up to a constant. The derivation follows from the Gaussian noise assumption in the forward process, which is a well-established technique in score-based generative modeling.", "experimental_designs_or_analyses": "As a theoretical paper, the primary contribution lies in the mathematical analysis of distributed diffusion model training, and the experimental results in the appendix serve as a supplementary aid to help understand the practical implications of distributed training under resource constraints. The experimental design focuses on how pruning strategies and pruning levels affect training loss and generation quality, providing empirical insights into the role of sparse training and coordinate-wise aggregation and effectively supporting the theoretical claims.", "relation_to_broader_scientific_literature": "The authors extend the theoretical study of diffusion models from single-worker settings to distributed training under resource constraints, contributing to both generative modeling and distributed optimization literature. Prior research has established error bounds, stability, and convergence properties for single-worker diffusion models (e.g., Benton et al., 2024; Chen et al., 2022, 2023), while this work introduces the first generation error bound for distributed diffusion models, demonstrating that the error maintains a linear dependency on data dimension d. By bridging the gap between diffusion model theory and distributed optimization, this paper provides a foundational step toward resource-aware generative models in distributed environments.", "essential_references_not_discussed": "It cites the most relevant theoretical works on diffusion model error bounds, distributed optimization, and federated learning, ensuring a strong foundation for its contributions. However, while this paper is primarily theoretical, incorporating discussions on experimental works related to distributed diffusion models would further highlight the importance of this theoretical analysis.", "other_strengths_and_weaknesses": "Pros.:\n\n— The paper is the first to establish a generation error bound for distributed diffusion models. By demonstrating that the error bound scales linearly with data dimension d and remains consistent with existing single-worker results, the paper provides a foundational contribution to the theory of distributed generative modeling.\n\n— The theoretical claims are well-structured, logically derived, and supported by rigorous proofs, making the results convincing and mathematically sound. The decomposition of the generation error into multiple contributing factors (time discretization, distributed training dynamics, early stopping, etc.) helps provide a deeper understanding of the challenges in distributed diffusion model training.\n\n— While the paper is primarily theoretical, it addresses key challenges in real-world distributed learning, including data heterogeneity, resource variability, and privacy constraints.\n\nCons.:\n\n— Some of the notation choices, particularly in the reverse SDE derivation and loss function formulation, might require prior familiarity with score-based generative modeling. While the notation table in the appendix helps, adding a brief intuitive explanation of key equations in the main text could improve readability.\n\n— This paper emphasizes the importance of learning rate and noise scheduling, but only gives the results in the Remark. However, as far as I am concerned, the results of these hyperparameter selections are not intuitive, and adding more details will improve persuasiveness.", "other_comments_or_suggestions": "1. While the notation is mathematically rigorous, some equations, particularly in the reverse SDE derivation and loss function formulation, may benefit from a brief intuitive explanation to make the theoretical framework more accessible.\n\n2. The paper highlights the role of learning rate and noise scheduling in distributed diffusion training, but a brief guideline or theoretical intuition on hyperparameter choices would be useful for researchers implementing these ideas.", "questions_for_authors": "1. In Remark 4.12, for the iid setting of data, does the bound of each local loss omit the constant C1, and shouldn't sigma2 be zero when all F_n are the same?\n\n2. In Theorem 4.11, the generated error bound is affected by both the pruning level w^2 and the minimum parameter occurrences \\Gamma. Can the authors explain this in more detail? And discuss the limitations of the current work?\n\n3. The results highlight the role of learning rate and noise scheduling in distributed training. Are there any general guidelines for selecting these hyperparameters to ensure the error bound remains tight? In other words, how is the choice of hyperparameters established in Remark4.12?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8V8VXSmF9", "reviewer": "Reviewer_YCna", "summary": "The authors explore the theoretical performance of distributed diffusion models, particularly in environments where computational resources and data availability vary across workers. The authors establish a generation error bound for distributed diffusion models under resource constraints, demonstrating a linear relationship with the data dimension and alignment with existing single-worker results. Key contributions include a novel training mechanism that maintains data privacy via synchronized noise scheduling and sparse training strategies, an analysis of the impact of hyperparameter selection on model performance, and a theoretical framework for optimizing distributed diffusion models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer YCna for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to “Theoretical Claims”:**  We agree that supplementing technical proofs with intuitive explanations greatly enhances accessibility. And we have added some explanation to the key results. For example, regarding the reverse SDE derivation, we now clarify that reversing the SDE entails “rewinding” the forward process by inverting the drift term, which is conceptually consistent with the notion of reconstructing the data distribution from noise.\n\n**Response to “Experimental Designs Or Analyses”:** We have added a discussion regarding the limitations and future directions in the appendix. Specifically, we offer the following reflections: While this work provides a theoretical perspective for distributed diffusion models under resource constraints, real-world constraints are often more complex. These constraints include heterogeneity in computational power, memory availability, communication latency, and the frequency of parameter updates. Such variability can pose significant challenges for maintaining model quality. Extending our framework to explicitly model these factors—such as by incorporating asynchronous optimization or adaptive, resource-aware pruning—represents an important direction for future research.\n\n**Response to “Other Weaknesses 1” ＆ “Other Comments Or Suggestions 2”:** We have added a detailed derivation for Remark 4.12 in the Appendix. For more details, you can refer to our response to Reviewer ySb9.\n\n**Response to “Other Weaknesses 2” ＆ “Other Comments Or Suggestions 1”:** We have added a discussion on the important theoretical results, limitations and future directions in the appendix. The details are as follows: \n\nOur analysis relies on several standard assumptions commonly adopted in the distributed learning literature, such as bounded gradient variance. While these assumptions facilitate tractable theoretical analysis, they may not fully capture the complexities of real-world distributed systems.\n\nIn our theoretical results, the term $w^2$ reflects the extent of pruning applied to the distributed model. As pruning becomes more aggressive (i.e., fewer parameters are retained), this term increases, leading to a looser error bound. The parameter $\\\\Gamma ^*$ captures the frequency with which each model parameter is updated across the distributed workers. A small $\\\\Gamma^{*}$ indicates that some parameters are rarely trained, which may lead to suboptimal or imbalanced updates and thus a larger bound. This motivates us to seek more effective pruning strategies that achieve a balance between resource availability and generation quality, which we leave for the future.\n\nWhile this work provides a theoretical perspective for distributed diffusion models under resource constraints, real-world constraints are often more complex. These constraints include heterogeneity in computational power, memory availability, communication latency, and the frequency of parameter updates. Such variability can pose significant challenges for maintaining model quality. Extending our framework to explicitly model these factors—such as by incorporating asynchronous optimization or adaptive, resource-aware pruning—also represents an important direction for future research.\n\n**Response to “Questions For Authors 1”:** While our error bound shares a structural similarity with those in single-worker settings—relying on factors such as model dimension—there are key differences in how errors accumulate. Specially, distributed systems introduce additional challenges due to inter-worker heterogeneity and uneven parameter training. These are captured in our analysis by terms like $\\\\sigma _ 2^2$ and  $\\\\Gamma^*$, which do not appear in single-worker settings. Moreover, pruning may lead to imbalanced updates, which is quantified by the terms $w^2$ and $\\\\Gamma^*$ in our bound. In this context, the assumption of perfect score function estimation—often made in single-worker analyses—no longer holds. The key contribution of our theoretical framework is to explicitly characterize new errors that are unique to distributed generative modeling.\n\n\n**Response to “Questions For Authors 2”:** We agree that identifying real-world applications can help illustrate our theoretical contributions. However, to maintain the paper’s main theoretical focus, we refrain from expanding on specific applications within the main text. Nonetheless, we briefly highlight several potential application areas where our theoretical findings may provide guidance: (1) Federated generative modeling in privacy-sensitive domains such as healthcare and finance. (2) Collaborative generation across edge devices for generative tasks like image synthesis or speech enhancement, where each device has limited compute/memory.\n\nIf there are any further confusions, we are happy to clarify them. Thank you again for your recognition of our work.", "claims_and_evidence": "The claims in the paper are generally well-supported by theoretical derivations and empirical validation. The generation error bound is rigorously formulated using mathematical proofs, and its consistency with single-worker results enhances its credibility. The experimental evaluation demonstrates the practical viability of the proposed training method by assessing convergence behavior and data generation quality under different pruning strategies.", "methods_and_evaluation_criteria": "The methods used in the paper, including distributed diffusion models training with noise scheduling and sparse model updates, align well with the challenges posed by heterogeneous computing environments. The evaluation criteria are reasonable, including convergence analysis, Inception Score (IS), and Frechet Inception Distance (FID), which are commonly used for assessing generative models.", "theoretical_claims": "The theoretical claims, particularly the derivation of the generation error bound, are well-supported. The proofs follow established methodologies, such as using Girsanov’s theorem for error quantification. The assumptions made (e.g., Lipschitz continuity, pruning-induced error constraints) are reasonable and align with prior work.\n\nWhile the proofs are detailed, it would be beneficial to provide more intuition behind the key theoretical results for better accessibility.", "experimental_designs_or_analyses": "The experimental setup effectively simulates real-world distributed training scenarios, using datasets such as CIFAR-10, SVHN, and Fashion-MNIST. The study examines different pruning strategies (random and top-k pruning) and their impact on training loss and data generation quality. The inclusion of multiple pruning levels ensures a comprehensive analysis.\n\nAlthough this is a theoretical paper, further exploration of how more diverse resource constraints affect the performance of distributed diffusion models will enhance the paper's persuasiveness.", "relation_to_broader_scientific_literature": "This work is well-positioned in the context of distributed machine learning and generative modeling. It builds upon foundational studies in diffusion models and distributed optimization, such as prior work on stochastic control and Girsanov-based error bounds.", "essential_references_not_discussed": "The paper cites most of the relevant literature but could consider discussing more recent advances in federated learning techniques for generative models, particularly those addressing resource efficiency.", "other_strengths_and_weaknesses": "Strengths:\n1. It establishes the first known generation error bound for distributed diffusion models under resource constraints, a significant theoretical contribution that advances our understanding of model performance in distributed settings.\n2. The mathematical proofs are comprehensive and well-structured, utilizing advanced techniques such as Girsanov’s theorem to quantify training errors, making the study highly rigorous.\n3. The paper employs advanced mathematical tools, such as Girsanov’s theorem, to derive a precise error bound. The assumptions made (Lipschitz continuity, bounded variance, pruning-induced errors) are reasonable and well-motivated. Theoretical claims are clearly stated and justified, making them highly credible.\n4. The paper is well-structured, with clear mathematical notations and logical flow, making it easy to follow.\n\nWeaknesses:\n1. The results of the hyperparameter selection are not intuitive, and the authors could add more discussion to explain it in detail.\n2. The authors do not explicitly discuss the limitations of their proposed approach or suggest potential future research directions. While the study provides strong theoretical and empirical contributions, a clear discussion on the scope and boundaries of the proposed method would help contextualize its impact and guide future work.", "other_comments_or_suggestions": "1. Consider providing a high-level summary of theoretical results for accessibility.\n2. The author should further explain how the constraints on hyperparameters are derived.", "questions_for_authors": "1. The paper states that the derived error bound is consistent with single-worker results. Could the authors elaborate on the key similarities and differences in how errors accumulate in distributed vs. single-worker settings?\n2. What real-world applications might this work provide theoretical guidance for? Adding a corresponding discussion would benefit the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper considers training diffusion models in resource-constrained distributed settings, whereby it provides a generation error bound which can be controlled based on the choice of certain hyperparameters, depending on the training dynamics. Its techniques draw on a variety of tools from probability theory, and the reviewers are all in agreement that this work presents important theoretical contributions to our understanding of distributed training for diffusion models. The paper would therefore be a welcome addition to the conference.
Decision: Accept (poster)