Principled Long-Tailed Generative Modeling via Diffusion Models
Authors: Pranoy Das
Kexin Fu
Abolfazl Hashemi
Vijay Gupta
Venue: NeurIPS 2025 poster ()
Keywords: Generative Conditional Diffusion Models
Deep Mutual Learning
Long-Tailed Learning
Learning in Games
Primary area: theory
TL;DR: We provide a mathematically rigorous framework to study the use of Deep Mutual Learning in the context of conditional diffusion models for long tailed generation.

Abstract: Deep generative models, particularly diffusion models, have achieved remarkable success  but face significant challenges when trained on real-world, long-tailed datasets- where few "head" classes dominate and many "tail" classes are underrepresented. This paper develops a  theoretical framework for long-tailed learning via diffusion models through the lens of deep mutual learning. We introduce a novel regularized training objective that combines the standard diffusion loss with a mutual learning term, enabling balanced performance across all class labels, including the underrepresented tails. Our approach to learn via the proposed regularized objective is to formulate it as a multi-player game, with Nash equilibrium serving as the solution concept. We derive a non-asymptotic first-order convergence result for individual gradient descent algorithm to find the Nash equilibrium.  We show that the Nash gap of the score network obtained from the algorithm is upper bounded by $\mathcal{O}(\frac{1}{\sqrt{T_{train}}}+\beta)$ where $\beta$ is the regularizing parameter and $T_{train}$ is the number of iterations of the training algorithm. Furthermore, we theoretically establish hyper-parameters for training and sampling algorithm that ensure that we find conditional score networks (under our model) with a worst case sampling error $\mathcal{O}(\epsilon+1), \forall \epsilon>0$ across all class labels. Our results offer insights and guarantees for training diffusion models on imbalanced, long-tailed data, with implications for fairness, privacy, and generalization in real-world generative modeling scenarios.

Principled Long-Tailed Generative Modeling via Diffusion Models
Pranoy Das, Kexin Fu, Abolfazi Hashemi, Vijay Gupta
School of Electrical and Computer Engineering, Purdue University, W. Lafayette, IN, 47906 {das211, fu448, abolfazl, ggt.llla2ellggttt886666 edu
Abstract
Deep generative models, particularly diffusion models, have achieved remarkable success but face significant challenges when trained real-world, tailed datasets- where few "head" classes dominate and many "tail" classes are underrep- resented. This paper develops theoretical framework for long-tailed learning via diffusion models through the lens of deep mutual learning. We introduce a novel regularized training objective that combines the standard diffusion loss with mu- tual learning term, enabling balanced performance across all class labels, including the underrepresented tails. Our approach to learn via the proposed regularized objective is to formulate it as a multi-player game, with Nash equilibrium serv- ing the solution concept. We derive non-asymptotic first-order convergence result for individual gradient descent algorithm to find the Nash equilibrium. We show that the Nash gap of the score network obtained from the algorithm is upper bounded by O VTrrgto
+ 3) where 3 is the regularizing parameter and Train is the number of iterations of the training algorithm. Furthermore, we theoretically for training and sampling algorithm that ensure that we
establish hyper-paramett find conditional score networks (under model) with worst case sampling error O + 1), Ve > across all class labels. Our results offer insights and guarantees for training diffusion models imbalanced, long-til long-tailed data, with implications for fairness, privacy, and generalization in real-world generative modeling scenarios.
Introduction
Successful integration of deep learning models into society requires working with real-world data. This comes with many challenges: data quality issues such inaccurate data, data bias, ethical issues such as breach in privacy, transparency, technical issues such data integration, generalization, scalability, etc. Furthermore, real world class-labeled datasets are not uniform, but follow a skewed or sometimes referred to as "lone-tailedd" distribution. It is characterized by a "head" classes that occurs with high probability while the probability the rest of the classes, often referred to as "tail" classes fall off very quickly. It is well known that the performance of traditional deep learning generative models (300) suffer significantly when trained on long-taii tailed distributions.
and
One might be curious to ask, "Should deep learning or generative models be concerned with class labels which occur with very low frequency?" The answer is Yes! Even though individually each class occurs with low frequency, collectively these classes may occur with high probability. Diffusion models, which are the focus of this work, are no exception to this phenomenon. Diffusion models are latent variable generative models which learn diffusion process for given dataset, such that the process can generate new elements that are distributed similarly as the original dataset (See section 3 for more details). They have become popular techniques in image generation beating traditional models such as GANs [3] 7 38), natural language processing [39] time series forecasting [26] and in fields of applied chemistry (I]. biology [9] and medicine [15 to name few. However, the study of diffusion models for long-tailed learning is limited.
(23) showed that when the traditional conditional Diffusion Denoising Probabilistic Model (DDPM) is trained on long-tailed distribution,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the conditional DDPM model as shown in [23 Figure 11, "generates head class images with satisfying performance, whereas conversely. the generated images on tail classes are very likely to show unrecognizable semantics". Moreover, there might be privacy and ethical concerns if the model overfits (memorize) to the tail class label data and replicate them during generation.
Motivated by this, we develop a theory of Loig-Tailed Learning for diffusion models in a mathe- matically rigorous manner through the perspective of Deep Mutual Learning. Our main results are:
1. Under suitable metric (KI.- divergence) that captures the distance between the learnt class distribution and the ground truth distribution, we derive an upper bound on the worst case distance across all class labels. To do so, we employ Deep Mutual Learning along with the score based diffusion model objective in literature [29 32] We present the formulation as game across conditional score networks and propose Nash equilibrium as the appropriate solution concept.
2. Borrowing ideas from [3] Deep Mutual Learning, we derive non-asymptotie first order convergence result for the individual gradient descent algorithm to find the Nash Equilibrium of the proposed game. We show the Nash gap of the score network obtained from the algorithm is upper bounded by O VTtrase + (3) where 3 is the regularizing parameter and Tirain is the number of iterations of the training algorithm. Finally, we show we can find hyper-parameters for training and sampling such that the score networks obtained from the algorithm enjoys a worst case error bound of O(e + 1) for any > 0 for any class, tail and otherwise.
2 Related Works
Long-Tailed Learning for Diffusion Models. To tackle the issue with long-tailed distributions, diverse techniques have been proposed such as re-sampling (27], re-weighting [27]. transfer learning
23], J, and feature augmentation [0O. The closest work ours is that of (3 titled "Class- Balancing Diffusion Models" or CBDM and its followups [33 35 The paper proposes a distribution adjustment
regularizer as solution along with the usual DDPM objective. This represents a modification in the training phase. Their experiments show that the images generated by CBDM exhibit greater diversity and quality in both quantitative and qualitative ways when trained on CIFAR LOO/CIFAR 10LT datasets. As mentioned in 333, "CBDM (23) represents an inaugural inquiry into the performance of DDPM within the context of long-tailed data scenarios". Motivated by CBDM and contrastive learning, (33) propose adding penalty function to demarcate the distribution boundaries of differ- ent data categories. However, the derivation of the distribution adjustment regularizer 23 24 Proposition 2, Appendix A] relies on strong assumptions. They follow traditional machine learning framework that optimizes over a single objective function with a single neural network and give empirical verification of their method's performance. On the other hand, define game across conditional score networks and propose the Nash equilibrium of this game as the egalitarian solution to learn fair score function for equally good generation over all classes. Furthermore, our framework and analysis do not rely on the strong assumptions made in 23 24
Proposition 2, Appendix A].
Deep Mutual Learning, Deep Mutual Learning (DML) 36 is a knowledge distillation process that allows the transfer of knowledge from a highly powerful model smaller faster efficient model. In DML, an ensemble of students (models) learn collaboratieely and teach each other throughout the training process. DML has shown promise visual object tracking [37), metric learning (22), multi-modal recommender systems [66. and classification tasks trained on Long-Tuile Tailed distributions
211. The theoretical performance guarantees for models trained with DML are scarce. [13] gives a non-asympiotic first order convergence result for training models for classification task using DML. Deep Mutual Learning literature proposes various methods for optimizing Deep Mutual Learning objectives without specifying the solution concept they seek. In contrast, we show that the individual gradient descent (one method for DML) is seeking a Nash equilibrium of underling multiplayer game. While this result of ours could be of independent interest, in this work further leverage this result to obtain a generalization result for diffusion models for long-tailed generation.
Training and Sampling of Score-based (Conditional) Diffusion Models. The performance of score-based diffusion models have been rigorously studied in the literature of generative modeling. M 32 provided full error analysis of training and sampling from diffusion model. [ [7]

---

parametrize the score network using a random feature model and use gradient flow to train the model. [T] leverages Neural Tangent Kernels obtain an approximation and generalization error for diffusion models. 32 parametrize the score network by deep neural network and prove exponential convergence of its gradient descent training dynamic on the empirical loss function. For conditional diffusion models, [8] provides data- dependent approximation bounds of the conditional score function by multi-layered neural network and also give an expected sampling error of the approximated distribution over all class labels. Compared to S 32]. we consider a finite label class and make no assumption on how the data distributed. While our result can readily be extended to deeper neural network in line with (32), parametrize the score function using two-layer ReLU network (as [77) due to the nice properties it induces the proposed game.
exponentii
Basics of Score-Based Diffusion Generative Models
Notation Let ||-I] denote the & norm for vectors and matrices, I|-I| g be the Frobenius norm. For the discrete time points, we use & to denote time point for forward dynamics and t for backward dynamics. (z) where z E Hd refers to the ReLU activation function applied element-wise while o refers to the variance of the forward dynamics. algorithm, which in our case is gradient descent. By is the training parameter for score for label E y while 8-y the training parameter for score for all label y' E y {y) Given two distributions p
[Ttrain] represents the iteration of the training
and g. the KL divergence from to is defined as DKL (pllg) = SsR p(x) log ((2)
p(z) dz.
In subsequent sections, we introduce the basics of diffusion model training and generation. Denote the initial conditional distribution as PX zly), Vx A c Rdd .A is a compact set of all possible features and y where y is the finite set of class labels.
3.1 Forward and Backward Processes
The use of diffusion model in generative modeling involves two processes:
1. Forward Process: The forward process pushes initial distribution Po(--v) to Gaussian by adding noise progressively to Xg, and is usually described as an Ornstein-Uhlenbee (OU) process,
(I) where fr.gt are functions of t E [0, and dW, is the incremental Brownian motion or Wiener process, X, is a d- dimensional random variable with X₁ ~ P.(.y). The choice of
dX, = -f-Xdt + g.dW,, with Xo ~ Po(.ly). Vy
Jt. St results in various diffusion model schemes such as Variance Preserving (VP), Variance Exploding (VE) SDE (see [29] for more details).
2. Backward Process: To generate a new sample, the forward dynamics can be reversed conditioned the final distribution X²= ~ Po(lly) get the backward or reverse diffusion process defined as:
dx; f + of V V. loggr- 18) dt + gr gd, Pr(-1)) (2)
where X5 ~ Pr and p. is the density of P,. Then X²_, and X, have the same distribution with density P-(-l)), which means that the dynamics will push near- eear-Gaussian distributions back to the initial distribution Po (-(g),
. Vy E
3.2 Training via Denoising Score Matching
From 22.. to generate samples conditionally, one needs access to V log ( [8). the conditional score function, which is unknown. Let ez, y) be an estimator of V, log Po (zly). To estimate the conditional score function, a natural loss function to train a model would be the following objective:
1 T
Lconts (8) (. (0) = Ey
2 J
A(t)En(1)
[| 2 logp:(z(t)l)) (t. st,0
T
== 2 t0
((t)
[|| log sr,o(x(t), dt.
3

---

Once the conditional score function is learnt, a datum from class label y y is sampled using the reverse diffusion process given below:
dx;- fr--Xt= + 1), dt gr- with X ~N(0, I).
(3)
To measure how well the learnt score function approximates the ground truth distribution, KL- divergence is employed as the metric. To assess the goodness of the learnt score function through the optimization of (0), we have to relate the KL-divergence between the learnt distribution and the ground truth to the training objective. Informally, the KL divergence between the learned distribution and the ground truth distribution bounded by the score based diffusion model objective
(fronti (0)) as (see (28 Theorem IJ or [8] Appendix DJ for detailed proof)
Ey D ( S (0)
Achieving a bound on the expectation as g Theorem 4.1] gives no insights into the worst case sampling error over all y V. In this work, provide methodology achieve upper bound on maxgey DKL (Po(-1y)] ( Po.o(ll))))))))) thereby addressing the long- issue in generative modeling.
Long-Tailed Learning
4.1 Egalitarian Solution Concept
Previous work in conditional diffusion models [8] have focused on optimizing the following objective
Lconti (0) Eyy ("Comes (0)
Ely)) (0)
yEy
for classifier guided sampling 29 or the unconditional score function along with the conditional score function (rom5]oor classifier free guidance. The above objective is sound when the marginal density of the classes ply) itself is uniformly distributed. Observe that when optimizing Lconts (0)
(eq. 5). an optimization algorithm will give more weight towards reducing L (0) for head classes (classes with high p(y), appearing with higher frequency in the data). Thus, the trained model overfits the head class, while performing poorly on the tail classes. One way of ensuring that each class label is equally weighted during the training process is to re- re-weigh each class objective function by factor inversely proportional the class marginal density p(y). This ensures that both head and tail classes receive equal weighting during the training process.
Lconti,batancen (0) = EE,, (P()) C% Comme (0) ] E C%
(6)
DEY
However, in many real world scenarios the marginal density ply) is unknown and hence such an accurate reweighting is not possible. For Long- Taigee Learning. we desire to perform well (in terms of generation quality) for every class label, the natural objective would be to minimize maxvey (Po.. (Po(-Iv)]l Po.((()))))))) that is, minimize the worst-case KL divergence over all y E y.
Suppose C (0) is convex the training parameter 0, then so is f(0) = maxyey L Loonty (0) as maximum of finite convex functions is again convex. (0) may not be differentiable even if conts (0) are differentiable in for all y E y. One could use sub-gratd gradicnt methods to optimize the worst case class loss max, L (0). However, in practice one has to work with the empirical version of these losses which might be noisy and lead to parameters that are sub-optimal with respect to the population loss.
4.2
Nash Equilibrium as Solution Concept
To enable diffusion models for Long-tailed learning, we modify the DM objective to add the mutual learning objective defined as
1 Ly man (0y, 0_ w())) 2 1" ((t) Ey By
(x())) (10000
2 s,,, (x(1)) dt, (7)

---

to obtain regularized version of the DM objective function denoted as Coo (8y, 0 0-y). In the setting of Mutual Learning, the distribution ter over which one could optimize. From now on, we will drop the weighting arguments u.)) in the objective functions, leading to the following regularized objective for each class:
uniform. But, the distribution can be a hyperparame-
Ceontt reg (oy: 0-y = Co (0,) + 33 (0y: 0-y)-
Learning the score Va att)) log Pe(((t)))))) is difficult as it is intractable. Conditioning on Xo and using law of iterated expectation, one can rewrite the objective function as (see [32 Appendix A] for detailed proof) with discretized time points as 0 < to < < <... = ' to get the training objective
Cree 0-y) = (0,)) (0y 00.
1 \(tz)(ts Ex, || ((8))()(()()) 0,,,0,
j=1
1 Ew ((t))(t -t txx1) Ex, 1x Lo.Biing (130, st,0, (x.(tz) C(v) 3
(9)
where C(y) =
X()( (g-1)C, and Ci(v) = Ex, || (-y)i(-
Ex,Ex. V log pi (1 (o y)l?? 132 Remark 1] point out that Cly) < and hence the first summand in Eq. 9 always bound below by -Cly). Clv) along with the entire first summation in cnrrorred to L (B,) while the third term is Lmme (9g. A f-y). As C(y) doesn't 't depend on 0, we can ignore it for the purpose of training. But we note that Cly) will appear in our final worst case sampling error. When the drift and diffusion coefficient of the forward dynamics satisfy some nice properties, the distribution of pe(xr/zz is normally distributed, whose mean and variance (ō+) can be explicitly computed. Exploiting this knowledge, one can rewrite the objective function in eq 9 as (See Appendix [.3 for details)
Crey( (0y, 0-y) = L (0,) + B 100 0-y)
My
X(1) -1--1)
(0 + E:gll2
2n
( [se),,, $1, (0(()))) (m,,))))))))))))))))))))))))))))
(10) where Ereg (0y. is the empirical version of L (0y, 0-)) with By samples, {ar) with Ty Pol-ly) denotes the initial data, {Gig) where Eig NO, Ia) denotes the noise and input
+ Bw(tt))(tz - t)-1) ~e
N data of the neural network is (t))) where rt,()) (-y()) is obtained from the forward diffusion process.
4.2.1 Neural Network Architecture for Score Parametrization
The approximation power of two-layer ReLU network with randomly sampled input layer are well understood from numerous works [12 25 and has been used to study the generalization properties of Diffusion Models in 17]. We also parametrize the score function St,0, for each label y E )
using a random feature model
81,8, (1)
AAoo(W U,e(!) m
1
m Σ ay ( (, + uT
m
(11)
where (()) = mmx (0. is the ReLU activation function, A, (dy,1, (ymn) E Rdxm is the trainable parameter, W,, (wy,1, WT R and U, = (N My.m ) E Rdxde are randomly initialized embedding matrices that are frozen during training, embedding function for the time. The above model represents neural network with one hidden layer with m neurons and d- dimensional vector as an output. Suppose oy i Wy and uy are i.i.d. sampled from an underlying distribution p. Then as m - co, we can view
R>0 + R& is the
8t,9, (x) + 3,, (x) = wo,,, ayo x x u E(t))
Ew.. ay (w, (o ut (e(tt)
5

---

Algorithm 1 Individual Gradient Descent(IGD)
Input parameters: Learning rate 7)r Initialize: (Wy, Uyyyy and Og: Vy E y for T Or..Ttrain do
for y 0... [ do
0-+1 + 07
7, Ve, Creg 0-y)
end for
end for Output: (8y, 0yy) = min, [Terain NE-gap(f7..0) 0-y)
with ag (w, u) po(w,, Su ayp(a, w, w)da, and pa(w, u) == Sax p(a, w, ujda. The above relation represents St,0, (x:) as an approximation of the continuous version St,ū, (æ), which can be viewed as neural network with infinite width, i.c., infinite number of neurons in the hidden layer (m - oo). Furthermore, assume the embedding matrices Wy and Uy are sampled independently for every y E y from a set with bounded support.
Having defined our loss function, we define the strategy space as O2 = {Ag E Rdxm 1|Ayll S B), Vy E V. Now, consider the |ll-erayer game ()', (Frig) yyey, (oy)gey),
Definition (Nash Gap). Let By ② yy Gy represent the best response function for label y E y defined as (y E argmingge, (0, (0,00 f-v)- Using the best response function, we define the Nash gap of strategy profile (8y)yEy E Xyey as:
NE-gap((0u)VE5)) = max Chey (0y, - yEy
(12)
Definition 2 (Nash Equilibrium). strategy (By)EE E XyEyO, is an e- Nash equilibrium of the game (y, (Erey) (0y)vey) if NE (E-gap((f,))eE)) S e. When NE-gap((()) = 0, then vey is a Nash equilibrium.
The ability to find an Nash equilibrium of the game (V) (Crigless, (Qy)vey) is crucial in our analysis to bound the worst case sampling error.
4.3 Algorithm
In this section, we propose the individual gradient descent algorithm to find an approximate Nash equilibrium of the game (D), (CVrg) (Oy)vey). The input parameter for the algorithm is the step-size 7,r where is the 4h step of the individual gradient descent algorithm. The initializa- tion step samples the embedding matrices and fixes an initial condition for the training parameter (Wy, U,, ((0) )yey. The individual gradient proceeds for Trrain steps and within each step individ- ual gradient update is performed by computing the gradient Vo, (07, O
The complexity of finding Nash equilibrium: One of the most celebrated results in game theory (6) proved that the computational complexity of the problem of computing of a Nash equilibrium in an arbitrary game lies the complexity class PPAD. So far, there does not exist an polynomial time algorithm that can find an approximate exact solution to problems in PPAD. The game (V, (Crrg)9))) (Oy)yey) is a convex minimization game (See B.5.. 20 showed that concave maximization games (convex minimization games) also lie in the class PPAD. We present positive result that in our game (Y, (C"sg)yey (Oy)vey), individual gradient descent finds an approximate
Nash equilibrium whose NE-gr ME-gap is bounded by O VTenin +))
m²
5
Main Result
We now present the main result of the capability of diffusion models in learning through deep mutual learning. We derive a data-in data-independent worst-case bound for D (pu(-Iy) Ipo.o,,. a (-lg)). Let 0 = argming, (0,),, Vy y and let 0 be the optimal solution when the true score function

---

$t,0, (I)) is replaced in the class-label objective function (6y)) (equation 9) by its approximation &r,8 A (z)) We make one assumption on the support of data distribution (justified in Remark J.
Assumption 1. We assume that the target distribution (xl is continuously differentiable in and has compact support for every E y. Let for any E V, x E A C I2, | S K
Generation Algorithm. We consider the DDPM sampling scheme. Under this scheme
and
@t 2 in Eq. Denote the backward time schedule as {tj JosjEN such that 0 to < t ... 'N = T az To simulate the backward SDE, we use the exponential integrator scheme [34 which can be piecewisely expressed continuous-time SDE: for any E ( 1(41)).
dr. = 7Yy= 28T-15 8, (Y15 )) + 22dW.
(13)
Denote (e(-lv)= = Law (8)), VE [0,
7h = + 1 and assume there exists x >> 0 such that "A S x (1, ). Let u² be such that Pe [ll====11ll111111=) S u <0, Vy E V.
0o,
Remark 1. Assumption ensure the data belong to bounded set and the score is well defined. This also ensures the second moment of the data distribution are bound which necessary for convergence of forward SDE. Some works 4 321 do not require the existence of score function for the data distribution Po(-ly)- These works employ early stopping of the reverse (sampling) process. They do so because for non-smooth data distributions V log qt can blow up as t T. This means that the model will approximate ar-a rather than qr Po(-ly), which acceptable since for small a the distance (e.g. in Wasserstein-p metric) between qr-a and Po(-ly) is small [4].
We now present the main result of the paper.
Theorem 1. Given Assumption for 0 < ā < 1, we have with probability 1 N(Eyed Ryy) that
Ly m² smooth w.rl to their own parameter 1. The empirical loss functions Lreg (0. 0 are
Vy E (See Lemma in Appendix B.5
2. If one runs individual gradient descent with step-size Thr S maxyyy Ly / Treasa for Terain iterations and selects the parameter from (0y, 0 re( that minimizes the Nash Gap
of the game (V, (Cren)vEy, (Oy)vev) and samples according Eq
13
the sampling error
max L (D) 5
m
max DEL (aPa yEy
S
8)+
vEY
VTtraan
1
1
Co Nu + kT exp (-2T)u²) + C (14)
Vmn
where n² mingey Reg maxyey -y) as Eq. R.2 Nu? 3 + KT an upper bound the discretization error due the reverse SDE, exp (-2T) u22 is the error due to the convergence of the forward SDE and constant Co some constant. Ô hides the log factors, 1)/2 and bounds on strategy space, embedding matrices and other constants.
Corollary 1 (Full Error Analysis). Fix > arbitrarily. If 1,a 1 and N > log 1 then there exists
e T+oog =) and
to t1 .... = -T such that for some Thk S mmin(1, T 1x + 1jvk 0,1, ,N If we take T 1 log d. N O o dl 21²), B = Ôc)) Tirain = 6 (1) and m = Ō then under similar conditions as Theorem we achieve
max DKL (Pat (Alv)) S max L" +C+E NEy
(15)
vEy
where Ö, Ô and S hides the polynomial of log 1 ()) and bounds on strategy space, embedding matrices and other constants. L (*(0)) is the universal approximation error of approximating the score with two layer network with random ReLUs.
Corollary [ gives us the range of hyper-parameter such as width of hidden-layer, number of training steps, discretization of sampling, etc. to achieve worst case sampling error of O(e+1). The O(1) term C(v) in Eq. can viewed as the error incurred due to diffusion model's 's nature approximating v log Pr (z 3) which is intractable by V log Pe (xe /z0, y) with reverse SDE.
7

---

5.1 Proof sketch of Theorem 1
We provide a sketch for the proof and defer the details the Appendix. We use a slight variant of 4 Theorem 1J (See Appendix [ for more details) to upper bound the KL-divergence between the distribution approximated by our model and the ground truth to get
max ( NEY
& max (0,,) - Co + KTT3 + exp (-2T)uz). (16)
yE
We then perform the following decomposition for maxyey L (0y) (See Appendix B.1 where
min
max (6T) I max L (05)
max sup NEY (0,.0 v)
(0 f-y) (0, 0-g)
TE(aaratal yEy
yEY
(17)
min
max sup (0, O-y). NE-gap(0 0 3 yEt (0,
Y (E(traann)
Proposition 1 (Training and bounding the Nash Gap). Suppose Lreg (0%, (-)) is ty y E y. Then by selecting constant learning rate S VItrasn S maxyey Ly that depends on the total iteration Troinn and using Ô to hide the log factors, have
smooth for all
m
min NE-gap(0T, 0 0-y) 2 min max Tefara)
Vo, Freg (0) 0 0I,) 0
m
+ B). (18)
TE(Tiram NEY
/Trrin
The proof is presented in Appendix B.6 Proposition gives non-asymptotic first order convergence of individual gradient descent. When no further assumption on the gradient mapping (e.g., (strong) monotonicity of the game (V,, (Creg)yey, (Oy)vey)) is considered, this is the best can hope for. The iterate at which the minimum Nash Gap is achieved can be tracked by storing the parameters (0%, 0- )',,) for which the max a, Frey (0r, 0" "I"
is the least.
Monte-Carlo Estimate. To bound m L (67). we employ ideas from [7 Lemma 6]. Infor- mally (See Prop2] in Appendix B.7 for 0 < < 1, with probability 2N we achieve
max (O) S max (O) O NEY
(19)
yEY
m
where Ô hides the log & factors. ( is the error associated with approximating the score of the data using a two layer networks random ReLUs.
Rademacher Complexity. Finally, we bound the generalization error (See Lemma 9 in Appendix B.8 for the derivation) by the Rademacher Complexity
max sup BEY (B,.
L (0,, 0-y) - Eryy (0y, 0- 1= 1 Vmn + C
(20)
where n² min ny. C max Cyy) δ hides the log 8 factors, 1/ and bounds on strategy space, embedding matrices and other constants.
Bound Mutual Learning Loss The final term in Eq. 17 maxyeY sup(e, P/ (Oy, 0 0-y) is O(1) (See Lemma Gin Appendix B.&
5.2
Interpretation of the Main Result and Implications for Long-tailed Learning
Firstly, when the training objective function are nice, Proposition] shows that individual gradient descent employed in Deep Mutual Learning literature is seeking a Nash Equilibrium of an underlying game across different models. Second, when diffusion models are employed for long-tailed generation, Theorem [ shows that Nash equilibrium of an underlying game across conditional score network achieves an egalitarian solution w.r.t to sampling error. Our result give insight into the bottleneck process in diffusion generative modeling when faced with limited computing resources and longetailed data. To the best of our knowledge, our result the first to provide a comprehensive view of Deep Mutual Learning and long-tailed generation(learnnng with diffusion models.

---

21 20 (g)
pi (x
|| V log (all))))
S1,8, &, (x)ll²
V, logn(arl))
Suc, (r]11r111,,
10
10
-2
5
3
500
time
Gime
Figure 1: Fitting error on toy demo with and without mutual learning. Top row represents the tail class and bottom row represents head class. The middle column represents mutual learning and the right column represents without mutual learning (3 0). Lighter areas represent higher probability region (left column) and larger fitting error (middle and right column)
6 Numerical Experiments
6.1 Toy Model
Diffusion models are trained to learn the score function V (x y) of the forward process which is then used in the reverse SDE to sample. From Figure (left column), the score model works well when Pr (x]l)) is large but suffers from large error when Pt (z) y) is small. This observation can be explained by examining the training loss on Figure Figure[middie middle and right columns). Since the training data is sampled from pr (rrl),, in regions with a low pt (zly) value, the learned score network is not expected to work well due to the lack of training data. As a consequence, to ensure Yo is close to To. one need to make sure Y, stays in the high Pe (xlly) region Vt E [0, The mutual learning term aligns the score network for the tail classes with the high confidence scores of head classes at the high noise regime area (t >> 0) decreasing the fitting error. This can be seen from a comparison of the heatmap of the tail class (top row middle column) with mutual learning having larger portion of area with low fitting error compared to the case with no mutual learning (top row right column). [
6.2 Real World Datasets
Datasets We perform empirical validation of our theoretical findings with the widely used CI- FARI0 dataset in the domain of image synthesis, specifically its loig-tailed versions CIFAR IOLT. The construction of CIFARIOR 10LT follows from S where the size decreases exponentially with its class label index according to the imbalance factor imb = 0.01. We also perform exper- iments on synthetic dataset such Gaussian Mixture Model and include them in Appendix
Method
FID(J)
IS(T)
Vanilla DDPM (8 0)
16.58
8.78 + 0.15
Mutual Learning
14.58
8.92 + 0.19
CBDM
15.28
8.11 = 0.14
Table 1: Best Performance for Various Methods
Implementation Details We take the code from 23 and modify the training procedure according to individual gradient descent. The Neural network Architecture employed is U-net as in 23). To
'The code is available at tt/ttts: http:shtbb..oo/pppaaayydd/////aa

---

be able to make direct comparisons to DDPM and rudimentary comparison to CBDM, we modify the code of CBDM and employ error networks for mutual learning (individual gradient descent) instead of score networks as above. We run both CBDM and Individual Gradient Descent(IGD) for Ttrain = 60k training steps. We generate 15k samples per class and make the comparison at the 60k: training step mark. We provide FID, IS across various parameter settings in Appendix
Comparison with baselines The baseline model for is DDPM models trained individually on each class label dataset (3 = 0). We also make a comparison of mutual learning with Class Balancing Diffusion Models. While empirical experiments on CIFARIOLT shows Mutual Learning perform better than CBDM, we do not make any claim such as mutual learning outperforms CBDM. Since our contribution is theoretical in nature, comprehensive numerical comparison with CBDM is left as a future direction.
7 Discussion
Choice of X(1) and w(). We choose w() as an increasing function of / (as in (23) and X(4) such that A(e) is non- non- aon-nncrassig in t. The motivation behind this is to ensure that the training process gives more weight to fitting to the data distribution for smaller 0 < t < T and give more weight to t < T of the forward diffusion
a,
the mutual learning objective for high noise regions i.c. larger process. There might exist better weighting function. Our analysis doesn't 't involve the investigation of an optimal weighting function. We leave this as a future direction to pursue.
Bound on Approximation Error L (0))) Given universal approximation results for two-layer networks of Random ReLUs such as [[]] Theorem 3.6] and assuming V log (2,, ly) to be Lipschitz continuous w.r.t. Ft, we can follow [II] achieve an upper bound for maxyey L (0)). This bound can be made arbitrarily small by controlling hyperparameters such as bound on RKHS norm of 8r,0 and 0 < < 1.
Extension of Theoretical Results to Deeper Neural Networks Following 22] which proves that stochastic gradient descent (SGD) can find global minima in Deep Neural Networks (DNN) in polynomial time (given that the inputs are non-degenerate and the network is perrpaaramtterized), and [32), which extends [2] to determine the training complexity for diffusion models and determine the generalization eπror of sampling with DNNs, our theoretical analysis can be extended to Deeper Neural Network architecture in three steps. First, we can use [3 to obtain the generalization bound using Rademacher Complexity for DNNs with ReLU activation function. Then, using the fact that (r 0-y) = (By) + 33 3. (8y. 0-y), we observe that (32 Lemma 9] proves the semi-smoothness of (m,((O,) with high probability. Thus, we can use
2 Theorem 3] to obtain
the semi-Smoothness of Li (0y, 0-y)) The only thing that one needs to compute are the various hyperparameter dependent constants. The final step would be to derive a PL like inequality as in [2] Theorem 3] [32] Lemma (Appendii D.1)] with high probability. Proving whether Crag (8y, 0_y) satisfies PL like inequality is challenging. [32, Lemma 1(Appendix D.I)] considers the case without mutual learning. Even though _ (0y) and (m f-y) individually satisfy a PL like inequality, their sum may not. We leave this as conjecture for future work.
Application to Federated Learning. Consider the following scenario, each class label y E y is thought of as client that holds private training data with variable number of training sample points. Individual Gradient Descent then represents local training of score network with global sharing of updated score network parameters while preserving the privacy of local client data. This allows fair learning and generalization among all classes and prevents overfitting (memorization) for class labels with low training data frequency.
Limitations. While we achieve bound on the worst case generalization (sampling) error, the current analysis should be extended to provide insight into whether the performance of the head class score networks preserved upon adding the mutual learning loss. Further, we set Q = Uni f f)r and further investigation is warranted on the effect of the distribution Q on the worst-case sampling error. It is worth examining if generalization (sampling) error can be made arbitrarily small (also noted in 32 Section 3.3]) i.e. the (11) bias be removed. Finally, while we support our analysis with empirical experiments, validating our findings on larger real world datasets CIFAR100LL and a detailed comparison with CBDM (23 could further strengthen the approach.
10

---

Acknowledgments and Disclosure of Funding
We thank Mainak Pal for his help with the numerical simulations. The first author was partially supported by ARO grant W91 1NF23100666. the second by ONR grant 13001274, the third by NSF under Grants ENNS2233109 and DMS- -500260, and the fourth by ONR grant N000142312604.
References
[1] Amira Alakhdar, Barnabas Poczos, and Newell Washburn. Diffusion models in de novo drug design. Journal of Chemical Information and Modeling, 64(19):7238-7256 2024.
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via wwerpparametriizationn In International conference on machine learning, pages 242-252222222222222222 PMLR, 2019.
[3] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint orXiv:21/2.03122 2021.
[4] Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d- linear convergence bounds for diffusion models via stochastic localization. arXiv preprint WxXiv:2308.03668 2023.
[5] Kaidi Cao. Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with deteeddiiiiiutionavveeeee margin loss. Advances neural information processing systems, 32, 2019.
[6] Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. The complexity of computing Nash equilibrium. Communications of the ACM, 52(2):89-97, 2009.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[8] Hengyu Fu, Zhuoran Yang. Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory. arXiv preprint arXiv:2403.1190 2024.
[9] Zhiye Guo, Jian Liu, Yanli Wang, Mengrui Chen, Duolin Wang, Dong Xu, and Jianlin Cheng. Diffusion models in bioinformatics and computational biology. Nature reviews bioengineering, 2(2):136-154 2024.
[10] Pengxiao Han, Changkun Ye, Jieming Zhou, Jing Zhang, Jie Hong, and Xuesong Li. Latent- based diffusion model for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2639-26433 2024.
[11] Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. arXiv preprint arX1v:2401.15664, 2024.
[12] Daniel Hsu, Clayton H Sanford, Rocco Servedio, and Emmanouil Vasileios Vlatakis- Gkaragkounis. On the approximation power of two-layer networks of random relus. In Conference Learning Theory, pages
PMLR, 2021.
[13] Weipeng Fuzzy Huang, Junjie Tao, Changbo Deng, Ming Fan, Wengiang Wan, Qi Xiong, and Guangyuan Piao. Rényi divergence deep mutual learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 156-172. Springer, 2023.
[14) Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging con- trastive learning. In International Conference on Machine Learning, pages 4927-4939, PMLR, 2021.
[15] Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, lIker Hacihaliloglu, and Dorit Merhof. Diffusion models in medical imaging: comprchensive survey. Medical image analysis, 88: 102846,, 2023.

---

[16] Jianing Li, Chaoqun Yang, Guanhua Ye, and Quoe Viet Hung Nguyen. Graph neural networks with deep mutual learning for designing multi-modal recommendation systems. Information Sciences, 11911, 2024.
[17] Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties of diffusion models. Advances Neural Information Processing Systems, 36:2097-2127, 2023,
[18] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Open long-tailed recognition in dynamic world. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(3): 1833))1836-18111 2022.
[19] Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 3-17. Springer, 2016.
[20] Christos H Papadimitriou, Emmanouil- -aaaanouil-VVaileiioss Vlatakis- Gkaragkounis, and Manolis Zam- petakis. The computational complexity of multi-player concave games and kakutani fixed points. arXiv preprini arXiv:2207.07555 2022.
[21] Changhwa Park, Junho Yim, and Eunji Jun. Mutual learning for long-tailed recognition. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2675-2684, 2023.
[22] Wonpyo Park, Wonjae Kim, Kihyun You, and Minsu Cho. Diversified mutual learning for deep metric learning. In Computer ECcon-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 709-725. Springer, 2020.
[23] Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang. Class Claas-balancing diffuusio models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18434-18444 2023.
[24] Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang, Class-Balancing Diffusion Models (arxiv version), 2023.
[25] Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimiza- tion with randomization in learning. Advances in neural information processing systems, 21, 2008.
[26] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denois- ing diffusion models for multivariate probabilistic time series forecasting. In International conference on machine learning, pages 8857-8868. PMLR, 2021.
[27] Jie Shao, Ke Zhu, Hanxiao Zhang, and Jianxin Wu. Diffult: Diffusion for long-tail recognition without external knowledge. Advances in Neural Information Processing Systems, 37:123009 123031, 2024.
[28] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021.
[29] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
[30] Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the fairness of deep generative models without retraining. arXiv preprint arXiv:20/2.0484 2020.
[31] Lan V. Truong. On rademacher complexity-bast based generalization bounds for deep learning, 2025.
[32] Yuqing Wang, Ye He, and Molei Tao. Evaluating the design space of diffusion-based generative models. arXiv preprint arXiv:2406. 12839, 2024.
[33] Divin Yan, Lu Qi, Vincent Tao Hu, Ming-Hsuan Yang, and Meng Tang. Training class- imbalanced diffusion model via overlap optimization. arXiv preprint 1082,, 2024.
12

---

[34] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2244.1390, 2022.
[35] Tianjiao Zhang, Huangjie Zheng, Jiangchao Yao, Xiangfeng Wang, Mingyuan Zhou, Zhang, and Yanfeng Wang. Long-tailed diffusion models with oriented calibration. In The twelfth international conference on learning representations, 2024.
[36] Ying Zhang, Tao Xiang, Timothy Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4320-4328, 2018.
[37] Haojie Zhao, Gang Yang. Dong Wang, and Huchuan Lu. Deep mutual learning for visual object tracking. Pattern Recognition, 112:107796, 2021.
[38] Yuanzhi Zhu, Zhaohai Li, Tianwei Wang, Mengehao He, and Cong Yao. Conditional text image generation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14235-14245, 2023.
[39] Hao Zou, Zae Myung Kim, and Dongyeop Kang. A survey of diffusion models in natural language processing. arXiv preprint 4677,, 2023.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] pprovided proper justificaiion is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We discuss under related works, the scope and how our work differs from those in existing literature. Theorem 1, Proposition 1, Corollary along with numerical experiments reflect the theoretical contribution of our paper and support the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
14

---

Answer: [Yes]
Justification: In discussion section, we highlight limitations of our theoretical analysis.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
We clearly state the assumptions we make and cite references that use existing literature. We provide proof sketch of the Main Theorem of the paper. The complete proof and supporting lemmas have been cited are provided in the suppleman- tary material.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
15

---

We provide the parametr values and provide additional graphs in Appendix C
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide github link to the code within the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce)
www.sabiissioooooooooooo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips cc/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
16

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: We clearly state the dataset chosen and the hyper-parameters used for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bar 1 standard deviation and provide the experimental setting in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
17

---

Justification: We provide the details of our computing resources in the supplementary material in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:://euurpp:: https///eeerippcccppbbib/ee httbblic//hiiccsuuddell........
Answer: [Yes]
Justification: We confirm that our research is in line with NeurIPS Code of Ethics. We perform our experiments on synthetic datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We point out our abstract, introduction and discussion the broader impacts of our work to privacy, copyright related issues in generative modeling and existing literature on diffusion models for Long-tailed generation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
18

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We provide the link to github repository where include the code used for the numerical experiments our paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We modify the code in CBDM [23] from their github repository and we provide link to the github repository for our numerical experiments. We use the probability flow ODE sampler used in 17 and we mention this the main text under numerical experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, wwwppsaiitthooeooo thcode.hoc.eco com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We use simple synthetic data verify our theoretical findings along with experiments on empirical real world datasets such CIFARIOLT. We provide link to the github repository where our code is.
Guidelines:
The answer NA means that the paper does not release new assets.
19

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowd sourcing experiments or research with human subjects were conducted for this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The numerical experiments were performed on synthetic data. There were no subjects on which experiments were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if is an important, original, or non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM wasn't use for any task during the development of the research or the preparation of the manuscript.
Guidelines:
20

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https://neurip:: /ttp://pp.ccc//offfeeeee//////// for what should or should not be described.
21

---

A Appendix supplemental material
Notations
Symbol
Meaning
Number of neurons in hidden-layer of score network
Cu
Upper bound on ||wg,sll |lug,llli
F7
Upper bound on Ex,, Be, (Wo + (e(t))1
oostsT )
Ly
Creg (0y, 9-y) is T smooth w.r.t. By
Qy
Lipschitz constant of C (0,, 0-y) w.r.t. Og
0g
Lreg (0y. fyy)) is By Lipschitz. in Og
B
Upper Bound of the Frobenius norm of Ay
B
Sampling
Denote the backward time schedule as JosijN such that 0= 45 <... Lower case ps represents the density of P,. We consider the exponential integrator scheme for simulating the backward SDE with
< T a.
The generation algorithm can be expressed as a piecewise continuous-time SDE: for any t e [t 14.))
dY, (Y, 2sT-17 o, (8 ))dt + V2dW,
(21)
Denote Gr Law (Y,), Vt --..
Theorem 2. 4 Theorem 1] Let Assumption Thold. Then there exists numerical constant Co > 0, such that
Deu (-a((a))))lrr--a
Co + ED Er)
(22)
where ED S Nu2 + Tugy is the discretization error due to the reverse SDE, EF S exp (-2T)) is the error due to the convergence of the forward SDE and Eg is the score estimation error
Es(O,))
Σ 5=
|| [
loggr-I (xly) - ST-15 -005)-5-15-00 0,
(23)
where = 15t1 -,,, Vj o... is the step-size of the generation algorithm.
When the training is done over the forward discretization given by (IN-j 7 T 1/ N-1 we have
N-1 JIN .VN--D) (tN-j (N-j-1)Exx Bx Xew Noo 1TeN- (N (New-s) + Ell
Es
A(tN X(tN-j)0ON-s
5=
N-1
Oiy M(1x X(IN-))(tN-) - IN -NN-
+
X(tN-j) 1=
JIN-s (A,,)
mamax
A(tN-j)
where
1 N-1
(9,))) 2
Σ
Ex, x x 180 X(tN=s)(tN-) - -N---11
Ex
1=0
V Vz((x=z) log Ptn (a(tn)))((-))))) Sex By By (x((N
(24)
N-1
(1)
22

---

Theorem 3. (Appendix Band A Theorem 11) Let Assumption 7 Thold. Then there exists a numerical
constant Co > 0, such that
D
& Co( + ED + EF)
(25)
where ED S NuZNu + KT is the discretization error due to the reverse SDE, EF exp (-2T)u is the error due to the convergence of the forward SDE.
B.1 Decomposition of (6y)
Let 0r - argming, (0y)). We further decompose L (0y) as
max NEY
(*((,) -C*(0))
S max DEY (O,() + mut
+ (0,, 00-) (CO(0)) + MCmme -yy)))
+ B( -B( (0y, 0-y) - -ty 0-y))
(0) max yEY
(00(0)) (C:
0-00000)))00 (C (C*
+ B. (B(0- ((0-))).-)))) + max L (B(6-y), yEY
< max yEY
(ieg (0y, 0-y) -
(B( (B(O-)),0
+ β max sup yEY (0,,0-y)
C (0y 8-y)
where (a) follows from the fact that (Bcg this to obtain an upper bound on ma min, L (bt)
S CVey(0s) 0-y). We further decompose
max (8,) S max (8,) + max (o (B(0)))))) yEY
yEY
yEY
+ 8 max sup Cy (0y, 0yy) yEY (8,
(0)
max (0)) + max yEY LT (0,, 0-y) CTeg( (B(0 (00-): -):0 0-y)
Cy (0,, 0-))= - L (0y, 0 0-y)
yEY
+ max yEY
max yEY
C% (B( y), 0 0_y) ) Ereg (B(0-g),
+ 3 max sup Ly (0y,0 0ny) yEY (0,
(6)
max C"(OO) + max sup yEY
Cy Ciee (0, 00y,0-y) -Cre (0y. 0ny)
yEY (0,
+ NE-gapp6,, 0 f-v) + B max sup (0, 0_y) yEY (0, a)
(0)
max (0,)) re[rnain] yEy
min
<
max L (B,)+ yEY
+ max yEY (0,)
sup
C Cheg (0,, Ereg (Oy, 0-y)
min
NE- NE-gap 0r 0-,)) + 3 max
sup L 00,
rE Ttrain]
NEY (0,
23

---

where (a) follows from adding and subtracting the empirical losses (0y. 8-y) and
Creg (By(0)) ( d-y) and using triangle inequality of the max norm, (b) follows from the gradi- ent domination property for strongly convex functions, (c) follows from taking the minimum over the iterates of the algorithm.
B.2 Boundedness of Forward Dynamics
Lemma 1. Consider the forward diffusion process with linear drift coefficients. For any 8 > 0,8 < 1, w.p. (with probability) of atleast ô. we have
2 (5 (Ilo00)l + | log nt²
|0((1)l s cr
SC
(26)
where Cr == max r(0),(())())))
Proof: The proof similar 17 Lemma 1] When the drift coefficient f(,tt: R" + R is linear in i.c f(x,t) f(t)z, the transition kernel Pεjo has a closed form
p1)(z(())=(0)) (((((())
(27)
where u(t) = exp (1O f(E))E), = 2 lo exp (2 - 2s. Together we get, xxt) = (0) + (()22, ~ La)
(28)
For any e ~ NO, 1),c c > 1, we have
1
2
Pfc: |c| > c) =2
dx S
2xe T dx
e dx
----
2m
2
V2m e
π
(29)
Let 8 = - then
P(e: |e| S Vlog log nt2
2
} > 1 - 8
(30)
Hence, for any 5 E (0,1) with 5 < 1, w.p. at least 1 - 6, we have
[x(t)l SCr [1a(0)1l>> + log πθ²
(31)
where Cr = max.eja.r Let =
CT(K= v 1log 707
B.3 Boundedness Loss function Eieg (0y, 0yy)
In this section, study some properties of the game defined by ()), (Creg)eEY, (Oy)vev). From Eq.
we have
C Ca
(0y, 0-y) = Choomsi( + 3 3.% (0,, 0y)
(32)
where
I ((t))Ex(0~m) Byy 11000, (z((1) 8e,0, (c(0)]] dt
L mon (By, 0 (0))))
/0
and
2 / ((())) [[Veco (og)( w>)1 dt
Cu (0y, (-)=
Conditioning on Xo and using law of iterated expectation, we can write [2 Appendix A]. we get
Coomtiree (0y (0y, - (' Ex, Ex,XXo x(t) ||se,, (z(t)) Va(() log lo
1
1- + (w(() ~a (((e) (x(t)) se,, (1(EI) dt + 22.20 (xCe)C(0)))
24

---

where Ci(y) = Ex, Pi px Ex, No (V u)1 V Vx(1) log (=( y).
to learn the score
Furthermore, we discretize the time points 0 = to < < < <tN to the objective function
Feeg = (0,)) - 3L (0y, 0-y)
1 s(t))(t) -t, ( -xx,
|| V log (2)((()))) -,,,00 (or(t)))
Ex Rx,, 1N0
j=1
+ +C(1) + 8
1
Σ w(tt)(tz - - Ex
N
(1,,,,))(())) Eo sr.0, (xx(tz)) 2
j=l
(33)
where ((y) 1 Mt,
- Ce, (v) From
32
Appendix A], we have X,
N(e t4 Xo, 02 and its density function is
| e 14 1120l
pr(x)20)= = (2TT,)) $ exp(
20,
Then,
A Ex,
st, 8, (x:(t)) - Va(e,) log (2 (x(t))(=oo)]
E t |
Xoll
2
Ex,,Ex Xx
Sty,0, (x;(1) - V.
20
X₁ Elee Xo
2
-Ex, Exx,Ex,1x0
56, (x. (t,)) +
-Ex,Ee Eee
St (wi(()))
Let E 5. ~N(011
_ Ex Ee oorst, o, (xx(tz)) + E||
A
(34)
ōt
Finally putting all of it together, we get the empirical loss function
Ry
N
X())(1) - (4-1) (1,,)t),,,(3:((5)) + Eiill2
C (0 (0,, 0-y) = 2ng
(35)
+ (w, - t 1) v [1ss,,e (1;,,,,(x; (x; Sn (x,
We will show that the empirical loss function for the label y E Y, Lie (Oy, 0 0_y) that is optimized convex and smooth in By with high probability.
Lemma 2. For 8 > 0,5 & 1, up.1 - N5, the empirical loss function
ny
X(t,)(t, tz-1)
Che (0, 0-y)
|6,,,5,, (x,,,t,,, + Essll
2m i=1 1=1
σt,
(36)
+ Bw)() - -( ~ ( (x:(()) sn, (z:
bounded i.e.
A(tz)(t; -1--1) + Bw(tt)(t) - -t--))
(0, 0
01z
25

---

Proof:
From Lemma 1 we have 0,0,8 <
2 T52
V
S
(37)
Thus, w.p. - Rly Ny Nô, we have |Etyl S V and] -,,,, N Thus, w.p. 1 N6
2 x8z
and hence we have | S CeNis Vi wi=1, 1, ,by
ngy Cre (0g 000 2nly
X(1)((1;4) -1--1)
|| (,,,,))),,( + E0ll2
Tt,
5=1
+ (w(tt)((tz tj 1) /m@ (2,,,, - s1, (1. (z. [1]
1 By N A(tz)(t) - 1)-1) nyy
(0 || (x,(t,)) ||6,1l?)
02,
i1j=1
(z,(t))) tt--))
max sy (z. (t,)))
For a bound on (x(4)))
m
st,,,, (z(t,)))12
a a ow + ((,))(t))))
(38)
m
IA3
1
|agilll |a( ((ty) why et))l
(39)
m
IA
| 2 (/wyalll |I= || lle(())l
(40)
1 (Cew o [l + max [le(g)]l (1
(41)
(Ct,, C B
(42)
where (a) follows from triangle inequality for norms, (b) follows from the fact that the ReLu function satisfies (axx)] |x| and Holder incquality and (c) follows from the bounds the embeddings and x(tt)) with [ l /ll. & Cw, Vi [m]] Thus, for > 0, 8 < 1, we have w.p. 1 n,,
N X())(11-44 -4-1) Bw(tt) tt-1)
Che 00y0 SC
(43)
0t,
where C (ot + 2), Cty.e) ug max, oz, N2N-
Since oG, is non-decreasing j. so B² Z8.-
26

---

B.4 Boundedness of Gradient of Loss function Ling (8y, 0 y)
V Freg (0,, 0-0)
V (Av)a Creg (0.
s(t)((s) - ( + (a)) + Uge(t)))
k=1
+Bw(t,)
t-- -ttp:///|(gosoeooooo---/////////
s
(tt
By
+2
(t,) - ((,, 0,,00 (x(t))))
Ty
k=1
1=1
+ [(W,r(()+U,e(()))
nly
N
d (4 - -4-1)²
Σ
1 (z((4) + Exy (WW,)(()))) + (1))))
fl
1=1 j=1
1=1
N
+2,3²
w(,) - 11-1)²
nly
d
Σ Byr 1/ 10, (a((1)) s1,00 s,,0,, )10 (W + Uye(()))1
s <Nyalo((W,a(tt) + (t max - t,- (w(t) b (0y, Q_y)
<ANd + Ctv,e) Rey.e, max (A(4)((s - - (w(( -1--1)) 1 (0y 0 vyy
Since w.p.11 - N6 the empirical loss function Creg (0y, 0_y) is bounded, | A. Creg (0,, 0 n is bounded with the same probability.
This also shows that for fixed 8-y- (Wy. Uylyey, w.p.1 Lic f-y) is Lipschitz function in by with Lipschitz constant oy such that of = Nd
2 Ce max, (A(L)((1)-
t)-1) (t) - -)- Σ N
E
Aic, ( + (w(t)) - tz-1))
9,
B.5 Smoothness of Loss Function Cheg (0g, 0 0-y)
Lemma 3. Let (W Uy)UE, 0 0-y: (t be fixed.
Let by d(Ciw.5
22 Wy Ex (x(t)((s) - t + Bwltt)(tz - ))-1)).
Then for 0,6 < 1.
w.p. 1 - N6, Che (0y, Q_y) is !ix smooth and convex in 0y.
Proof We have,
N X(1i(1) -4--1)
Dny (0,, 0-))
11,1, (x((1)) + E0
2ny
0t,
1=1
(44)
+ (t) - BB ~
(,(,,,((()))) - su,, (x(t))
27

---

To show smoothness, we will show that the function f(e,) 1 se, o, (a(t,)) + Eis 1 and g(0,,) |1 o, s,, (z((t)) 2 are individually smooth. Once prove this, it is easy show Creg( (0,, 0-y) is smooth as the linear combination of smooth functions is again smooth. To show smoothness, we need to show that ((,)) and V [ have a bounded norm. Recall that (,,0(()) (W (t) Uye(t). Let (2, t) == oW y + (22(t)),n2(,; t) Sr,0y (1)); hag(i,)) Eij. have
2
((0,) - 10 0, 0, (x((1))) + E
(45)
Ay 52 Mi (z(tt) (24))))))) 14) (z())))) 20, m
h ((()(()))))
(46)
+ hi (1, haa(t))
(47)
20,, trace Ay Bz) + constant
trace Ab A,
(48)
m²
m
o
x Tyvect (Ay) 2ō,, vecl B3 vec(AA)))))))))))))))))))))))) + constant
vec( (ey(AA))) (y)) (B,
(49)
m
where (a) follows from the identity x.' Ay = trace( Byx* ). (b) follows from the following identities
trace( AB) -trace -trace(AAAA)) = vecl ( ()))(AA)
trace = =ec((A)) bec(B") =ce"
and B3 = (,( (i.5)- Similarly, we have for g(0,))
g(0g) = $1,0, (x(tt)) sx,
Ay hr (x(stt))
m2 hT (on - 2h (x(s)))
+ (x((()) (x)))
trace (A A1) 2
trace( (A, constant
771
m
1
vec(A,) A²) (B1 T))) vecl (cc(BB) vec(A) constant
m²
where B := hy (x((t)), 12)h /h (x (r((t))) (2) and :=
= h1 , V2 f(0,) v0,,(0,)= = V Veee(A,)9(1)) = m² B x I)
(x((t),, ty). Thus,
1
(50)
The eigenvalues of [ X 7) is the same as B2 with multiplicity. Thus, to show smoothness, we need bound the maximum eigenvalues of Bi. For any m
0 T Biu (v he (x((t) 2)) s (1g of
(51)
Now,
(W(Wx + Upe(t,)))ls = max aw (f xtt,) ) + T c(t))
(52)
s=1,
< max Iw T 2 (1,, + T e(,))
(53)
s=l...,
< max
||wgssh [|=(t)))l + || [|e(t))ll
(54)
1= I... mm
< (Oiw + Ctxxee
(55)
28

---

Thus, we have for any E R
0 os Bru S d(CtNv + C
Cuy, oF
(56)
Since w.p. 1 1- No we have {llasillo S Ccv,8) ) M and g(0,) are smooth in Og for every Wy, Uy, x(,) 0-y.
we have with the same probability f(0,))
(X(e)(ss -0)1) +
Thus, Ereg(0n, 0-y) is Ly - - m d( + O u. Σラ
(w(ts)((z - t)-1)) smooth.
B.6 Proof: First order convergence of the algorithm
Proof Our proof follows closely along the lines of 13 Let Crēg (0,, 0 0-y) be the empirical version
of Leg (0y, 0 O-y) with By samples. By Ty smoothness of Lreg (0y. 0 yyy we have, for any E V,
Cre (0 0 ,,, S Greg (8) 0 y) + (Vo, Cng (07, 0 0-y))
07)
Ly
(57)
10 671 l|
Creg (0) ,
fre (0 0 , 1// || Vo, Lre (0T 0 ,
L 2
(58)
+
1
Vo Erv 10 0- ,)1
Thr Vog Prgg 00 0 01))
< Lreg 0r 0 0yy
City (0 0 y)
Ly 2
(59)
2 Vo, Eny (0) y: 0- 11
+
T crain
Tirain
Trr Ve, Creg y.
(0 0 0 )| S Er (0t) Ln (0F+1) 1) + 3
20, y
y
(60)
T Tirai
Lg
2 n Ve, reg y, -_ m)
7=1
T Terain
Tereia
Ly
17 Ve, Crog (87, 0 ,1,)²² s -h (0)) - L' (05+)) +
2 1 y
(61)
Terai
B
110T+1 ,0 0_
Ln (B,) - L (8+) + Lu o Σ T Terein 1/2
min rE Tevaii)
(0 0 11,)
2
=1
Σ Etrriin Tr
(62)
E
Tirain ((0 ,r, RT,))
+ B
Σt= 17
where (0 0z,) = - (0,, F2y)= - Lna y (0
B.6.1 Analyzing the Bias Term
Lemma 4. Suppose p_y. (W y. Ug)EY are fixed.
Let by d dL.N(CCiN.
(cv.e)>uu,, B max, (tt)((t) - t-))) Then 8 > 0,8 << 1, w.p.1 N5, we have
N
(l))(1) - 4 1) 0,, (x((T)) - rr, (x(,)) [1] (63)
Ehree (0,, 0-w)=
2nl
1=
is Φy Lipschitz in 0y.
29

---

Proof:
V Ay Eegg.mu 00, - = V (An)e Emin (0, 0-0
Σ
h=1
Σ ny
(tt)((t) -t-- [[5t, My (a(tz)) 0, (alt))))o +U
N
Fly
N
((t)) - -t
Tly
Σ By 51
(x(t;)) (,,,t - By (z((t))) /| (1( + U,e(t,)))
(o Uye(tt)))l Nd Nd tt- 1)-1) (0y, p-y)
s N 5 + P m max to (tyy fyy)
- NCi + Ctv.e) wwn, B² max (())( ) Σ (t 1 1
Since Ly (0y, 0-y) Erny (0y. 0-y) and w.p.1 N6, Ling (By, p-y) is bounded. Thus, IV Emeut (0y, 0 0y) is bounded and hence Lmuu (0y, 0-)) is Lipschitz in 0g with dy = d (L + Cxx.e) PC uy B Bmax,w((,)))) (t, 1)-1) Here,
(0 . 0r 0- = Cmicc (0 0g, - (0 0 109+11000
(64)
say 10 -0 0g+1|
(65)
SoyT || A. CTee 0-ni]l
(66)
(67)
By taking 1/r max Lyy Tiraii
m²
Vy y
2(L"" (09) Ln (07))
max min vEy re r((raa))))
Ve, (0 0- ,11
max O yEy
30y
max Ly / Terain
VTtrain
(68)
m²
+ 3
(69)
T TTrain
For (0y, 0-y), we have
max yEY
my reg (0y, 0-y) Crey (B(0=g), 0-y)
(70)
< max
V 0, Dry (0y, f 111 11, - B(B 2111
(71)
Since the strategy space for 0y is bounded in norm. We have
NE-gap(O,, 0-w) x max yey
V Eyy (0y, 0 022
(72)
m²
min NE-gap(0r 0r = 0 T + TE Traca
(73)
30

---

B.7 Monte Carlo Error of the Finite Neural Network
Observe that
N
(0,) \(tz)(tz ty 1x Ex,,,,,, Va(e,) log (z)(())))) - s, (2, (2,))))
2
N
+ \(tz)(tz - -,,
2
X(tz)(tz +)-))) [1s,,,, (x(t))) V logpt (x(tz))
For each E V, L (0)) is the optimal loss function for the unregularized version under the current hypothesis class. Let (u(6r) be the optimal unregularized loss function under the continuous version of the random feature model. Then,
- (0,) Σ -t ()) ([ V. log (x(pt) (0(t
(74)
N
[1 (x())) (z(()) - V (ogg)(()(()))
A(tz)(t) -t-11 -b)-1)En(t)
(75)
2
N
2 ((t))(t, (a:0(tt--t111-0000 -t-1))n((e))pp || (x(())) s, (20(1) (a(tt))][
(76)
j=1
(2C) (r o; (t (X(t))
(77)
Proposition 2. Monte Carlo estimates. Define the Monte Carlo error
1r ((()))))) Σ ((t)((ts t)-1).
(78)
Ex(t [136,,0((0000) (x(()) - 00 (x(1,)) (1(())))]
Suppose that [[X S K and the trainable parameter and embedding functions W, e(.) are both bounded. Then, given any 0. for any 8 > 0,6 < 1, with probability of at least - 2N8, there exists 0 such that
ErT (
2C2 u. (2 + 2d2 log
2 ((t))(t,
Σ
N - -x-1)
5
m
(79)
Proof. The proof closely along the line of (17]. Fix any 0. For notational convenience, will drop from By and fy. For k = 1,2, ... d, define
Ze, (W, = (15),.((0)
5,,0,
E
[sL.)A()) - (x)
(80)
an x + uf e(t))
B(a,w)lan (w) ww nf e(t))]
1=1
(81)
31

---

Then, we have
2
Exopp
(x) sr,,, (x)
B Berpi (110)1((()) - 82,
(82)
Z (W.
2
(2,+(W Ear--EEwulee [Z + |B (Bw.olZe.t 12 ( (U]]
(8)
(2,A(W,U) Ew: (W,,
k=1
Ew.u/ (2,,
(83)
where (a) follows from the fact that (a+ < 2(a² + +b² and Jensen's Inequality E² (W, U)] <
Ew K,,,W, (W,U), U) According to Lemma for any 8 0, < 1, w.p. atleast 1 5, we have
| Cix.6
(84)
If (W, Ù) is different from (W, U) at only one component indexed by i, we have w.p. 1 - 5
(W,U Z,, Ú)
(85)
[10)0.(())= s, (x) (2(p)) - ()..ā,(())) (, (P(P))
(86)
(0)
(1) )1)x(()) (2())) (e()))
(87)
|alko(wf I + uf e(t)) x (ffe((t))
(88)
[ar,i| [w + uf e(t))) + | lo x + पن (e((t) ((t))llz())) pmw
(89)
|ar,el | x + uf e(t)) (t)))l + | ( + uه e (o())
(90)
Gex,s + ||ullla [le(())lao)) [ [bll Cey. |1 lle(()llz)
(91)
172
(e) 2 m
BC.,,C Ctx.s)
(92)
where (a) and (b) follows from triangle inequality ||al|| |b||| S ||a bbl and ||a bll < ||||++|b|| (c) follows from the fact that [a < lyl. (d) follows from Lemma I and Holder Inequality, (e) follows from the bounds on ||w/l| ||,]|,, , (exa),e(tt).
Thus, w, p.1 (W, U) has bounded increment property. Using McDiarmid's inequality, w.p.1 - 28, we have
B B CC + Cen-e) dlog 2 m
Z (W, U) Zw,
(93)
32

---

Now we compute
Ew.ul
Z2 (W,, U)]
-EEu
(x)
(b)
Exwp.
m
(as x + ut e(t)) E Bw, u))( x af e(())) $1
Pempp
Bw.u
m
Rwwpi m
Σ
(a x + of e(t)) F Fu 1)
(ag x + uJ e(()) Ex, (wo x + u² (=e(()()))) E(()))
(c)
Exwpi BwEU
Σ
(airo(wf uf e(t)) - E (u, v('()))))))
m
1=1
Σ
Burnin (ci:00(wf x e()) - wo (e(())))
+ m12
Euy.ay .yy (a)to(1off x uff((t) (Ew,laas ( +Ue(()))
(d) 1
EWA (aiao(wf x uf e(t)) Em,ulan( (w, uw u (())))
Exrpr
m²3
(6)
1
Ez-pp
E
Enwy
x ot ((1))
772
(
]
Ex-pp
Ev (lay, (i, Cex + ||u|l, [e(())))a)))
1
$ m
(Ctv.6
Ce B²
where (b) is due Fubini's theorem, (c) is due to independence of sampling (w,, uz) and (wy,, u,), (d) is due to (w x + uz] e()) being an unbiased estimator of the continuous version of score network, (e) follows from Var(X) < E[X"], (f) follows from (b()) S ly/ and Holder's inequality, Thus. w.p.1 25,
202 B2 o2 log 2 (
Eaen (s.,,,1)) (x) - 5,,8, (xr) 1 < w.u
(94)
Finally, we have w.p.l -2N6
Errmo(0,0, (, 202 (e, 2d
N ((--1
log (
Σ
X(1)(((
1=1
(95)
B.8 Radamacher Complexity
In this section, we will bound the term related the generalization bound
sup (0,
(0 P-v) 0-w)
L.e, (8,, 0-y)
(96)
33

---

The Rademacher complexity of real valued function class F is defined as:
1
Rr (F) = En, dn E Eor.
sup Σ o fEF i=1
(97)
The variables di, Fm are iid Bernoulli random variables that take values [+1,-1] with equal probability and are independent of "1, 2m. However, for our random feature model, we have vector valued function class
{ f(x) (xx W.
Fw:U- =
Σ oro(1, wk, ux ||Al| = B
(98)
Theorem 4. 19 Theorem 3] Let X be nontrivial, symmetric and subgaussian. Then there exists
constant C < oc, depending only on the distribution of X, such that for any countable set S and
functions v.. S 0. 12, S i S n satisfying
ss, E S, v.(ss) (ss)) & [lx(s) o,(s'))l
(99)
we have
Esup #ES
ccbi(s) I Cupup E (s)oi(s) sES i,k
(100)
where the Xak are independent copies of X for isis n and k S and (s($)) is the k-th coordinate of (b(s). X X is a Rademacher variable we may choose C = -2, X is a standard normal C=
Corollary 2. 119 Corollary 4] Let X be any set, (x1,...,, ,In) E X" let F be class of functions
f:A + l2 and let h4 : l22 + R have Lipschitz norm L. Then
E Sup
Σ Eehhe((a))) S V2LE sup FEF
Σ
(101)
fEF s,k
where eq is an independent doubly indexed Rademacher sequence and fü (x;) is the k-th component of (x;). Lemma 5. [19] Consider the function class J (x in
Ao(a,WUU) W,U): A B), S B}.
Then the empirical Rademacher complexity of is
B vm
Rad, (F)
T. sup fer
Σ fa
Σ
10(x, W, 2
(102)
Moreover, Ef 1|,(1, W,U) C2, the Rademacher Complexity of F is
BC
(x)
(103)
v mn
Proof:
Rad, (F) = E Eup Σ
=
1
E sup
Σ (ak, Σ (k2D
(104)
fEF 1,k
|| SB
k
1
E
sup SAB
tr(D' A)
BE D 1,
(105)
m
where E B(H, R)) is the random transformation
(u.
Σ
(1
(v. Σ (xzi))
(106)
Thus,
D 11. E,
Σ
Σ Euo W.U)
<
|0 W,U)!??
(107)
34

---

Thus,
B
Rm (F) Exy." an RRdd S
Ery an an
11 W2
(108)
Vmn
B
E Ben an 110(2,, W,
(109)
nl m
BC
(110)
Vmn
Suppose 0 <h <.. < iN T are the chosen points of discretization for training, we have from
the forward process
X(1)
e e -(() + VI 2-22,,2 ~ 1)
(111)
e -221200) + 1-
= (t]x
(112)
2
1 - 2
E2 [X= (t,))
K2
<0<t, <T
(113)
2
Using the above bounds along with bounded support of embedding matrices W, U and embedding function e(t) and Assumption it is easy to show that
Ex, Be, (e(Wa(t)++Ueeee + (e( 2
PA. 4O <IST
(114)
for some constant F7 and x(() ===xx(0) + - ~ ~(001
Lemma 6. The term
N
2
(0, = Σ w(t))(ts - --- Ex. 1x Ey Eu 00,00 - (zi(t)))
j=1
(115)
o Fr Fr -1--1)
Proof:
Using the fact of bounded support of embedding matrices W, U and embedding function e(t), bounded strategy space and Assumption and eq 114 we get the desired bounded.
Lemma 7. Suppose La, oī BFT + Vdl log x Then, with probability the function
h: A Rd R
Mxx |ō,, + Eiil |
(116)
is Lipschitz in x, where A = {x E Rd || S Fr B}.
Proof. sufficient to show the norm of the gradient of hxx)) is bounded for E A. With
probability 1 - 8,
V V2((l)22 =ō, ||,, + Eullz S ST FT + vd log T82
(117)
(118)
Lemma 8. Suppose Icc, = 2F Bl) Define A C Rdd + R where
g(2 22, , ) E ||| - 2. 11, 11.1./ [1,22 .
(119)
is Lipschitz in where = (x _2 ||=||| Fr
35

---

Proof:
V "2. By [(x; - -xyy
(120)
2, """,,
".,,,,,,.,,,,,. mp1)) =2p(z))(x, - zz).J #
(121)
[V>g(>)]| V 21 .2, m3,37
Σl
V (01, "" mللil
(122)
j&t
<2B |||;; - Ty |1] + Σ
||0 - xzll - 2F
(123)
We know
1
N
A(tz)(tz - -1--1) Ex, Be, (x((t)) + =01
(0,)
(124)
2
==1
N ((tz)(t,- -ty-1)
+ 2 1=1
Σ
(125)
where C((y) = Ex, | pe.p((-ll))1 Eーー N
Ex, Vog pe ( y)1l?
Let C(y)
=
(v)
Lemma 9.
With probability Nnyō, an upper bound for the generalization gap i.e.
sup (0,88y)
(oy 0-y) (o 0-u)
(126)
2/2BFT Lcs
N A(t;)(tz - -t--1)
2/2BF 2212
N Los Ew(l)(() - -4-1) + (127)
/mnny
Tt;
Vmny
where Lc, = BFr + vd vd log 2
= - -F max Ic(y)]
T82
Proof.
Observe that, we can rewrite Eq. using triangle inequality as
sup LTee 0-g) - Ereg (0y, 0 0.v) Is sup 1C" L. (0,)) (0g,0-v)
8,
sup ,0, x)
1C (0, 0_y) Emiu 0-y)1
(128)
Further decomposing them, we get
N A(t,)(t, - -t--))
sup (0,) (O,(), -
sup LC - (o,(() + (129)
9
0,
1=1
where ICO(O,)(i)
(o)(O,,
=1
2n, D
110,, (x,(1t)) +Eyl12
+ E, 1'] and
Ex, Es, 10,, (e (e +
sup
( 0ny) -hh 0-w)l
(0,)0_x)
N Σ w(t))(t, - -t--1) sup
(2 0-))() En (By, 0-))() | (130)
1=1
(0,
36

---

where
[ (0: Lmut (0,, 0-y(()) |=1
1
By/ne/
| | (x=(t))){
se,0, (2:
2ny i=1
-Ex, Byr s No v1 c- Eg) (e (1-e-221)
(131)
Finally using Corollary
2
Lemmas
19
Section 4.1] we have
2/2BFr La Vmmmy
N
sup (0))) (0,))
X(4)(1 - -
(132)
8y
and
2/2BFr|D1
N Los
sup (0 .
S (0,, - Cnuue(0 0-1)1 S
Σ w(t;)(t) tt-1) (133)
Vinly
j=1
C Numerical Experiments
Computing resources. The numerical experiments were conducted on MacBook Air (2023) and Gilbreth. Gilbreth has heterogeneous hardware comprising of Nvidia V100, A100, A10, and A30 GPUs in separate sub-clusters. All the nodes are connected by 100 Gbps Infiniband interconnects. We used sub-cluster B with 16 nodes, 24 cores per node, 192 GB memory per node, 3 A30 (24 GB) per node, For more information follow this link.
C.1 Gaussian Mixture Models
Dataset We perform empirical experiments on synthetic datasets to verify our theoretical findings. The synthetic dataset randomly generated under the true distribution and fixed. We detail out the underlying distribution
case by case basis.
Implementation Details We employ the random feature model with the width of network m 16. learning rate 7/+ = 10 Vr, Tirain 5000 is fixed for Adam optimizer. We set A(t) = oz, (t) = e', total number of training samples is 50.
Case one We perform more empirical experiments on = 1, imbalance ratio r = 2.5, B = 0.01. We compute the KIl-divergence between the ground truth distribution and the learned model using the procedure in [1]. P(rly KI. divergence for the mutual learning
NN(H,, o²) and class 2 is P(zly = 2) ~ N(,, a²). We observe is lower than the vanilla when
Fig. 2 the worst change the distance between mean and the variance of each class label. The performance of head class doesn't 't worsened for small μ. However, the head class performance suffers for mutual learning case when the distance between the mean increases. This might be because when the support of class distribution are farther apart mutual learning is not advantageous as transfer of knowledge between the class is not useful.
Case two We now consider a case with two classes with imbalance ratio 2.5,8 0.01. Class I and class 2 is
itself is a uniform mixture of two Gaussian i. P(x y = 1) ~ ( We observe the Mutual Learning objective with our formulation
(4, -
P(rly= = 2) ~ (0, 2) as in Fig. have lower KL-divergence for both the classes compared to the vanilla diffusion models trained on each class. In this case, mutual learning allows useful transfer of knowledge between the classes increasing the performance for both. We hypothesize that under some notion of similarity between various class distributions, mutual learning advantageous improving the performance of all classes.
37

---

15 Ratua, clask
Vont C364
KL Divergence
RutAI CI95
KL Divergence
VO1E CRSS
14
KUUDA: daSS
MARINN
0.6
04.
Means
Variances
Figure 2: Case one: (Left) The first plot shows the KL-divergence for each class with and without
mutual learning objective as μ is varied. (Right) shows the KL-divergence for each class with and without mutual learning objective as is varied (μ fixed).
0.200
Gaussian Mixture
0.175
0.150
-10.0
-7.5
-5.0
-25
0.0
2.5
5.0
7.5
10.0
Mutual learning Vanilla
10
Min worst grad norm
1.4
KL Divergence
0.8
0.6
0.
Class
Class 1
Epoch
Figure 3: Case two: (Top) The first plot shows class as gaussian mixture with class as Gaussian.
(Bottom Left) Shows the KL-divergence for each class with and without mutual learning objective.
(Bottom Right) Shows min, max,ey V L"ey (07, 0z,)] decreasing with training epoch.
38

---

Different B Values (1) 2 x 10-4)
Different Values (B 0.1)
Method
FID(J)
IS(1)
Method
FID(4)
IS(1)
B 0.0
16.58
8.78 0.15
7) 2 x 10
18.61
8.94 + 0.10
B 0.1
18.61
8.94 0.10
? 10-4
14.58
8.92 + 0.19
B
16.74
8.55
?) 10-5
18.62
8.62 + 0.21
Figure 4: FID and IS Scores for Different & and Y) Values
Figure 5: Visualization of image generated from Vanilla DDPM (3 = 0) (Left) and Mutual Learning (Right)
C.2 Experiments on CIFAR1OLT
In this section, we present the numerical results for varying hyperparamters ry and 3 values. Further- more, for completeness, we provide visualization of the images generated from various methods.
39

---

Figure 6: Image Visualization of CBDM
40


Reviews:
[{"review_id": "7ExvCIRfXU", "reviewer": "Reviewer_aRKS", "summary": "This paper tackles the underexplored challenge of training **diffusion models on long-tailed datasets**, where many classes have very few examples. The authors propose a **game-theoretic framework** that treats training diffusion models as a multi-player game over conditional score networks, using **Deep Mutual Learning (DML)** and a **regularized objective**. The solution is sought at a **Nash equilibrium**, and the authors derive convergence guarantees for gradient descent toward such an equilibrium. Theoretical bounds on **worst-case sampling error** across all classes are provided, along with architectural and training details using ReLU networks.", "questions": "- How would this approach scale to larger architectures?\n- How does this approach experimentally compare to other long-tailed methods?\n- How is the quality of the samples? Would the quality still be maintained for larger generative tasks?\n- Could there be a diagram or primer on mutual learning?\n\n- please fix Line 108 typo in section header", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- High importance: long-tailed class imbalance is a critical bottleneck in real-world generative applications\n- Novel game-theoretic formulation with Nash equilibrium applied to conditional diffusion\n- Theoretical guarantees (non-asymptotic convergence + worst-case bounds)\n\n\nWeaknesses:\n- experiments only show FID to measure the quality of the samples\n- Could benefit from more high-level intuition—especially why mutual learning is the right approach/ what mutual learning even is\n- Small architectures used in experiments\n- no experimental comparison with other long-tailed methods", "was_revised": true, "final_justification": "Significant theoretical contributions and adequate experiments. Theoretical contributions could go further however.", "rebuttal": "Thank you very much for taking the time to review our paper and provide valuable feedback. We appreciate your constructive comments, which have helped us improve the quality of our work. Your insights and suggestions were truly helpful in strengthening our submission.\n\nThank you again for your effort and contribution to the review process. Please find our response to your questions..\n\n1.  ...scale to larger architectures?\n\nOn the Theory side: We strongly believe our approach can be extended to deeper neural network architectures with more hidden layers in line with  ``Evaluating the design space of diffusion-based generative models\" Wang et al NeuRIPS 2024. The objective functions satisfy nice properties when the score function is parametrized through a random feature model, enabling the computation of an $\\epsilon-$ Nash Equilibrium. For deeper neural networks, we present a high level approach:\n\n(a) Prove Semi-Smoothness of the regularized objective function: Observe that $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})=\\bar{\\mathcal{L}}^{n_y}(\\theta_y)+ \\beta \\bar{\\mathcal{L_mut}}^{n_y}(\\theta_y,\\theta_{-y})$. Lemma 9 [1] proves the Semi-smoothness of   $\\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ with high probability. This can also be proved for $\\bar{\\mathcal{L_mut}}^{n_y}(\\theta_y,\\theta_{-y})$ in line with Theorem 3 in [2]. \n\n(b) PL type inequality: Then next step would be to derive a PL like inequality as Lemma 2 in [1] with high probability. As the mutual learning learning term takes a quadratic form, we believe after some work this property continues to hold for deeper neural networks.\n\n(c)  Lipschitzness of mutual learning term: With high probability the gradient of norm $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})$ w.r.t to $\\theta_y$ is bounded which implies $\\bar{\\mathcal{L_mut}}^{n_y}(\\theta_y,\\theta_{-y})$ is lipschitz in $\\theta_y$.\n\nCombining the above results will allow one to extend our framework to deeper neural networks.\n\n2. ..experimentally compare to other long-tailed methods?\n\nThere are various methods that are used to address the problem of Deep learning task in the presence of long-tailed distribution. Most of the long-tailed methods employed in machine learning task are yet to be extended to Diffusion models. In our work, we take a more theoretical approach to extend mutual learning to the problem of Diffusion generative modeling in the presence of tong-tailed distributions. As we take a more theoretical approach, we provide comparison with empirical data distributions such as non-uniform mixture of gaussian where each Gaussian distribution represents a class label.  These empirical studies allow us to verify our theoretical findings and understand scenarios when mutual learning is useful and when it is not. We understand your concern and have performed experiments to address these in the supplementary section. Performing statistical analysis on maximum of a sequence of random variable whose distribution are unknown is challenging. To circumvent this, in Appendix C, we directly compute the KL divergence between the distribution characterized by the learned score network and the ground truth distribution as in Section 4 in [3]. This allows for direct comparison among class labels and within class labels. As you will observe in Supplementary Material Appendix C, we have provided more empirical studies to verify our theoretical findings. In case 1 (Appendix C), the performance of the head class goes down when mutual learning is used.  However, the worst case KL divergence is still lower for mutual learning. This is because the head class contains enough training samples to learn a good score network and achieves no performance increase from transfer of knowledge from the tail class. Furthermore, in case 2 when the relative position of the class labels in the embedding space provides additional knowledge (information); mutual learning is beneficial for all classes.    We observe improved performance in both the classes (Figure 3 in Appendix C) when mutual learning is employed vs when it is not employed. This raises interesting questions in representation learning of the input class label data into embedding spaces where the relative positions of classes enable efficient learning through mutual learning. Our work opens the door to multiple future research directions. \n    \n\n\n3. ..Could there be a diagram or primer on mutual learning?\n\nAs we cannot post images or documents in the rebuttal phase, we refer the reviewers to Figure 2(left) in ``Fast Sampling of Diffusion Models with Exponential Integrator\"  Zhang et al 2022 arXiv for a  high level intuition for now. We will ensure a high level intuition and diagram  for mutual learning applied to diffusion models is included in future revisions/ versions of our work.\n    Intuition: Diffusion models are trained to learn the score function $\\nabla \\log{p_t(x_t|x_0)}$ of the forward process which is then used in the reverse SDE to sample/ generate. As it can be seen from Figure 2 (left) [4] the score model works well in the region where $p_t(x)$ is large but suffers from large error in the region where $p_t(x)$ is small. This observation can be explained by examining the training loss on Figure 2(right). As explained in the above mentioned paper, since the training data is sampled from $p_t(x,y)$ in regions with a low $p_t(x,y)$ value, the learned score network is not expected to work well due to the lack of training data\" Figure 2(left). \n     As a consequence, to ensure $\\bar{Y}_0$ is close to $x_0$, one need to make sure $\\bar{Y}_t$ (as defined in the paper) stays in the high $p_t(x,y)$ region $\\forall t \\in [0,T]$ during the generation/sampling process. Since the tail classes have very few training data, their (tail classes) score do not lie in this high density region in Figure 2 (left). The mutual learning term aligns the score network for the tail classes with the high confidence scores of head classes (in the high noise region, in other words larger $t$) in a way such that the score network of the tail classes stay in the high probability region during sampling/ generation. The above reasoning also justifies the choice of weighting function $\\omega(t)$. For $t\\approx 0,  p_t \\approx p_0 \n     $, we wish the training objective give more wait to fitting to data distribution while for larger $t\\gg0$ the training give more weight to mutual learning objective. This implies $\\omega(t)$ should be an increasing function of $t\\in [0,T]$. This is employed in \"Class-Balancing Diffusion Model\", but lacked  proper derivation and justification.  \n\n4. Please note that we provide the limitations of our work in Section 7. Regarding the potential societal consequences, given the theoretical nature of our work we do not foresee any direct implications. However, mitigating the adverse effects of the long-tailed phenomena could lead to more robust, secure, and fair generative models.\n\n        [1] Wang et al. NeuRIPS 2024 ``Evaluating the design space of diffusion-based generative models\"  \n        \n        [2] Zhu et al. ``A Convergence Theory for Deep Learning via Over-Parametrization\"\n\n        [3] Li et al. NeuRIPS 2023 ``On the generalization properties of diffusion models\" \n\n        [4] Zhang et al. 2022 arXiv ``Fast Sampling of Diffusion Models with Exponential Integrator\"\n\n\n\n\n\n4. We took the code from ``Class Balancing Diffusion Model” Qin. et al 2023 and updated the training procedure according to individual gradient descent. Given the time and resource constraint, we were able to run CBDM for 100000 training steps and individual gradient descent for 10000 steps. To perform evaluation, we generate 500 total samples and make the comparison at the 10000 training step mark. We provide FID, IS across various parameter settings. We will include the link to the github repository for the code and images in future revisions, versions of our paper. We provide  tables summarizing the results. \n\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| Vanilla, τ=0 (DDPM) | 423.06 | 1.23 (0.02) |\n| CBDM  | 419.82 | 1.22 (0.01) |\n| Mutual Learning, τ=0.1 | 409.50 | 1.22 (0.01) |\n\n## Table 2: Different τ Values (lr=2e-4)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.0, lr=2e-4 | 423.06 | 1.23 (0.02) |\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=1.0, lr=2e-4 | 421.82 | 1.22 (0.01) |\n\n## Table 3: Different lr Values (τ=0.1)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=0.1, lr=1e-4 | 409.50 | 1.22 (0.01) |\n| τ=0.1, lr=1e-5 | 442.35 | 1.20 (0.01) |\n\nAs seen from the above table, CBDM does better than the vanilla case of DDPM trained on individual class labels and mutual learning does better than the two. Mutual learning doing better than Vanilla case highlights the impact of mutual learning. We were not able to replicate the V-shaped curve as in Fig 1 (right) in our paper for real world datasets due to resource and time constraints.\n\n5. We also performed to replicate case 2 in our supplementary Apeendix C. Class 0: mixture with means [-2,0][2,0] and sigma=1; class 1:single guassian with mean [0,0] and sigma=2).\n| Method         | FD (Head Class) | FD (Tail Class) |\n|----------------|-----------------|-----------------|\n| IGD (τ=0.0)    | 2.766 ± 0.534   | 1.984 ± 0.321   |\n| IGD (τ=0.001)  | 2.751 ± 0.517   | 1.996 ± 0.317   |\n| IGD (τ=0.01)   | 2.720 ± 0.470   | 1.988 ± 0.297   |\n| IGD (τ=0.1)    | 2.467 ± 0.467   | 1.965 ± 0.287   |\n| IGD (τ=1.0)    | 1.954 ± 0.403   | 2.021 ± 0.370   |\n\nWe hope the reviewer finds our response satisfactory. Please let us know during the discussion phase if we can provide further clarification regarding our work, notably it's theoretical contributions, novelty, and significance."}, {"review_id": "vYrTFU9IEE", "reviewer": "Reviewer_9nrw", "summary": "This paper proposes a long-tailed learning framework via diffusion models. This paper introduces a regularized training objective that combines the standard diffusion loss with a mutual learning term, addressing the issue of poor generation performance for tail classes under long-tailed data distributions in diffusion models. The paper analyzes the convergence and generalization performance of this method from a theoretical perspective, and verifies its effectiveness through experiments.", "questions": "See above weakness. Since I am not familar with the theoretical part in this paper, authors need to address the above weakness well: 1) Comparsions 2) hyperparameters choosen 3) more indicators 4) More visual results.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength：\n\n1.This paper combines deep mutual learning and diffusion models and proposes a regularized training objective that balances the generation performance of all categories.\n\n2.The paper gives an upper bound of Nash gap of the score network obtained from the algorithm and worst case error bound. Furthermore, the theoretical analysis is comprehensive and in-depth.\n\nWeakness: \n\n1.This paper should give sufficient comparisons of quantitative and qualitative experimental results.\n\n2.This paper points out that the choice of hyperparameters has a significant impact on model performance, but there is a lack of in-depth discussion on the impact of these hyperparameters on performance results\n\n3.This paper should include results of more indicators, such as Recall and IS used in CBDM [1].\n\n4.This paper gives enough formulas, but lacks necessary visualization, which creates obstacles to understanding the methods and effects.\n\n[1] Yiming Qin, et al. “Class-balancing diffusion models.” CVPR 2023", "was_revised": true, "final_justification": "Since I am not familar with the theoretical part in this paper, although the authors have addressed partial of my concerns, but other reviewers ' question seems not to be solved now. So I would like to keep the score.", "rebuttal": "Thank you very much for taking the time to review our paper and provide valuable feedback. We appreciate your constructive comments, which have helped us improve the quality of our work. Your insights and suggestions were truly helpful in strengthening our submission.\n\nThank you again for your effort and contribution to the review process. Please find our response to your questions\n\n\n1.This paper should give sufficient comparisons of quantitative and qualitative experimental results.\n\n2.This paper points out that the choice of hyperparameters has a significant impact on model performance, but there is a lack of in-depth discussion on the impact of these hyperparameters on performance results\n\nIn Corollary 1, we provide the hyper-parameters as a function of the network parameters, training iterations, sampling time duration, dimension of data, etc. to achieve and $\\mathcal{O}(\\epsilon+1) $ sampling error.  Although our analysis is focused on two layer random ReLUs,  We strongly believe our approach can be extended to deeper neural network architectures with more hidden layers in line with  ``Evaluating the design space of diffusion-based generative models\" Wang et al NeuRIPS 2024. The objective functions satisfy nice properties when the score function is parametrized through a random feature model, enabling the computation of an $\\epsilon-$ Nash Equilibrium. For deeper neural networks, we present a high level approach:\n\n(a) Prove Semi-Smoothness of the regularized objective function: Observe that $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})=\\bar{\\mathcal{L}}^{n_y}(\\theta_y)+ \\beta \\bar{\\mathcal{L_mut}}^{n_y}(\\theta_y,\\theta_{-y})$. Lemma 9 [1] proves the Semi-smoothness of   $\\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ with high probability. This can also be proved for $\\bar{\\mathcal{L_mut}}^{n_y}(\\theta_y,\\theta_{-y})$ in line with Theorem 3 in [2]. \n\n(b) PL type inequality: Then next step would be to derive a PL like inequality as Lemma 2 in [1] with high probability. As the mutual learning learning term takes a quadratic form, we believe after some work this property to continue to hold for deeper neural networks.\n\n(c)  Lipschitzness of mutual learning term: With high probability the gradient of norm $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})$ w.r.t to $\\theta_y$ is bounded which implies $\\bar{\\mathcal{L_mut}}^{n_y}(\\theta_y,\\theta_{-y})$ is lipschitz in $\\theta_y$.\n\nCombining the above results will allow one to extend our framework to deeper neural networks. Our work forms the stepping stone and motivation for future work in Generative AI, Deep Learning and Optimization to be reformulated as games that would allow for their expansion to Federated, distributed setting and much more. \n\n        [1] Wang et al NeuRIPS 2024 ``Evaluating the design space of diffusion-based generative models\"  \n        \n        [2] Zhu et al ``A Convergence Theory for Deep Learning via Over-Parametrization\"\n\n\n\n3.This paper should include results of more indicators, such as Recall and IS used in CBDM [1].\n\n4.This paper gives enough formulas, but lacks necessary visualization, which creates obstacles to understanding the methods and effects.\n\nAs we cannot post images or documents in the rebuttal phase, we refer the reviewers to Figure 2(left) in ``Fast Sampling of Diffusion Models with Exponential Integrator\"  Zhang et al 2022 arXiv for high-level intuition for now. We will ensure a high-level intuition and diagram for mutual learning applied to diffusion models is included in future revisions/ versions of our work.\n    Intuition: Diffusion models are trained to learn the score function $\\nabla \\log{p_t(x_t|x_0)}$ of the forward process which is then used in the reverse SDE to sample/ generate. As it can be seen from Figure 2 (left) [4] the score model works well in the region where $p_t(x)$ is large but suffers from large error in the region where $p_t(x)$ is small. This observation can be explained by examining the training loss on Figure 2(right). As explained in the paper mentioned above, since the training data is sampled from $p_t(x,y)$ in regions with a low $p_t(x,y)$ value, the learned score network is not expected to work well due to the lack of training data\" Figure 2(left). \n     As a consequence, to ensure $\\bar{Y}_0$ is close to $x_0$, one need to make sure $\\bar{Y}_t$ (as defined in the paper) stays in the high $p_t(x,y)$ region $\\forall t \\in [0,T]$ during the generation/sampling process. Since the tail classes have very few training data, their (tail classes) score do not lie in this high-density region in Figure 2 (left). The mutual learning term aligns the score network for the tail classes with the high confidence scores of head classes (in the high noise region, in other words larger $t$) in a way such that the score network of the tail classes stays in the high probability region during sampling/ generation. The above reasoning also justifies the choice of weighting function $\\omega(t)$. For $t\\approx 0,  p_t \\approx p_0 \n     $, we wish the training objective would give more wait to fitting to data distribution while for larger $t\\gg0$ the training give more weight to mutual learning objective. This implies $\\omega(t)$ should be an increasing function of $t\\in [0,T]$. This is employed in \"Class-Balancing Diffusion Model\" but lacked proper derivation and justification.  \n\n   [4] Fast Sampling of Diffusion Models with Exponential Integrator\"  Zhang et al 2022 arXiv\n\n5. We took the code from ``Class Balancing Diffusion Model” Qin. et al 2023 and updated the training procedure according to individual gradient descent. Given the time and resource constraint, we were able to run CBDM for 100000 training steps and individual gradient descent for 10000 steps. To perform evaluation, we generate 500 total samples and make the comparison at the 10000 training step mark. We provide FID, IS across various parameter settings. We will include the link to the github repository for the code and images in future revisions, versions of our paper. We provide  tables summarizing the results. \n\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| Vanilla, τ=0 (DDPM) | 423.06 | 1.23 (0.02) |\n| CBDM  | 419.82 | 1.22 (0.01) |\n| Mutual Learning, τ=0.1 | 409.50 | 1.22 (0.01) |\n\n## Table 2: Different τ Values (lr=2e-4)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.0, lr=2e-4 | 423.06 | 1.23 (0.02) |\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=1.0, lr=2e-4 | 421.82 | 1.22 (0.01) |\n\n## Table 3: Different lr Values (τ=0.1)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=0.1, lr=1e-4 | 409.50 | 1.22 (0.01) |\n| τ=0.1, lr=1e-5 | 442.35 | 1.20 (0.01) |\n\nAs seen from the above table, CBDM does better than the vanilla case of DDPM trained on individual class labels and mutual learning does better than the two. Mutual learning doing better than Vanilla case highlights the impact of mutual learning. We were not able to replicate the V-shaped curve as in Fig 1 (right) in our paper for real world datasets due to resource and time constraints.\n\n6. We also performed to replicate case 2 in our supplementary Apeendix C. Class 0: mixture with means [-2,0][2,0] and sigma=1; class 1:single guassian with mean [0,0] and sigma=2).\n| Method         | FD (Head Class) | FD (Tail Class) |\n|----------------|-----------------|-----------------|\n| IGD (τ=0.0)    | 2.766 ± 0.534   | 1.984 ± 0.321   |\n| IGD (τ=0.001)  | 2.751 ± 0.517   | 1.996 ± 0.317   |\n| IGD (τ=0.01)   | 2.720 ± 0.470   | 1.988 ± 0.297   |\n| IGD (τ=0.1)    | 2.467 ± 0.467   | 1.965 ± 0.287   |\n| IGD (τ=1.0)    | 1.954 ± 0.403   | 2.021 ± 0.370   |\n\nWe hope the reviewer finds our response satisfactory. Please let us know during the discussion phase if we can provide further clarification regarding our work, notably it's theoretical contributions, novelty, and significance."}, {"review_id": "mvOW0nK8YZ", "reviewer": "Reviewer_HV1G", "summary": "The paper proposes using Mutual learning to address the problem of class imbalance (i.e, minimizing the KL can lead to a class with class conditional KL way worst than the class that is more represented) in learning a generative model. It proposes mainly a theoretical analysis of the convergence of the proposed algorithm to the Nash equilibrium for the proposed Game in a simplified setting. Then, it illustrates briefly in a toy example the performance of the model.", "questions": "Questions:\n1. I don't understand the phrase in lines 157-158 \"The above objective is sound when the marginal density of the classes...\". As I reckon the above objective is always \"valid\", but maybe what the authors mean is that they would not lead to an Egalitarian solution? \n2. At what point do you really nead to train different networks for each class (suggested by the notation $\\theta_{-y}$? This is specially important, because one of  the alternatives of the current method would be to estimate the marginal weights and this is of course difficult in the cases with a large number of classes. But training different networks would be as well.\n3. How adding the regularization term helps ? What is the fundamental difference between doing so and just learning one score for each class? It would be interesting to compare to such method  in the numerical section.\n\nSome typos I found:\nTypos:\n1. But we not (line 188)\n2. p_t(x_t | x_0) is always a Gaussian with the forward process defined as previously.\n3. Line 177: \"To obtain a regularized version of the DM objectinve function\" $\\textbf{\\mathcal{L}^{y}{conti}}$ I suppose.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper does a rather complete analysis of the convergence of the proposed method to the Nash equilibrium. Most of the hypothesis used are minimal, with two exception being the depth of the neural net and the way the parameters are defined (which leads to a convex objective), compacity of the support of the data and differentiability of the score, the latter being easily eased by the so called \"early stopping\" of the backward diffusion. Compacity is another aspect that is restrictive but we might consider it ok in mostly current applications of generative modelling (images for example).\n\nWeaknesses:\nThe main weakness is the rather lacking numerical experiments section. While I understand that the paper is mainly theoretical, one would still appreciate a comparison with other methods and not only the vanilla one. Also, some of the graphs are not properly explained. Namely what do the bars in Worst FID score represent?", "was_revised": true, "final_justification": "Strong theoretical work with limited evaluation.", "rebuttal": "Thank you very much for taking the time to review our paper and provide valuable feedback. We appreciate your constructive comments, which have helped us improve the quality of our work. Your insights and suggestions were truly helpful in strengthening our submission.\n\nThank you again for your effort and contribution to the review process. Please find our response to your questions..\n\n1. I don't understand the phrase in lines 157-158 ...  We apologize for the confusion. Yes, if the marginal densities of the class labels are not uniformly distributed, training on the total (over all class labels $y\\in \\mathcal{Y}$) empirical loss will lead to a bias by putting more weight to the head classes during the training process. This will not lead to an egalitarian solution as shown in Figure 1 of   Yiming Qin, et al. “Class-balancing diffusion models.” CVPR 2023. We will clarify this in future versions of our paper.\n\n2.  At what point do you really need to train......\n\n(a) If the reviewer is asking when the parameters $\\theta_{-y}$ are trained within the algorithm, then the answer is that all parameters $\\theta_y, y \\in \\mathcal{Y}$ are  trained simultaneously, as can be seen in the for loop over $y=0,\\cdots|\\mathcal{Y}|$ in the algorithm.(b) Case of training a single neural network for all classes: This is a special case in our problem setting which is similar to the objective in CBDM [2]. The error analysis for such a case will follow from arguments underlying gradient descent convergence properties for convex, smooth loss functions. However, in such a case we can only derive $E_{y}[ D_{KL}(P_0(.|y)||P_{0,\\theta}(.|y))]$ and deriving a worst-case sampling error, a desideratum made possible by our formulation, will not be possible.  (c) Application to Federated Learning: If the question is regarding when does such a scenario arise where one needs to train separate score network for each class label, then Federated Learning is a classic example where our problem formulation will thrive. Consider the following scenario, each class label $y \\in \\mathcal{Y}$ is thought of as a client that holds private training data with variable number of training sample points. Individual Gradient Descent then represents local training of score network with global sharing of updated score network parameters while preserving the privacy of local client data. This allows fair learning and generalization among all classes and prevents overfitting (memorization) for class labels with low training data frequency. In Continual Learning, multiple score networks can also be thought as different subspaces over which training occurs and each subspace corresponding to a class label. This idea stems from ``Continual Learning in Low-rank Orthogonal Subspaces\" Chaudhry et al NeuRIPS 2020. Mutual learning allows for transfer of knowledge among task ``Meta-learn\" during training within these subspaces while preventing catastrophic forgetting. However, such a concept is a new research direction in itself, and one has to employ manifold optimization theory which is beyond the scope of our work. \n\n3. How adding the regularization term ...\n\nThank you for giving us a chance to respond to this interesting question. It is important to distinguish the empirical version of the objectives $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y}),\\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ over which training occurs and the population loss $\\mathcal{L}^{y}(\\theta_y)$ that determines the sampling (generalization) error. The Individual Gradient Descent is employed to find an $\\epsilon-$ Nash Equilibrium of the  $|\\mathcal{Y}|$-player game $<  \\mathcal{Y},(\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})),(\\Theta_y) >$ where the cost function is given by the empirical regularized objectives. When $\\beta=0$ or as $\\beta \\to 0$, $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})\\approx \\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ and the algorithm finds $\\hat{\\theta}_y \\in \\arg\\min \\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ (no mutual learning). This is the classical DDPM training paradigm.\n\nAs present in Theorem 1, in such a case, the empirical training objective will be low, and the sampling error will be dominated by generalization gap which is inversely proportional to $n_y$. For tail classes that have low training data frequency, this generalization bound will be large resulting in large sampling error. One might feel this can be overcome by selecting a score network with more hidden neurons (large $m$); however, this increases the training error as shown in Theorem 1. \n    When $\\beta$ is large, the mutual learning term enjoys more weight during training. Hence, there is a sweet spot of $\\beta$, not too large, not too small (V shaped curve of Figure 1 (right) in our paper) where employing mutual learning is beneficial. \n\n As we cannot post images or documents in the rebuttal phase, we refer to reviewers to Figure 2(left) in ``Fast Sampling of Diffusion Models with Exponential Integrator\" Zhang et al 2022 arXiv for high-level intuition. We will ensure a high-level intuition for mutual learning applied to diffusion models is included in future revisions/ versions of our work.\n    \nDiffusion models are trained to learn the score function $\\nabla \\log{p_t(x_t|x_0)}$ of the forward process which is then used in the reverse SDE to sample/ generate. As it can be seen from Figure 2 (left) the score model works well in the region where $p_t(x)$ is large but suffers from large error in the region where $p_t(x)$ is small. This observation can be explained by examining the training loss on Figure 2 (right). As explained in the paper mentioned above, since the training data is sampled from $p_t(x,y)$ in regions with a low $p_t(x,y)$ value, the learned score network is not expected to work well due to the lack of training data\" Figure 2(left). \n\nAs a consequence, to ensure $\\bar{Y}_0$ is close to $x_0$, one need to make sure $\\bar{Y}_t$ (as defined in the paper) stays in the high $p_t(x,y)$ region $\\forall t \\in [0,T]$. Since the tail classes have very few training data, their (tail classes) score do not lie in this high density region in Figure 2 (left). The mutual learning term aligns the score network for the tail classes with the high confidence scores of head classes (in the high noise region, in other words larger $t$) in a way such that the score network of the tail classes stays in the high probability region during sampling/ generation. The above reasoning also justifies the choice of weighting function $\\omega(t)$. For $t\\approx 0,  p_t \\approx p_0   $, we wish the training objective would give more wait to fitting to data distribution while for larger $t\\gg0$ the training give more weight to mutual learning objective. This implies $\\omega(t)$ should be an increasing function of $t\\in [0,T]$. This is employed in \"Class-Balancing Diffusion Model\" but lacked proper theoretical derivation and justification. In short, mutual learning/ regularization improves the generalization capability of the learned score networks by allowing knowledge transfer and preventing overfitting.\n\n4. We took the code from ``Class Balancing Diffusion Model” Qin. et al 2023 and updated the training procedure according to individual gradient descent. Given the time and resource constraint, we were able to run CBDM for 100000 training steps and individual gradient descent for 10000 steps. To perform evaluation, we generate 500 total samples and make the comparison at the 10000 training step mark. We provide FID, IS across various parameter settings. We will include the link to the github repository for the code and images in future revisions, versions of our paper. We provide  tables summarizing the results. \n\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| Vanilla, τ=0 (DDPM) | 423.06 | 1.23 (0.02) |\n| CBDM  | 419.82 | 1.22 (0.01) |\n| Mutual Learning, τ=0.1 | 409.50 | 1.22 (0.01) |\n\n## Table 2: Different τ Values (lr=2e-4)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.0, lr=2e-4 | 423.06 | 1.23 (0.02) |\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=1.0, lr=2e-4 | 421.82 | 1.22 (0.01) |\n\n## Table 3: Different lr Values (τ=0.1)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=0.1, lr=1e-4 | 409.50 | 1.22 (0.01) |\n| τ=0.1, lr=1e-5 | 442.35 | 1.20 (0.01) |\n\nAs seen from the above table, CBDM does better than the vanilla case of DDPM trained on individual class labels and mutual learning does better than the two. Mutual learning doing better than Vanilla case highlights the impact of mutual learning. We were not able to replicate the V-shaped curve as in Fig 1 (right) in our paper for real world datasets due to resource and time constraints.\n\n5. We also performed to replicate case 2 in our supplementary Apeendix C. Class 0: mixture with means [-2,0][2,0] and sigma=1; class 1:single guassian with mean [0,0] and sigma=2).\n| Method         | FD (Head Class) | FD (Tail Class) |\n|----------------|-----------------|-----------------|\n| IGD (τ=0.0)    | 2.766 ± 0.534   | 1.984 ± 0.321   |\n| IGD (τ=0.001)  | 2.751 ± 0.517   | 1.996 ± 0.317   |\n| IGD (τ=0.01)   | 2.720 ± 0.470   | 1.988 ± 0.297   |\n| IGD (τ=0.1)    | 2.467 ± 0.467   | 1.965 ± 0.287   |\n| IGD (τ=1.0)    | 1.954 ± 0.403   | 2.021 ± 0.370   |\n\nWe hope the reviewer finds our response satisfactory. Please let us know during the discussion phase if we can provide further clarification regarding our work, notably it's theoretical contributions, novelty, and significance."}, {"review_id": "i4Igr0epOq", "reviewer": "Reviewer_tuny", "summary": "This paper addresses the challenge of training diffusion models on real-world long-tailed datasets where rare classes are underrepresented.\nIt proposes a novel regularized training objective that combines standard diffusion loss with a mutual learning term, formulated as a multi-player game with a Nash equilibrium solution. The authors provide theoretical convergence guarantees and bounds on sampling error across all classes, including the tails. The framework offers principled insights for improving fairness, generalization, and performance in imbalanced generative modeling scenarios.", "questions": "1. The Nash gap you mentioned is: “The Nash gap of the score network obtained from the algorithm is upper bounded by $O(\\sqrt{(1/T)} + β)$, where β is the regularizing parameter.” So wouldn’t it be best when β is 0? Then what is the point of the proposed regularization?\n\n2. I understand that class imbalance is an important issue, but in practice, a more difficult and widespread problem arises when there is dataset bias even in the absence of labels. For example, issues addressed in works like *[ICLR 2024] Training Unbiased Diffusion Models From Biased Datasets*. In such cases, do you think your theoretical analysis could be extended to cover these scenarios as well? If extending to that setting is too difficult, it would be helpful to clearly distinguish your work from those in that line of research in the related work section, for better positioning.", "limitations": "1. While CBDM is used as the closest baseline, the comparison is lacking. Compare the results on CIFAR10LT, CIFAR100LT, and other datasets from CBDM with metrics such as FID, class-wise FID, precision, and recall. What is the meaning of only measuring worst-case FID?\n\n2. My main concern is the potential side effects introduced by the regularization. The fact that the experimental results focus solely on worst-case FID makes me wonder if this is due to such side effects. A comprehensive comparison across multiple metrics and with relevant baselines is, in my view, the most important point.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strength**\n- Theoretically Grounded Objective: The paper introduces a novel regularized diffusion objective rooted in mutual learning and game theory, with convergence guarantees toward a Nash equilibrium—providing strong theoretical backing for long-tailed generative modeling.\n\n- Tail-Aware Performance: The method explicitly improves generation quality for underrepresented (tail) classes, balancing sample diversity and fidelity across the entire label distribution, which addresses a core limitation of standard diffusion models.", "was_revised": true, "final_justification": "Too limitied empirical results.", "rebuttal": "Thank you very much for taking the time to review our paper and provide valuable feedback. We appreciate your constructive comments, which have helped us improve the quality of our work. Your insights and suggestions were truly helpful in strengthening our submission.\n\nThank you again for your effort and contribution to the review process. Please find our response to your questions\n\n1. The Nash gap you mentioned is: “The N...\n\nThank you for giving us a chance to respond to this interesting question. It is important to distinguish the empirical version of the objectives $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y}),\\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ over which training occurs and the population loss $\\mathcal{L}^{y}(\\theta_y)$ that determines the sampling (generalization) error. The Individual Gradient Descent is employed to find an $\\epsilon-$ Nash Equilibrium of the  $|\\mathcal{Y}|$-player game $<  \\mathcal{Y},(\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})),(\\Theta_y) >$ where the cost function is given by the empirical regularized objectives. When $\\beta=0$ or as $\\beta \\to 0$, $\\bar{\\mathcal{L_reg}}^{n_y}(\\theta_y,\\theta_{-y})\\approx \\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ and the algorithm finds $\\hat{\\theta}_y \\in argmin \\bar{\\mathcal{L}}^{n_y}(\\theta_y)$ (no mutual learning). This is the classical DDPM training paradigm.\n\nAs present in Theorem 1, in such a case, the empirical training objective will be low and the sampling error will be dominated by generalization gap which is inversely proportional to $n_y$. For tail classes that have low training data frequency, this generalization bound will be large resulting in large sampling error. One might feel this can be overcomed by selecting a score network with more hidden neurons (large $m$), however this increases the training error as shown in Theorem 1. \n    When $\\beta$ is large, the mutual learning term enjoys more weight during training and . Hence, there is a sweet spot of $\\beta$, not too large, not too small (V shaped curve of Figure 1 (right) in our paper) where employing mutual learning is beneficial. \n\n As we cannot post images or documents in the rebuttal phase, we refer to reviewers to Figure 2(left) in ``Fast Sampling of Diffusion Models with Exponential Integrator\"  Zhang et al 2022 arXiv for a  high level intuition. We will ensure a high level intuition for mutual learning applied to diffusion models is included in future revisions/ versions of our work.\n    Diffusion models are trained to learn the score function $\\nabla \\log{p_t(x_t|x_0)}$ of the forward process which is then used in the reverse SDE to sample/ generate. As it can be seen from Figure 2 (left) the score model works well in the region where $p_t(x)$ is large but suffers from large error in the region where $p_t(x)$ is small. This observation can be explained by examining the training loss on Figure 2 (right). As explained in the above mentioned paper, since the training data is sampled from $p_t(x,y)$ in regions with a low $p_t(x,y)$ value, the learned score network is not expected to work well due to the lack of training data\" Figure 2(left). \n     As a consequence, to ensure $\\bar{Y}_0$ is close to $x_0$, one need to make sure $\\bar{Y}_t$ (as defined in the paper) stays in the high $p_t(x,y)$ region $\\forall t \\in [0,T]$. Since the tail classes have very few training data, their (tail classes) score do not lie in this high density region in Figure 2 (left). The mutual learning term aligns the score network for the tail classes with the high confidence scores of head classes (in the high noise region, in other words larger $t$) in a way such that the score network of the tail classes stay in the high probability region during sampling/ generation. The above reasoning also justifies the choice of weighting function $\\omega(t)$. For $t\\approx 0,  p_t \\approx p_0   $, we wish the training objective give more wait to fitting to data distribution while for larger $t\\gg0$ the training give more weight to mutual learning objective. This implies $\\omega(t)$ should be an increasing function of $t\\in [0,T]$. This is employed in \"Class-Balancing Diffusion Model\", but lacked  proper derivation and justification. In short, mutual learning/ regularization improves the generalization capability of the learned score networks by allowing knowledge transfer and preventing overfitting. \n\n2. I understand that class imbalance is an important issue,...\n\nAlthough the reviewer raises a very important issue, based on our reading of [ICLR 2024] ``Training Unbiased Diffusion Models From Biased Datasets\", we believe the analysis may not be a straightforward extension of our work. The analysis for the theory of unbiased diffusion models will be in line with the works of ``Evaluating the design space of diffusion-based generative models\" Wang et al NeuRIPS 2024\". We too draw inspiration from the above mentioned works but deviate considerably in terms of problem formulation and analysis. For unbiased diffusion models, there would need to be consideration of additional error terms that arise as a result of estimation error of time-dependent density ratio $$w^{t}_{\\phi^*}(x_t)$$, assumption on $$\\nabla log w_{\\phi^{*}}^t(x_{t})$$, computation of generalization gap  and other terms based on the error decomposition employed. The fundamental difference will be in that mutual learning head classes augment the learning of tail class score functions my making them be close to their score function for high noise region while for the case of importance re-weighting density ratio estimation, the learning process wants to move away from the score of $ p_{bias} $  and move towards the score of $ p_{data} $. .\n\n3. My main concern is the potential side effect..\n\nWe thank the reviewer for raising this concern. We understand your concern and have performed experiments to address these in the supplementary section. Performing statistical analysis on maximum of a sequence of random variable whose distribution are unknown is challenging. To circumvent this, in Appendix C, we directly compute the KL divergence between the distribution characterized by the learnt score network and the ground truth distribution as in Section 4 in [3]. This allows for direct comparison among class labels and within class labels. As can be observed in the Supplementary Material Appendix C, we have provided several empirical studies to verify our theoretical findings . In case 1 (Appendix C), the performance of the head class goes down when mutual learning is used.  However, the worst case KL divergence is still lower for mutual learning. This is because the head class contains enough training samples to learn a good score network and achieves no performance increase from transfer of knowledge from the tail class. Furthermore, in case 2 when the relative position of the class labels in the embedding space provides additional knowledge (information), mutual learning is beneficial for all classes.    We observe improved performance in both the classes (Figure 3 in Appendix C) when mutual learning is employed vs when it is not employed.  \n    \n    [3] ``On the generalization properties of diffusion models\" Li et al NeuRIPS 2023\n\n4. We took the code from ``Class Balancing Diffusion Model” Qin. et al 2023 and updated the training procedure according to individual gradient descent. Given the time and resource constraint, we were able to run CBDM for 100000 training steps and individual gradient descent for 10000 steps. To perform evaluation, we generate 500 total samples and make the comparison at the 10000 training step mark. We provide FID, IS across various parameter settings. We will include the link to the github repository for the code and images in future revisions, versions of our paper. We provide  tables summarizing the results. \n\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| Vanilla, τ=0 (DDPM) | 423.06 | 1.23 (0.02) |\n| CBDM  | 419.82 | 1.22 (0.01) |\n| Mutual Learning, τ=0.1 | 409.50 | 1.22 (0.01) |\n\n## Table 2: Different τ Values (lr=2e-4)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.0, lr=2e-4 | 423.06 | 1.23 (0.02) |\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=1.0, lr=2e-4 | 421.82 | 1.22 (0.01) |\n\n## Table 3: Different lr Values (τ=0.1)\n| Method         | FID     | IS          |\n|----------------|---------|-------------|\n| τ=0.1, lr=2e-4 | 415.81 | 1.23 (0.02) |\n| τ=0.1, lr=1e-4 | 409.50 | 1.22 (0.01) |\n| τ=0.1, lr=1e-5 | 442.35 | 1.20 (0.01) |\n\nAs seen from the above table, CBDM does better than the vanilla case of DDPM trained on individual class labels and mutual learning does better than the two. Mutual learning doing better than Vanilla case highlights the impact of mutual learning. We were not able to replicate the V-shaped curve as in Fig 1 (right) in our paper for real world datasets due to resource and time constraints.\n\n5. We also performed to replicate case 2 in our supplementary Apeendix C. Class 0: mixture with means [-2,0][2,0] and sigma=1; class 1:single guassian with mean [0,0] and sigma=2).\n| Method         | FD (Head Class) | FD (Tail Class) |\n|----------------|-----------------|-----------------|\n| IGD (τ=0.0)    | 2.766 ± 0.534   | 1.984 ± 0.321   |\n| IGD (τ=0.001)  | 2.751 ± 0.517   | 1.996 ± 0.317   |\n| IGD (τ=0.01)   | 2.720 ± 0.470   | 1.988 ± 0.297   |\n| IGD (τ=0.1)    | 2.467 ± 0.467   | 1.965 ± 0.287   |\n| IGD (τ=1.0)    | 1.954 ± 0.403   | 2.021 ± 0.370   |\n\nWe hope the reviewer finds our response satisfactory. Please let us know during the discussion phase if we can provide further clarification regarding our work, notably it's theoretical contributions, novelty, and significance."}]

Author rebuttal:

Meta-review: This work considers diffusion models for long-tail distributions through a game-theoretic approach. Overall the method is quite interesting with solid empirical performance. Through the discussion phase, the authors have also addressed most concerns raised by the reviewers. Hence, the meta-reviewer recommends this work for acceptance.
Decision: Accept (poster)