Long-tailed Diffusion Models with Oriented Calibration
Authors: Tianjiao Zhang
Huangjie Zheng
Jiangchao Yao
Xiangfeng Wang
Mingyuan Zhou
Ya Zhang
Yanfeng Wang
Venue: ICLR 2024 poster (2024)
Keywords: diffusion model
long tail distribution
score matching
Primary area: generative models
TL;DR: A novel approach for long-tail diffusion model generation using direct head-to-tail transfer and multi-target nature of diffusion process, achieving superior diversity and performance.

Abstract: Diffusion models are acclaimed for generating high-quality and diverse images. However, their performance notably degrades when trained on data with a long-tailed distribution. For long tail diffusion model generation, current works focus on the calibration and enhancement of the tail generation with head-tail knowledge transfer. The transfer process relies on the abundant diversity derived from the head class and, more significantly, the condition capacity of the model prediction. However, the dependency on the conditional model prediction to realize the knowledge transfer might exhibit bias during training, leading to unsatisfactory generation results and lack of robustness. Utilizing a Bayesian framework, we develop a weighted denoising score-matching technique for knowledge transfer directly from head to tail classes. Additionally, we incorporate a gating mechanism in the knowledge transfer process. We provide statistical analysis to validate this methodology, revealing that the effectiveness of such knowledge transfer depends on both label distribution and sample similarity, providing the insight to consider sample similarity when re-balancing the label proportion in training. We extensively evaluate our approach with experiments on multiple benchmark datasets, demonstrating its effectiveness and superior performance compared to existing methods. Code: \url{https://github.com/MediaBrain-SJTU/OC_LT}.

Published as a conference paper at ICLR 2024
LONG-TAILED DIFFUSION MODELS WITH ORIENTED CALIBRATION
Tianjiao Zhang', Huangjie Jiangchao Yao Xiangfeng Wang', Mingyuan Zhou', Ya Zhang' Yanfeng Wang 1.2.x
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
Shanghai Artificial Intelligence Laboratory
The University of Texas Austin
4 East China Normal University
[xiaoeyuzt], Surarker, ya_zhang, wangyanfeng)es jtu eeuu..du.cc. cn,
xfwangêcs ecnu. edu, huangjie zhenggutexas, edu. mingyuan, zhou@mccombs utexas.
ABSTRACT
Diffusion models are acclaimed for generating high-quality and diverse images. However, their performance notably degrades when trained on data with a long- tailed distribution. For long tail diffusion model generation, current works focus on the calibration and enhancement the tail generation with head-tail knowl- edge transfer. The transfer process relies the abundant diversity derived from the head class and, more significantly, the condition capacity of the model predic- tion. However, the dependency knowledge transfer might exhibit bias during training, leading to unsatisfactory generation results and lack of robustness. Utilizing Bayesian framework, we de- velop a weighted denoising score-matching technique for knowledge transfer di- rectly from head tail classes. Additionally, we incorporate gating mechanism in the knowledge transfer process. We provide statistical analysis to validate this methodology, revealing that the effectiveness of such knowledge transfer depends both label distribution and sample similarity, providing the insight to consider sample similarity when re-balancing the label proportion in training. We exten- sively evaluate approach with experiments multiple benchmark datasets, demonstrating its effecctiveness and superior performance compared to existing methods. Code: https: htt.://ww.....mmeeeeaaaaaaaaaaaa
the conditional model prediction realize the
INTRODUCTION
Diffusson models have emerged as a powerful class of deep probabilistic models. These models leverage techniques from statistical physics and probabilistic modeling generate high-quality, realistic samples from complex data distributions (Sohl-Dickstein et al.; 2015). The effective imple- mentation of diffusion model necessitates extensive training on diverse and sizable collection of image data. In general, there is a prevalent occurrence of a long-tail distribution (Yang et al.) 2022). wherein vast majority of images belong few dominant categories, while significant portion of the dataset comprises less frequently occurring categories. As a consequence, the training of dif- fusion models with long-tail tail data continues to pose a formidable challenge owing to the distortion in the entire dataset.
In current works, many attempts have been made address the lack of diversity and mode col- lapse issues in the generation of tail classes. A series of methods based on Generative Adversarial Networks (GANs) has been proposed. One of the common approaches is to adopt strategies that re- fine the general model's generation ability by improving its conditional modeling (Rangwani al.) 2022) on tail categories, which heavily depend on GANs structure. Another line of research focuses on alleviating the scarcity of samples in tail classes through appropriate data augmentation tech- niques (Karras et al.] 2020 Zhao et al., (2020) Rangwani et al. (2023), and diffusion process (Zheng al. 2023b Wang al. 2023). Nevertheless, such methods may not effectively capture the under-

---

Published as a conference paper at ICLR 2024
lying data distribution or introduce meaningful variations ( YYo et al. 2020). For long tail diffusion models, Class Balancing Diffusion Models (CBDM) (Qin et al. 2023) has proposed a distribution adjustment regularizer enhancing tail generation based the model prediction
the head class. However, the augmentation relying on the condition and prediction of the model might cause bias during training resulting in generated outcomes that do not meet expectations and leading lack of robustness.
In order alleviate the issue, di- rect knowledge transfer from head to tail categories should be established. Let's review recent study on the dif-
Fornez Kethods: Relying on condition
cagacity
Airplame
Prolktins Repulanzztioo
fusion process, Xu al. (2023) ob- served that the score function exhibits the highest variance during the inter- mediate steps, which is critical pe- riod for semantic formation (Zhang al.) 2023) with the undetermined target. Therefore, based on this ev- idence, we utilize the score in- formation from head classes to cali- brate and enhance the generation of tail classes in this period and finally improve the overall generation per- formance. In this study, leveraging the evidence that the score of the dif- fusion model could be estimated via referencing multiple targets in the dataset (Xu et al. 2023 we propose a calibration strategy for scores the tail class directly making use reference, shown in Figure. [ By employing the strategy that leverages the similarity of un- derlying data distribution, the reliance on the conditional capacity of the model is mitigated and the generation performance is improved.
Trick ---
Track
Gasetrion Model
Ouze: Using Head samples as referance
Instaacee
Head
Airplane
Truck
condition
Tail
Binst
Hnterne
Reaane
Goveration
Class index
Model
Figure I:
The illustration of our motivation. We directly uti- lize head samples reference for tail augmentation instead of depending on the model condition capacity.
head samples
To realize the augmentation for the tail scores, we begin with modeling the score as the weighted scores averaging towards different targets. For conditional generation, the score estimation of noisy tail samples is augmented via properly improving the contribution the score from the head sam- ples in the reference batch, denoted T2H (noisy Tail clean Head) operation enhance the diversity of the tail generation. Simultaneously, Batch Re-sample approach is utilized to alleviate the limitation on overall head-to-tail transfer strength T2H mode. Besides that, unconditional generation, the score function is predominantly influenced by samples from the head classes. Batch re- ee-ammllin is also employed to address this issue. While the method H2T (noisy Head to clean Tail), the reverse direction to T2H, has shown its effectiveness in unconditional generation.
Our contributions can be summarized as follows: (1) We have developed a method denoted as T2H based the multi-target nature of score estimation to effectively calibrate and enhance the gen- eration of tail classes in the semantic formation period, thereby significantly improving the overall generation performance. (2) A "Batch Re Re-sample" strategy is employed to construct a balanced reference batch, in order to address the extreme dominance issue of scores from the head class and promotes head-to-tail transfer under T2H mode. (3) We conduct extensive testing three differ- ent long tail datasets (CIFARIO, CIFAR100, TinylmageNet) to validate the effectiveness of our proposed method. The results consistently demonstrated the superiority of our approach.
2 PRELIMINARY
The diffusion model involves slowly adding noise to the existing training data in the forward pro- cess, and then utilizing a deep learning network to gradually recover the original data from the noise in the reverse process. In the forward process, clean image is progressively transformed by adding carefully calibrated Gaussian noise (Ho et al. 2020) at each diffusion step t, (<>, N M(axr, I),, 0 t < T where the coefficients ag and 0, are chosen that ((xx) is close to initial data density at t ze 0 and close to Gaussian at T. For the reverse process, the diffusion model firstly samples from Gaussian noise distribution p(ær) NO 7), and then gradually incorporates
2

---

Published as a conference paper at ICLR 2024
Denebiing
f1 v=l)
Torget
HZT
F2H
||
Eg
(24 1)
Sample
Pua(s)
d(v)) - v(s6)
Denebitg Praliction
o(v+) 9(36)
(1tas 8k)
Dessilag: Unst
Retwonse
Puoff in Deey
Baich
Figure 2: Overall flowchart for two strategies (H2T, T2H), where the target is determined by evalu- ating the probabilities of labels before and after the transfer to identify the satisfied mode.
various structures and semantic information at each step. Then transition p(x+-1((4) is estimated by training an image-to-image Unet parameterized by 0, p=( (xz=1 pe (2 (), E.). Instead of directly predicting the mean value, the Unet is trained to predict the noise content of 2t. which is optimized by the following loss:
Daa (6, .) -.- <o(
(1)
Another perspective on the diffusion model to start from a parameter-free designed stochastic differential equation (SDE) dz = faz f)dt + (()),,, which transfers the initial data distribution to a prior distribution as time goes from T. The w the standard Wigner process. Sampling the diffusion model via reverse time SDE (Anderson, 1982): dr = [f(x, g() The means time reverse. The score function namely s s(,,, t) = Va, a, log p(z,) should be
( log p(z, ) ] +
estimated in the training stage. The predicted noise in Eq. ( could be related to the score function via denoising score matching (Vincent 2011):
Eg (Zt, -01
From Eq. we could learn that the score is proportional to the noise predicted by the model. In this study, we start from the optimal score V., log ((r,( to formulate our methodology for long tail distribution.
3 METHOD
Our problem involves training data that exhibits long-tail distribution (x((9, ( sampled from p(r,y) and diffusion model parametrized via denoising Unet co (2 t). The y( {C1, C2....,. Cz} is the label of 2(1), assuming that the classes ordered in descending proba- bility label occurrence, Le., if < j then n; > njj where the n, is the number training samples belonging class Ci. With proper training methodology using long-tailed training data and dif- fusion model, we want to generate a more balanced and diversified data distribution p (Ir) in the inference time.
During the inference stage, a diffusion model utilizes a step-by-step reverse operation from prior distribution p(æx) to data distribution p(a))) with reverse-SDE discussed in Section. 2 The reverse- SDE uses the score function s(x,, t) at time step t obtained from the training stage. The optimal score s (I,, t) 2z, log qe (x,) could be expressed with the expectation:
Va, log (ax)) = Byg u.)) Va, log (<xx(20))
where the proof is provided Appendix A As the equation illustrated, the score for given It could be calculated by the expectation under the distribution g(zo|21). Since this distribution is intractable, we utilize importance sampling to sample from initial data distribution g(zo)-
Using the Bayesian rule, have (( ( ((ro)a))(()))))))) we can derive
q ((mo)g(ar((aa/((()
Given g(xx))
((se(1o)
(g Vz, log q( (20) = Eg(oo)
Br V., logg(sz(2o).
(4)
Thus, with mini-batch of samples from ((x0)) i.e. 20
(1:N) std g(20), we have
(x))(a(0)) j=1 g (1)))))))))
Eggeool)) Vz, log (((t|1()) R
V., log (ogg(1)
(5)

---

Published as a conference paper at ICLR 2024
Specifically, we have expanded upon the original one-to-one optimal noise estimator in denoising score matching and transformed into the one-to-many distributional matching technique, with conditional distribution that determines the mapping probability, which encourages mode-covering behavior in the score matching to enhance the diversity of generative modeling (Zheng & Zhou 2021]. The expectation in the equation approximate with weighted average of scores with respect to different samples {a) (ax") denoted as a reference batch.
T2H. In conditional generation, the labels involved and conditioned. The unconditional score V., log ((x+) and conditional score Vx, log (<rely) are both estimated in the training stage. With label participated, the sampling distribution should be 276" q(2ol))) with label y instead of x/)) ((rn in Eq. (5) as:
Va, (org()))))
g( I v)
Ve, log (2)//=00) (00a)//00,))))) .2) {(=0)) .y ) ~ g( vo).
Ej a (
do 90
(1))
Score Weight
where the proof is provided in Appendix, B In the denominator, we sample from (( yo) instead of ((Zg) in the nerf-norna technique with label. The distribution of q<<x|=))) follows a Gaussian distribution and could be calculated by B exp(
|| -xll where B is normalizing constant and 207
there is no label involved. The mixing weight af lan M) Σ, q( 1
from (z))) (3) 200) is dominant by the
L distance respect Tt- In addition the To2 distance in the Gaussian kernel, here make further assumption that the distribution q(zz//0, yo) is adjusted by 4()))
22 ry 12
((0,/ao vo) x q(u)) exp(
20
where 3 is pre-defined parameter, that controls the overall distribution density with respect to yo- When 3 0 then the x, distribution is only dependent on Zig-
For the score estimation of obtained from noisy tail sample (x"6, vo we employ method of score-oriented calibration. The ap- proach enhances the contribution of head class samples (xlf v by increasing the mixing score weight in Eq. 6 Consequently, it lever- ages the rich diversity of the head class to im- prove the performance of the tail class.
Inata Samph
To improve the mixing score weight for the head samples, we could assign B = in Eq. 7 since the g(vdd) in the equation has relative larger value. As shown in Figure. 3 the mix- ing weights towards head samples are improved compared with directly calculating the mixing
weight depending only exp(
|| (1x
Figure 3: Mixing score weights of the sample with ID towards different target in a reference batch under different weight calculation of q(xx /20. mo).
Furthermore, for faster and easier imple- mentation, we firstly sample a mini-batch (8) (
(x66) 330 fi=1- Then the typical training strategy of the diffusion model, for each sam- (7)
ple zo sample random 21 T) and random Gaussian noise e, (), and obtain the perturbed noisy Ht. The training objective is to predict the noise è; by our parametrized Unet
co <o( -0800(0
4

---

Published as a conference paper at ICLR 2024
Since the score for each target wt, the Va, log (x .y)
-_ee As consequence, we transfer
the Eq. (6 from score- soore-weghtted mixing problem to score selection problem
E Ezi 2 n (ps((2)) = g(32/100 M0
Σ, ( y0 ) 0
(8)
where (2) denotes the probability of selecting score with index 2 in the batch with size K. Ca calculated with 6= =t
(z) where the estimated noise is obtained by
va 20
clean target and
current noisy sample. So we firstly calculate the multi-nomial distribution in Eq. (8) with Gaussian
kernel: ((=zl==6)) vo ) x B Bexp(- || - ), and then we sample z from this distribution. 20
((162)) 2 q(vb") (() the transferring is allowed and the score is substituted with eg. which means the noisy tail sample is mapped to clean head sample. If not, the transferring is forbidden then the score is sent back to G, which means that the noisy head sample is forbidden map to the clean tail sample. Since the noisy sample is obtained from the head class, we denote such kind of transferring mode T2H (noisy Tail sample clean Head sample).
Here, we amplify the contribution of head samples by selectively transferring the target solely to the clean head samples in the reference batch. The equivalence between T2H and directly enhancing the contribution of head samples via setting 3 = in score mixing has also been validated in the experiments, as shown Figure.
We summarize the Algorithm in Alg. 1 We note for H2T mode, we just need to change the boundary
condition from ((10 (1)) 2 4(10') (6) in the algorithm to g()) s 9(36')
Algorithm T2H algorithm for conditional long tail generation
Sample mini-batch ((x00) (4) ) with balanced distribution (x
M/
for each sample (xg yu (1)) in the mini-batch do Sample x, with random t and Gaussian noise E, N(O,
Calculate (z) according Eq. with Gaussian kernel C' exp(
||mh
(-)
-
11
0 S
K
2n
Sample 2 ~ (2)
if q36b)) (=)) 2 q36b" ( (g(16%) q(36%) then
E, = c, (6 = 6,)
end if
Compute denoising loss if Conditional then LDiFS = || ( t:
else
CDiSS || co( 1)
end if
end for
For specific sample of tail class Cr, we could augment the score via encouraging the model to pre- dict the score towards the head class. Indeed, by leveraging the rich semantics of the head class and increasing the diversity of scores from the tail categories along the generation path, approach enhances the overall diversity of generated samples. The increased variety ensures a more compre- hensive generation of different classes and ultimately enhances the overall quality and diversity of the generated samples. The fidelity of the transferring could be found in Appendix. G
Batch Re-sample. T2H achieves enhancement in generating tail categories through head-to-tar transfer for conditional generation. Here we evaluate the strength of the transfer quantitatively. Consider two samples (x*f vdd and (arf v10') from different head class Cn and tail class Cr. Let's discuss the distribution with same perturbed noise level (0, with q(2, x H1 ) and q(2e|=0 v6 For the purpose of measuring the strength of transition from CT CH based on q(=|== .b
Langevin dynamics, then the Eygt,1=" ") Ex a(m, (m1"" (0)
should be calculated (Song & Ermon
2020). Then we have the following proposition:
5

---

Published as a conference paper at ICLR 2024
Proposition 3.1
Let ( 20) x B g(v0)²
exp(- ||k - then:
201
g vo
B (a(v") r(x"))² exp( 2
|| x 11
(9)
Σx aa 4o y/
(k) [k))
807
As you can see from the proposition, the transition strength is determined by the Le distance rep- resented by a Gaussian kernel between two samples and the product of label probability powered by 8. In the perspective of Eq. ( with algorithm T2H, when / = 1, The transfer strength is constrained by the product of the q(y) of the head and tail classes, which is relatively small value. Therefore, employing a Batch Re- Re-sampling strategy during training, which equalizes the appearance probability of each category, can significantly enhance the head-to-tail transfer.
Besides, in unconditional generation, the training data ((xo) exhibits long-tail distribution dur- ing the training process. As a result, If we directly use the results from the score estimated in Eq. (5) then the generated p(zo)) will also follow long-tail distribution. The frequent occurrence of head samples will lead to unconditional scores being dominated by the score of head samples. employed toy example to
To intuitively illustrate the necessity of using the Batch Re- Re-sample, illustrate the phenomenon of head class dominance in long-tail tail unconditional generation, as shown in Figure. 4 The Batch-Re-sample could also alleviate this issue. More can be found in Appendix. !
Here, we follow the common assumption in the long tail recognition: the balanced distri- bution and initial long tail distribution are re- lated by sharing the same conditional probabil- ity (([l)) = (x(g) (Zhang et al., 2013).
H2T, Full. In the former analysis, in order to augment the tail class, we propose method de- noted as noisy Tail to clean Head (T2H). Con- versely, there would exist a method of H2T de- noting noisy Head to clean Tail. In H2T, the weight of score towards target tail samples is improved corresponding to smaller 3 Eq. (7) eg. 3 = 11, as the inverse value q for the tail class is larger than the head class. Conse- quently, for noisy head sample, the contribu- tion of the tail sample targets are enhanced. In addition H2T and T2H, if we do not assess the probabilities of transferred labels, this mode denoted 'Full' (means allowing both trans- fer directions).
Figure 4: A toy example for simulating the score distribution. The red and blue points denote head and tail samples, respectively.
Connection with CBDM (Qin et al. 2023). The CBDM has employed score aligning loss in the training stage: [D Egey El v) en (z.,,, The conditional score with label y of æ, regularized with an another conditional score with label y'. The aligned strength is weighted with the diffusion time t. We could further prove that the optimal score of CBDM could be writ- ten as a weighted sum of the initial denoising score and an adjustment term, which is similar to Eq. (6. When the diffusion model converges, the optimal minimizer e" (are.) y) for the CBDM loss: LCBDM y, 1, [[co( y) - c|| + [ Eyeyy [|co(Te, v) co( yf I2)l[2 could be denoted as: (I1, - (1+ DJ Ey co co y'). where the proof is provided in Appendix. D The op- eration also can be considered as augmenting the tail class with the head with labels. The difference is that the reference of our method is obtained from data, whereas CBDM's reference is based on the predictions and condition capacity of the model that may cause biased scores, as we have discussed in Section. [
Our method possesses greater robustness besides the state-the-of-the-ar performance.
4 EXPERIMENTS
Experimental Setup. We started by selecting two widely utilized datasets in the field of image syn- thesis, namely CIFAR10/CIFAR 100, with their long-tailed versions CIFARIOLT and CIFAR 100L..

---

Published as a conference paper at ICLR 2024
Table Ablation study for both conditional and unconditional generation on CIFARIOLT.
Model
Conditional
Batch Re-Sample
T2H
H2T
Full
FID (
IS (1)
A
25.31 10.31
7.01 7.002
B
8.1510.03 16.9210.17
C D E F G
8. 8.1640.02 16.85+0.10
16.78+0.09
8. 8.11 +0.05
16.0940.11
8.27
10.72+0.23
9.37
10.20+0.13
9.65
H
SIII
8.20.10.09
1.771
7.52
9.73
J
6.89
9.75+0.05
The construction of CIFARIOLT and CIFAR TOLT follows the methodology proposed in Cao al. (201)) wherein the size of each category exponentially decreases with its category index, adher- ing imbalance factor of imb 0.01. For the CIFARIOLT dataset, we also implement a more skewed version with the imbalance factor of imb = 0.001. Two commonly used metrics for image generation are adopted namely Frechet Inception Distance (FID) (Heusel et al.) (217)) and Inception Scores (IS) (Salimans et al.) (2016). During the inference time, we generate 50k images for the eval- uation of the metries. A DDIM sampler (Song al. 2020a) utilized with 100 steps of steps skip comparison with initial DDPM (Ho et al. 2020) 1000 steps.
Our training schedules are strictly follows the implementation of CBDM (Qin et al. 2023). which follows the DDPM settings. The experiments are conducted with two settings corresponding to the methods, unconditional generation and conditional generation. For the unconditional generation, only adjust the training strategy with label information injected into the diffusion model for optimization. At the inference stage, the unconditional diffusion model is asked to generate 50k images freely. For the conditional generation, there label information injected into the diffusion model in the training stage. While at the inference stage, the diffusion model is asked to generate 50k/ T. images for each class where L is the number of classes.
Table 2: Comparison with other long tail generation methods
CIFARIOLT
CIFARI0OLT
Datasets with different imb factors
Imb factor 0.01
Imb factor 0.001
Imb factor 0.01
Metrics
FID (L)
IS (t)
FID (J)
IS (1)
FID ())
IS (1)
CBGAN Rangwani et al. 2021
37.23
6.01
46.61
5.77
33.01
7.04
GAN
[Rangwant et al. 2022
12.86
8.56
38.71
6.89
13.96
10.01
DDPM (Ho et al. 2020)
10.72
9.37
15.00
9.16
10.25
12.96
CBDM (Qn al.
2025)
7.27
9.37
12.71
9.01
7.82
12.40
Ours (DDPM+T2H)
6.89
9.75
11.56
9.17
6.68
12.94
Ablation study. We conduct both conditional and unconditional generation experiment on CIFARIOLT, the strategies that have been discussed are applied to a base DDPM uncondi- tional/conditional generation model. As shown in the Table [T] in an unconditional generation, the Batch Re-sample with H2T and T2H could improve the performance, where H2T is more effective as we discussed in the former section. In a conditional generation, T2H is more efficient than H2T and "Full' since improving the contribution of head samples for the noisy tail sample could enhance the diversity tail classes and promoting the overall generation performance.
Table 3: Results Tiny TinyI
Comparison with other methods. We conduct conditional generation, on CIFARIOLT CIFARIOOLS, dataset. And we do comparison with two GAN- based long-tail generation methods CBGAN (Rang- wani et al.) 2021) and gSR-GAN (Rangwani et al. 2022) and one diffusion-baser methods CBDM. The results shown in Table 2
datasets with diffusion baselines
Method
FID
IS
Base DDPM
19.24
18.20
CBDM
18.07
18.01
Ours + T2H
17.81
18.12

---

Published as a conference paper at ICLR 2024
As illustrated in the Table in the comparative analysis of conditional generation, the T2H method achieved the best performance of 6.89 FID score, surpassing base DDPM models with 3.83 and CBDM with 0.38. In more skewed version of CIFARIOLT with imbalanced factor of 0.001, '2H has achieved more than 1.0 FID improvements compared with CBDM. Under the CIFAR ROOTT benchmark, our method has also achieved more than 1.0 FID than the CBDM. The results have shown that our method has obtained a further improvement in the diffusion model on the long-tailed distribution, illustrating a more effective calibration and augmentation from data distribution than from the model prediction.
We also conduct experiments on a dataset TinylmaggNer200LT with more classes and higher reso- lution, which is the long tail version of Tiny EnylmmaeNet220 (Tavanaci 22200) The metric is based on 10k generated images referenced with its validation set, as shown Table.
T2H and H2T relation with different | value. For the purpose of validating the assumption with Eq. ( we di- rectly utilize the formula to calculate (2) instead of H2T and T2H. As shown Figure. [ the case un- conditional generation, the generation performance im- proves as the 3 increases. Conversely. the case of con- ditional generation, the generation performance deterio- rates as the 3 increases.
H2T
Conditionaal Unconditional
T2H
Training Robustness of our method. In order achieve better performance for image generation, a diffusion model is required to train for long time. Due various forms of data imbalance, which is essentially the long-tail nature of the data, the model tends overfit and conse- quently leads to a decrease in performance, as observed in Figure This phenomenon is evident in both conditional and unconditional diffusion model training without any additional processing. However, when employing CBDM in conditional generation, this situation can be mitigated to some extent. However, as training progresses, CBDM augmentation heavily relies on the model's prediction for other labels, this phenomenon still exists. Nevertheless, our method effectively suppresses the issue and stabilizes performance in the long-timi training process.
Figure 5: FID scores versus different B value. The performance of H2T (uncon- ditional) and T2H (conditional) is also marked on the figure as pentagram.
DDPM Base
12
DDP
DDPM+T2H1
390
34-
425-
300
Iterations (100k)
Iterations (100k)
(a) Conditional Generation
(b) Unconditional Generation
Figure 6: FID metric versus the long time training process on CIFAR10LT dataset. Our method could achieve better performance and stabilize the generation quality.
Transfer Probability with diffusion time. We make study selecting probability (2))(2) in Eq. 8 with diffusion time. The transfer probability ExØt Pset which denotes shifting the denoise target in the training stage, counted and calculated with 100k samples perturbed with levels of noise corresponding different diffusion times within mini- binii.btch.. As shown in Figure. 7 when the diffusion step size is between 500 and 800, the probability of transferring the denoise target increases from 0 to 1 the step size increases. If the step size exceeds 800, then the transfer will
8

---

Published as a conference paper at ICLR 2024
20
CIFARI OLT CIFARI OOLT
Transfer
60-
Conditional Unconditional
Transfer
No Thaugfa
Diffusion Time
Cutting Step
Figure 7: The left subfigure the transfer probability versus diffusion time. The right subfigure
shows the FID scores with different transfer cutting time.
occur with an approximately 1.0 probability. We also restrict the transfer cutting step, denoting that the target transfer (T2H, H2T) is only allowed below this diffusion step. The FID scores decrease dramatically for both conditional and unconditional generation with cutting time range from 500- 800, which consistent with the observation of high variance phase in Xu et (2023).
RELATED WORK
Diffusion Models. For the diffusion generation model, Ho et (2020) firstly announced that the training of the model is accomplished by utilizing a weighted variational bound. [Song et al. (2020b) proposed an alternative approach constructing diffusion model, which involves utilizing stochastic differential equation (SDE) that gradually injects noise to smoothly transform a complex data distribution into known prior distribution. Karras et al. (2022) presents a design space that distinctly delineates the concrete design choices for former works. As for the diffusion process, Xu et al. 2023) has observed three phases with distinct behaviors affecting the generation diffusion model. Raya & Ambrogioni (2023) demonstrate that the diffusion process could be modeled in manner analogous to symmetry breaking in physics.
Long Tail Recognition. Long tail recognition refers the task of accurately recognizing and clas- sifying rare or infrequently occurring classes in a given dataset together with frequently occurring classes (Zhou et al.] 2022). There several approaches to address the problem, including weighting (Huang et al. 2016). logit adjustment (Menon et al., 2020 Zhou et al. 2023). robust
distributional matching (Zheng al. 2023a Chen al. 2024). and knowledge transfer (Wang) et al., 2017| Chen et al., 2022] (023b). Cui et al. (2019) declare that the number of samples increases, the diminishing phenomenon suggests that there is a decreasing marginal benefit for model to extract additional information from the data due to the presence of information overlap.
Long Tail Generation. The objective of long tail generation to generate a more balancing and diverse dataset training with long tail dataset. CB-GAN (Rangwani et al. (202I) has used a regular- izer that makes use of a pretrained classifier the training stage to ascertain the balance learning of all classes in the dataset. gSR-GAN (Rangwani et al.) 2022) observes that the performance decline long tail generation primarily occurs because class-specific mode collapse in tail classes which correlated with the spectral explosion of the conditioning parameter matrix and proposes corre- sponding group spectral regularizer. CBDM (Qin et al.) (2023) makes use of distribution adjustment regularizer in the training stage for the purpose of augmenting the tail classes.
6 CONCLUSION
The main challenge for the long-tail diffusion generation is the lack of diversity for the tail class generation. To tackle the challenge, based on the multi-target characteristic of denoising score, T2H augmentation for the estimation of noisy tail samples is achieved improving the score contribu- tion of head samples in the reference batch. At the same time, the Batch Re-sample operation helps alleviate the dominant effect of head samples on the scores and promote head-to-tail transfer. The experiments are conducted to validate our approach on multiple benchmark datasets, demonstrating effectiveness compared with baseline methods and robustness versus training time.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by the National Key R&D Program of China (No. 20222D0160702), STCSM (No. 2251 106101, No. 22511105700, No. 21DZI100100), 111 plan (No. BP0719010) and Na- tional Natural Science Foundation of China (No. 62306178). We also thank Fei Zhang for his insightful discussion.
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326) 1982.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with diseeldistriuuionaawaaaa margin loss. Advances in Neural Information Processing Systems, 32, 2019.
Mengxi Chen, Jiangchao Yao, Linyu Xing, Yu Wang, Ya Zhang, and Yanfeng Wang. Redundancy- adaptive multimodal learning for imperfect data. arXiv preprint arXiv:2310.1 14496, 2023a.
Xu Chen, Siheng Chen, Jiangchao Yao, Huangjie Zheng. Ya Zhang, and Ivor W. Tsang. Learning on attribute-missing graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(2):700-757, 2022. doi: 10. 109/TPAM1.2000.03322
Xu Chen, Zida Cheng, Jiangchao Yao, Chen Ju, Weilin Huang, Jinsong Lan, Xiaoyi Zeng, and Shuai Xiao. Enhancing cross-domain throuhh rate prediction via explicit feature augmentation. arXiv preprint arXiv:23/2.000078 2023b.
Xu Chen, Yuangang Pan, Ivor Tsang, and Ya Zhang. Learning node representations against pertur- bations. Pattern Recognition, 145:109976 2024. ISSN 0031-3205 doi: https://doi.org/100 1016/j. patcog. 2023. 109976. URL htt.tttsss/. 0000333332330006700
www.sscieeeeeeee rect wwt wtt.///ieene///ati........
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9268-9277, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in 2020.
Neural Information Processing Systems,
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for im- balanced classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5375-5384, 2016
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-122 11000011-11111 24,4,,1 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 25:26765-26577, 2022.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain. Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.073 2020.
Yiming Qin, Huangjie Zheng, Jiangehao Yao, Mingyuan Zhou, and Ya Zhang. balancng diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1443--8444 2023.
10

---

Published as a conference paper at ICLR 2024
Harsh Rangwani, Konda Reddy Mopuri, and R Venkatesh Babu. Class balancing gan with classifier in the loop. In Uncertainty in Artificial Intelligence, pp. 1618-1627. PMLR, 2021.
Harsh Rangwani, Naman Jaswani, Tejan Karmali, Varun Jampani, and R Venkatesh Babu. Improv- ing gans for toig-tailed data through group spectral regularization. In European Conference Computer Vision, pp. 426-442. Springer, 2022.
Harsh Rangwani, Lavish Bansal, Kartik Sharma, Tejan Karmali, Varun Jampani, and R. Venkatesh Babu. Noisytwins: Class-consistent and diverse image generation through stylegans. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Gabriel Raya and Luca Ambrogioni. Spontancous symmetry breaking in generative diffusion mod- els. arXiv preprint arXiv:2305. 19693,, 2023.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jaaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song and Stefano Ennon. Improved techniques for training score-based generative models. Advances in Neural Information Processing Systems, 33:12438-12445 2020.
Yang Song. Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Amirhossein Tavanaei. Embedded encoder-decoder in convolutional networks towards explainable ai. arXiv preprint arXiv:2007.067/2 2020.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, 23(7):1661-167) 2011.
Yu- Yu-niong Wang, Deva Ramanan, and Martial Hebert. Learning model the tail. Advances in Neural Information Processing Systems, 30, 2017.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- GAN: Training GANs with diffusion. In The Eleventh International Conference Learning Representations, 2023. URL htttps:://ppnreeiieeee net Ltt.re/eeeeeoeeoeeeeee-e////
Yilun Xu, Shangyuan Tong, and Tommi Jaakkola. Stable target field for reduced variance score estimation in diffusion models. arXiv preprint orXiv:2302.00670, 2023.
Lu Yang, He Jaang Qing Song, and Jun Guo. A survey on long-tailed visual recognition. Interna- tional Journal of Computer Vision, 130(7):1837-188) 2022.
Jaejun Yoo, Namhyuk Ahn, and Kyung-Ah Sohn. Rethinking data augmentation for image super- resolution: A comprehensive analysis and strategy. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3375-8384, 2020.
Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly open-vocabulary semantic segmentation. Advances in Neural Information Processing Systems, 36, 2024.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819-827.
PMLR, 2013.
Zijian Zhang, Zhou Zhao, Jun Yu, and Qi Tian. Shiftddpms: Exploring conditional diffusion models by shifting diffusion trajectories. arXiv preprint arXiv:2302.02373. 2023.

---

Published as a conference paper at ICLR 2024
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances neural information processing systems, 53:7559-7570, 2020.
Huangjie Zheng and Mingyuan Zhou. Exploiting chain rule and theor't to compare probabil- ity distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 14993-15000 15006. Cur- www.:i/innnnnneenraaeeeeeee.e.ee cc/paper files/
htt//tppps//prrceednng..oooeeoe ran Associates, Inc., 2021. URL
Huangjie Zheng, Chen, Jiangehao Yao, Hongxia Yang, Chunyuan Li, Zhang, Hao Zhang, Ivor Tsang, Jingren Zhou, and Mingyuan Zhou. Contrastive attraction and contrastive repulsion for representation learning. Transactions on Machine Learning Research, 2023a. ISSN 2835-8856.
net. forum? id= f 99TDDwwC
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilis- tic models and ceffussionbbase adversarial auto-encoders. In The Eleventh International Confer- ence on Learning Representations, 2023b. htthstt:: /ptp//ppppeeeeeeeeeeeeeee net forum ?id= /
HDxgakk9561,
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pp. 27367-27377. PMLR, 2022.
Zhihan Zhou, Jiangchao Yao, Feng Hong, Ya Zhang, Bo Han, and Yanfeng Wang. Combating representation learning disparity with geometric harmonization. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2024
A
THE EXPECTATION FORMULATION OF OPTIMAL SCORE FOR 2t.
The optimal score for 24 could be calculated as
Va, log ( Bu(polz)) v., log (ogg(arlee
Proof A.1 Based Bayes theorem and definition of expectation:
V 2. log (((x))
((x)) Va. g(11(2))
/ ((2
((,)) Ve
g(
g(ar))
/ ((2s)1r)) ((zu)
Va, log (
((ze)
/
((02)
Vx, log (( (ro ddr
((x1)
((z0l122) V., log
(2oo Va, log (
B THE SCORE FOR THE CONDITIONAL GENERATION
The optimal score for v, given could be calculated as:
N
(<1//=0 (1) .2)
Ve, log (<<ll)) ~
Ve. log (2rol0) y); (40) .20)) g(00. vo)
(10)
E, ((re 20 y)
Proof B.1
Va, log ((<ely) = E Bg( Va, log q 8)
We use importance sampling:
g( xe, v) V., log (xze (( v)
() Va log g( (
(11)
((2oly)
worog(wo
since conditional probability formula:
(( v) g( g(xelv)g(zol)) q(xelv)
()((ool))) ( v)
(12)
Compared with unconditional generation, we here calculate g(zell)) as g((x()) 3)) v)q(2u y) dzo, we utilize Monte-Carle sample as:
((vxlv) = Σ a(mel=b)
()) .)) xo ~ ((vol))
(13)
We substitute the Equation into the Eq.
12
q (2) .4) Vz, log logg(zel200.1
Σ
(14)
N
E, E,0 x .y)
20
moq W)
Here, we augment the distribution q y) for larger generation diversity. Here we analyze the (3) y) that could not be sampled during the score estimation stage in the Eq. 14 For fixed little threshold probability ps:
13

---

Published as a conference paper at ICLR 2024
for the small q(xx)(000 .v) < ps. the final scores almost are not affected by these samples.
ii) for the q( 1 .) 2 Ps. then we make assumption that, there exists (xl 2 -oo (1))) that:
Va, log q (1') 3o0 ( xe Ve log oogg(a v) (x y0 ( ~ q(Tu, yo).
(15)
We make little justification of the assumption. Suppose Iı is obtained by adding sampled noise with probability pL to (zo-3) y). then we consider another sample (# y) ~ q(zuly) with probability
( v) 2 Ps- Under the q( Ft, y) x C Cxxp(- 1 (00-=:lla) We could easily check the similar- 22l12 S -20g
203
(log P1 +log ps) following the similar procedure in Proposition. G.1 26 lies in a ball with radius
ity between 21 and dgg ||
-1og(1 + loggs) center at zı.
If there exists a (x) 3o ~ q(zo, yo) in the ball, which could be sampled during the training stage with another label vū We could approximately substitute z. with z for probability and score. The substitution error of probability could be bounded with the [g(0z|=0 yō q( g)1 I max(q(Ia)0 v), q(xe)=z) vō (logP) + 25?). And the substitution error of scores could be bounded with V/,,, log q( 12 vō') V., log qlss v)l s +. The both substitution errors are suppressed at larger timesteps with larger at.
We broad the distribution of q( to entire distribution ((2)) as reference:
N
q (6) .8) Vz, log (( v): (x0). v00) g( vo)
V z, log ((<al))
(16)
E, ((<a|=0 .!)
Note that in practice we can evaluate the density with g(<el==)) ()) as the diffusion process is not related to the label. So we here abuse the notation little with substituting q(=a|z²)) y) with q( (()) yo (1) to represent the correspondence between 20 and v
(a (9) .y) Va, log q< v); (=0 (=)), 000)) g( (17) (1) ))
2, log (()l)))=
E, (1 (20"))
Note that the sampling initially operated with label y instead of yo
()
C THE PROOF FOR THE PROPOSITION 3.1
Proposition C.1 Let q(xel=o) ) x B a(v6") exp(
|| - (
then:
20
E
q( () ()
B
(9(v0)
(8)
1) exp(
11=0 (2) (3)
" Σx q 300
(5) ()))
2
80
Proof C.1
q(02 1x (3)
/
q(valan" vo (g005 12) ))))) Ex (x (1) (3)
26 (0) Σn ke ( |z0 :
(5)
dr,
while the term in the integral of the right side could be transformed into:
gl (a)60) , No (2<e () (5))
(8)
= ʃ ((me)100 (t) )g(e 0n 900
())
2
Σe ( ( 3300
(i) (5))
( g(( 1 .30 01
(i)
2
+
00..12
< V q(0, (() yo
(6)
1p( y.0 ()))
14

---

Published as a conference paper at ICLR 2024
Here we substitute the assumption q<<a)=0 (1) -00)) x B 40 exp( mula:
11
1
into the for-
207
V (((21100) (8) 1 (((=)10)))
))0)) =
B
2 "(v6") exp(
|
12
(30
(5)
exp(
|
(3)
20
20 2
B
2 (alvb (9)
1)8 exp(
1
(6)
+
|| - 3.
207
Here we take a twice look the integral and exclude terms unrelated to xz2.
/ 1 (alu6")attt")) Da(on
2 x6 112 + ||,, 26 11:112 jdr
exp(
20
2 (4(vo Nu(vo")) / exp( | - m0 112 +1100 1 ddz
B (1)
40²
We will extract the numerator term from the exponential:
||,, = 13 + || - x001 1 (xt (1) )T (I+ 10 ) + (x+
(1)
( - 20 (5))
(1) (xt
-2xr111 2 (5) (3)
+ +000 (5))
(s)T (1)
(iyT
+ =
20
+ x
200
=2||04
F0 x
(4) (1)
112 + 0o 20
GTT
19
20
GIT
T0
())
(x)
(6)
+ x
(1) )T (1)
(3)
Since the first term related to could be integral to a constant K which do not include To and (f)
To
expp
2n de K
owing to the Gaussian distribution normalization. Then the left term:
(x) +
(1)T
(a))
+ =(6))
(1) (1) 20 Fo
(J)
(s)T (4)
U)T
(1) (i)
+ 00 20
x To
+
+
xo
2 x
2
2
(1)
1
Substitute in the initial formula, we obtain:
af 1n 2o "E ) 2 (alin )olo)")) / eexp(
((<0e=10) (5) _10))
1
x
(3) +
1 - x0 .5)
E
<
)dxa
40
2 (a(vo (2)
(o((uo")((ubb)))) Da(ub"))0 / eexp(
1
20 42 (3)
(1)
2
ddx x exp(
1|:00"
(3)
20
(
113
207
807
BK
2 (q(yb)) a(v6) 1) exp(
||
(1)
112
80?
The K is absorbed in constant B, we finish the proof.
D THE RELATION WITH CBDM LOSS AND ANALYSIS
Proposition D.1 When the diffusion model converges, the optimal minimizer c" (x v) for the
CBDM loss:
LCBDM v, = () oll + rl.
Σ
|Kco(ze. v) (oolaa.)))coo..)))))) v0)1111
y
W'EY'
could be denoted as:
c (x+,8) 1 (1+tT) |
1 LT
Σ co(
15

---

Published as a conference paper at ICLR 2024
Proof D.1 When the diffusion model converges. for specific label y, the score co s/) for other
label yf is fixed. When the loss is minimized:
rt
LCBDM (2,, t, |10 y) - cl12
(10((1))))
ca c0 2/12
[
v'EY
= c w)² E v) + e e- (z v) e+ et
7L D
Σ D'EY'
(eo(zr, c0(2x,)) (o c y/ (/( 200( (e
= =1 + 7t)o co 200(
Σ W'E
co( v/)) + Const
DI
where the Const has relation with co( v). The term related with co (2(, v) could be expressed as a quadratic form:
LCBDM(TE, y. (1 + rt)co(1,, " co(21.v) (co(xee0)7(e)
7t | V'EY
Σ Ecoooan co(2x.10))
c+ It Evey co( (2)))
= (1 + =t)[co(Ze.)) co(,,,)) co(
TDT
(1 rt)
+ T2. EVEr c0(
= (1 + +t)llelle(
D
const'
(1+
When the LCBDM converges to the minimum. the L2 norm should approximately near zero, so that we have:
c" (* ~ tT (1+tT) |J
tt
Σ Eco(xi.0))
Analysis This approach has two limitations. Firstly, referring to scores from other labels requires training the entire model within conditional generation framework, thus restricting its applicability. Secondly, relying on scores from other labels for the same input (I₂) introduces potential biases, particularly when there to some degree of offset. In contrast, our method addresses these limitations. First, it can be used in both conditional and unconditional generation scenarios. Second, we utilize the inherent similarity of data in the L2 space where the diffusion model operates. For example, in Figure. 2 we consider the similarity between the red airplane and the red car, ensuring that our enhancement more logically grounded.
substantial semantic difference between the classes. This can lead
E CLASS- WISE FID SCORES T2H COMPARISON TO BASE DDPM.
We calculate the FID score using 5k images of T2H and base DDPM for each class on cifar1OLT OLT datasets. The final column of the table overall performance utilizing 50k images.
Table 4: Class-wise FID scores of T2H comparison to base DDPM.
Class
Airpl
Auto
Bird
Cat
Dog
Frog
Horse
Ship
Truck
All
Pas
0.403
0.241
0.145
0.086
0.052
0.031
0.018
0.01
0.0067
0.004
1.0
Base
31.49
15.20
40.81
32.32
32.32
36.34
40.47
23.20
26.31
23.31
10.72
T2H
31.84
14.58
19.42
28.32
18.31
26.83
29.90
17.70
21.04
22.82
6.89()3.83)
We have also modified the asssignment of head and tail classes based whether they animals or vehicles. As shown in the table below, By altering the assignment classes, our method demon- strates greater improvement compared to the baseline.
16

---

Published as a conference paper at ICLR 2024
Table 5: Class-wise FID scores of T2H comparison base DDPM with shifted categories.
Class
Horse
Bird
Frog
Deer
Dog
Cat
Truck
Airpl
Auto
Ship
All
Pas
0,403
0.241
0.145
0.086
0.052
0.031
0.018
0.01
0.0067
0.004
1.0
Base
20.19
21.99
22.45
20.57
38.07
32.73
24.87
49.24
29.48
39.12
11.79
T2H
20.16
18.52
19.83
16.23
24.60
27.50
14.73
30.08
20.99
21.41
7.15(-4.64)
F
THE IMPACT OF DATASET IMBALANCE ON PERFORMANCE.
We have investigated the impact dataset balance on performance. As illustrated in the following table, computed the FID scores using different reference sets: one balanced and the other long- tailed sampled. It is observable that, despite the long-tail dataset consisting of real images, its performance metrics inferior those of the balanced generated dataset, due its significant imbalance.
Table 6: The FID (IS) scores for imbalance real dataset and generation dataset. Bal means balanced dataset or generation, ref means reference real dataset
Bal/no Bal
No Bal
Bal
Method
No bal ref
T2H
Base
T2H
Base
Bal ref
26.22(5.80)
45.79 28.68
46.66(7.03 30.34
8.38(9.62) 33.19
12.95(9.52) 38.81
No-Bal ref
G THE VALIDATION OF TRANSFERRING TARGETS WITH T2H AND H2T.
In this section, we aim to verify the reliability and correctness of the transfer target. Suppose we have clean sample xn, perturbed with noise NN of) obtaining the noisy sample x, with probability Pt- B expl 1120-22ll where B x L. is some normalizing constant.
20;
Then Ft involving the T2H or H2T algorithm and transfer the target as Th with probability (s)((2) = g(x,|2(1) (*) "
(x)
15 C) 2 where q( (x) .y0") = B expl -== also calculated with
20
Gaussian kernel. Firstly, let discuss the similarity of æo and x measured with ||20 -=<²>113 with following proposition.
Proposition G.1
The L2 similarity with xo and my bounded by the ps and (oct
|| 1 1 -20 (log(P: (2)) + log(pt + Poet Poct(2)) log o1).
(18)
Proof G.1
|| 1
xxl3
2000 - 11.133
11z00
20=1 2//0111 $ -20
207
20
||01
xz
||20 xel13
20, (log expl
log expl
20
20
20, Pt Pset (z) E, q(areaab))
.900 (1)
B2
2 200 Psea(z)) logíp: + (=)) + 2 log 0z)
where involves utilizing the triangle inequality. And 2 comes from 2(0l1200 226)) > q(2e/20 yo) + q(xx)=(()) (2))
17

---

Published as a conference paper at ICLR 2024
(2) As can be observed from the proposition above, the similarity an between and To is bounded by an upper limit. Moreover, in the sampling process, the larger the values ps and Psel (2) the tighter this bound becomes.
Furthermore, from an alternative viewpoint, we evaluate the probability of obtaining using a clean To
(2) within the framework of one-to-one clean-to-noisy mapping.
Proposition G.2 The probability pi of obtaining 24t from To under single denoising target sce- nario could be:
p 2 Part
(19)
1 Psel (2)
where (z) comes from Equation. 8 and q yo) x B exxpl || () 2e
Proof G.2
g (2) (z))
300
Peel(2)
Σ, (<1el210) -10
(x 10 (2) (2)
30
g yo) + ( 1a
(3)
30
(3
because Pt (( yu) and pt q( mm ,36²)) (()) under the assumption of Equation.
so that
Pi
(2))(2) v
pł + Pt
so we obtain:
Port (z) Pt.
pi 2 1 (2)
We have obtained a lower bound for př. If we assume Pt = (sci(2) 0.5 then the 2 0.5 where (=) also valid noise-clean pair for 2, + z under the single target scenario.
H EXPERIMENTS ON IMAGENETLT DATASET
We conduct method on large scale datasets ImagenetLT, the performance is shown in the follow- We generate 20k images with 1000 classes and use the balanced validation set with 20k images the reference set for the calculation of FID scores. As shown in the Table, Our results large-scale are consistent to the observations on small-scale data, which validates the effectiveness of our method scale.
ing Table. 7
Table 7: Experiments large scale dataset ImagenetI.T
Method
FID
IS
base DDPM
26.95
15.99
CBDM
28.12
15.86
T2H (Ours)
25.42
15.96
FINETUNING EXPERIMENTS FROM NORMAL TRAINED MODEL
We also conduct the experiments of finetuning with the model pre-trained with normal denoising loss function. The finetuning starting step is ranging from 100k to 500k, and the results with no pretriaining and finetuning also provided. The conditional model is finetuned with T2H strategy, while the unconditional model is finetuned with H2T strategy.
As shown in the table, finetuning is capable of further improving model performance based pre- training. However, as the number of pretraining steps increases, the extent improvement gradually diminishes and eventually becomes stable.
18

---

Published as a conference paper at ICLR 2024
Table 8: Finetuning from normal pretrained models with different training steps.
Pretrained
Pretrained Steps
No finetune
No pre-train
100k
200k
300k
400k
FID
25.31
16.09
18.13
20.31
21.65
21.10
Base Uncond
IS
7.01
8.27
7.93
7.86
7.32
7.33
FID
10.20
6.89
7.48
7.87
8.01
8.12
Base Cond
IS
9.25
9.75
9.64
9.63
9.62
9.56
J IMPLEMENTATION DETAIL OF TOY GAUSSIAN EXAMPLES IN FIGURE. 4
We random sample 10k samples from distribution (NO0, 4), 0.2I) as head samples { {=o- and 0. samples from (N0,, -4, 0.2f) simulating tail samples fao (7))) The empirical distribution of the overall dataset could be denoted as:
Pdata (x) = Σ 8(x (H.)) + Σ 8(x 2 (35)
Simulating the forward diffusion process, convolute the empirical distribution with Gaussian noise ((0 o)
Pa. (x+) = Panta (N(0, Σ N( (1)) .ff + E (T)) .)
(20)
The score estimation is calculated as vx, log (Ix), we substitute the result in Eq. (0
Ve, log (xx) Vze log( Σ Na0 (H.) ,0 + Σ N(ax (T,) ,o(
N.Nia (H.) .otl) + E, Nau (T,) , x
N( (11)) of1) + Σ N(zi
(T,) ,f
Σ, (It To (Ht)) (Hil) ((z 11)) of1) E, - 4o W (T)) off)
N
(11)) ,otT) + EE,N(1s; No1i2 (T,) ,
The last step because of:
| x11
Vo,N(En2 of1 V. Caxpeep exp(
20
|| - x
12, x
-Cexpl
20
20
N(x of
The number of points in the head class is 100 times greater than the tail class, resulting in an imbal- ance factor 0.01. It can be observed that the score distribution across the entire space is dominated by the head class, leading to highly imbalanced dataset generation.
K FUTURE WORK
Now, diffuuion models are widely applied to the generation and learning research in multimodal fields. Because the definition of the long-tail problem is not yet clear the context of multimodal data, more efforts can be used to define the data defects in multimodal data (Chen et al. a (202aa) such as long tail distribution, how to solve the long-tail generation problem multimodal scenarios, and more downstream tasks in multimodal contexts (Zhang et al. 2024
19

---

Published as a conference paper at ICLR 2024
L VISUALIZATION RESULTS
Figure 8: Visualization of generation results of CIFAR IOLT dataset with T2H conditional generation
20

---

Published as a conference paper at ICLR 2024
Figure 9: Visualization of generation results of CIFAR10OLT dataset with T2H conditional genera- tion
21

---

Published as a conference paper at ICLR 2024
Figure 10: Visualization of generation results of TinyImageNet200 dataset with T2H conditional generation
22


Reviews:
[{"review_id": "rhha64ZzN5", "reviewer": "Reviewer_i52Z", "summary": "This paper addresses the problem of long tail diffusion model generation. In this paper, a directional calibration for the estimation of\nnoisy tail sample score is performed towards the clean head samples (T2H), leveraging the similarity within the data distribution from head to tail classes.", "questions": "1. T2H enhances the quality of image generation for the current category by calculating the multi-nominal distribution in Equation 11, and then samples $z$ from this distribution, and selecting categories $y_z$ with a sample density greater than that of the category $y_i$. While the authors provide detailed theoretical proof, the transition from equation 10 to equation 11 seems to lack coherence. When sampling z, why not model the distribution based on the similarity between the original images (the current sample image and other images in the batch, as in equation 10), rather than through the noisy version of the current sample image and other images in the batch?\n2. In the T2H method mentioned, the distribution $p_{sel}(z)$ adds weight to the other images within the batch, encouraging similar images to have a greater influence on the current sample. So, what would happen if we do not consider the distribution $p_{sel}(z)$ and assume that all images will have an effect? Also, if we keep the distribution $p_{sel}(z)$ unchanged and assume that all $y_z$ will have an impact, not just those with a high sample density, would this be more reasonable?\n3. The authors seem to lack an analysis in their experiments regarding how much growth the proposed solutions have brought to both the head and tail categories in the dataset, respectively.\n4. How effective are the proposed methods on larger datasets (ImageNet-LT)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper propose a strategy denoted as ”Batch Resample” to sample a more balanced reference batch. \n2.This paper has developed a method denoted as T2H based on the multi-target nature of score estimation to effectively calibrate and enhance the generation of tail classes in the semantic formation period, thereby significantly improving the overall generation performance.", "weaknesses": "1. The paper introduces two optimization methods, Batch Resample and T2H. The Batch Resample is a common solution to the long-tailed problem, and it does not seem to bring any novelty. \n2. There is a lack of connection between the proof and implementation of T2H, especially the transition from equation 10 to equation 11.\n3. The author lacks experimental validation on large-scale datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L7ZMuNzRMQ", "reviewer": "Reviewer_PNyr", "summary": "This paper presents an algorithm for calibrating diffusion models for long-tailed data. Based on the recent work by Xu et al., 2023, in this work, the target for the score estimation is re-written as a mixture of multiple targets computed from a reference batch. The reference batch, unlike the work of Xu et al,. 2023, is sampled from a class-balanced distribution in order to calibrate the model for tail-class samples. For the conditional generation, the authors further propose a strategy called Tail-to-Head (T2H), where the target of the score estimation for a tail class sample is chosen to be a clean head class sample. The paper also suggests a variant called Head-to-Tail (H2T), where in this case a noisy head sample takes a clean tail sample as a target. The proposed method is demonstrated to improve image fidelity when applied to popular imbalanced image benchmarks.", "questions": "- I'm very confused about the argument around Proposition 3.1. First, it is not clear from the text what the \"score weight\" or \"weight of score\" means and why it is important. I guess this term refers to the mixing coefficients (normalized transition probabilities) appearing in the multi-target representation of the score function. Secondly, what is the author trying to make out of Proposition 3.1? How is it related to the algorithm? The sentence \"Intuinitively (intuitively, I guess), improving the score weight of head-class samples for the noisy $x_t$ from tail-class samples, increases the generation diversity of tail categories\" is not clear at all. What is \"improving\" the score weight? Is it increasing the value of the score weight? If so, how is it related to increasing the generation diversity?\n- The introduction of the selection procedure (equation (11)) comes without sufficient explanation. If I have to understand it, the random sampling for $z$ can be understood as a Monte-Carlo estimator of equation (8); But then the proposed algorithm alters the target score if $q(z_y) \\geq q(y_i)$; as a result, the target of the score estimation becomes something different, yielding a biased estimator of the original target. Is this intended? If so, how do you guarantee that the proposed algorithm is learning a correct target?\n- In the beginning you assumed $q(x_t|x_0, y_0) \\propto B q(y_0)^\\beta \\exp(-\\Vert x_t-x_0\\Vert_2^2/2\\sigma_t^2)$, but $p_\\text{sel}(z)$ described to computed only with $C \\exp(-\\Vert x_z-x_t\\Vert_2^2/2\\sigma_t^2)$ without $q(y)^\\beta$ term. Is this a typo or am I missing something here?\n- Similarly, I don't get the intuition behind the H2T, since from the beginning I failed to follow how Proposition 3.1 is linked to the T2H. What is the meaning of setting $\\beta=-1$ and why is it beneficial for unconditional sampling? \n- The experiments mostly show the results of T2H on class conditinal generation and H2T for unconditional generation, but how do they work for opposite settings (T2H for unconditional and H2T for conditional)? Is the direction of transfer matters? Or both direction helps in either cases, but the effectiveness of transfer may vary depending on the task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper is tackling an interesting problem.\n- The proposed method is simple and easy to implement.\n- The experimental results, at least for the simple benchmarks, is promising.", "weaknesses": "- The presentation is poor. I don't really follow the description of the intuition behind the presented method. Please see my questions below.\n- Again, probably related to the first point, I don't see how the proposed method yields a valid regression target. The algorithm seems to be a heuristics without considering correctness.\n- Batch-Resample cannot be considered a novel contribution, as it is one of the standard approaches one could take for a problem involving a long-tailed dataset (e.g., long-tailed image classification).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pN9gvyUfTX", "reviewer": "Reviewer_Uefc", "summary": "In this paper, the author's primary focus is on tackling the diffusion model generation tasks in the context of long-tailed training scenarios. Different from previous works that enhance tail generation by relying on the abundant diversity derived from the head class (the condition capacity of the model prediction), this paper directly establishes the knowledge transfer from head data samples, based on the multi-objective characteristics of the score function in the diffusion process. A directional calibration for the estimation of noisy tail sample score is performed towards the clean head samples (T2H), leveraging the similarity within the data distribution from head to tail classes. Meanwhile, H2T is proposed for the unconditional generation. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "1. In the paper, T2H is utilized to estimate noisy tail samples using clean head samples, and this process is explained as a way to enhance the performance of long-tailed conditional generation. However, I have difficulty understanding why H2T, which estimates noisy head samples using clean tail samples, would improve the performance of unconditional generation. This is because, in my view, the style of the tail samples should be relatively homogeneous and might not provide substantial knowledge for estimating the head samples. Could the authors please provide an explanation?\n2. In Table1, why the \"Full\" (allowing both transfer directions) performs worse than the H2T and T2H (only allowing one direction) under the unconditional generation? Meanwhile, I'd like to see the performance of \"Full\" for the conditional generation.\n3. The author should explain more about why the baseline DDPM has a higher IS than T2H (Table 2 and Table 3).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper presents a novel idea that calibrates the estimation of noisy tail sample score by the clean head samples instead of relying on the condition capacity of the model prediction. \n2. This paper has a well-organized structure which makes it easy for readers to understand the research.\n3. The paper is well-supported by a strong theoretical foundation, which robustly underpins the proposed solutions.\n4. The text is clear with a good writing style.", "weaknesses": "1. The explanations of some figures are not sufficiently comprehensive. For instance, the author could enhance the clarity of differentiation by adding annotations or captions to both the top and bottom parts of Figure 1.\n2. Some problems, which I will raise in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)