Addressing Negative Transfer in Diffusion Models
Authors: Hyojun Go
Jinyoung Kim
Yunsung Lee
Seunghyun Lee
Shinhyeok Oh
Hyeongdon Moon
Seungtaek Choi
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Models
Multi-Task Learning
TL;DR: Analyzing and improving diffusion models via a multi-task learning perspective

Abstract: Diffusion-based generative models have achieved remarkable success in various domains. It trains a shared model on denoising tasks that encompass different noise levels simultaneously, representing a form of multi-task learning (MTL). However, analyzing and improving diffusion models from an MTL perspective remains under-explored. In particular, MTL can sometimes lead to the well-known phenomenon of $\textit{negative transfer}$, which results in the performance degradation of certain tasks due to conflicts between tasks. In this paper, we first aim to analyze diffusion training from an MTL standpoint, presenting two key observations: $\textbf{(O1)}$ the task affinity between denoising tasks diminishes as the gap between noise levels widens, and $\textbf{(O2)}$ negative transfer can arise even in diffusion training. Building upon these observations, we aim to enhance diffusion training by mitigating negative transfer. To achieve this, we propose leveraging existing MTL methods, but the presence of a huge number of denoising tasks makes this computationally expensive to calculate the necessary per-task loss or gradient. To address this challenge, we propose clustering the denoising tasks into small task clusters and applying MTL methods to them. Specifically, based on $\textbf{(O2)}$, we employ interval clustering to enforce temporal proximity among denoising tasks within clusters. We show that interval clustering can be solved using dynamic programming, utilizing signal-to-noise ratio, timestep, and task affinity for clustering objectives. Through this, our approach addresses the issue of negative transfer in diffusion models by allowing for efficient computation of MTL methods. We validate the efficacy of proposed clustering and its integration with MTL methods through various experiments, demonstrating 1) improved generation quality and 2) faster training convergence of diffusion models. Our project page is available at https://gohyojun15.github.io/ANT_diffusion/.

Addressing Negative Transfer in Diffusion Models
Hyojun Go JinYoung Kim Yunsung Lee
Seunghyun Lee
Shinhyeok Oh
Hyeongdon Moon Seungtaek Chois1
Twelvelabs Wrtn T Riidd EPFL Yanolja
(gohyojun15, seago0828)0gmail. com', sungdurtn. io?
{seunghyun. lee
shinhyeok. ohworiiiddd..roo co" hyeongdon moondepfl seungtaek. seungtaas..hooilyynloj.... com"
Abstract
Diffusion-based generative models have achieved remarkable success in various domains. It trains shared model on denoising tasks that encompass different noise levels simultancously, representing a form of multi-task learning (MTL). However, analyzing and improving diffusion models from an MTL perspective remains under- explored. particular, MTL can sometimes lead to the well-known phenomenon of negative transfer, which results in the performance degradation of certain tasks due to conflicts between tasks. In this paper, we first aim to analyze diffusion training from MTL standpoint, presenting two key observations: (01) the task affinity between denoising tasks diminishes as the gap between noise levels widens, and (02) negative transfer arise in diffusion training. Building upon these observations, we aim to enhance diffusion training by mitigating negative transfer. To achieve this, propose leveraging existing MTL methods, but the presence of huge number of denoising tasks makes this computationally expensive to calculate the necessary per-task loss or gradient. To address this challenge, we propose clustering the denoising tasks into small task clusters and applying MTL methods to them. Specifically, based on (02), we employ interval clustering enforce temporal proximity among denoising tasks within clusters. We show that interval clustering can be solved using dynamic programming, utilizing signal-to- noise ratio, timestep, and task affinity for clustering objectives. Through this, our approach addresses the issue of negative transfer in diffusion models by allowing for efficient computation of MTI. methods. We validate the efficacy of proposed clustering and integration with MTI. methods through various experiments, demonstrating 1) improved generation quality and 2) faster training convergence of diffusion models. Our project page available at https://// //go:///oooooo/o gi thub. io/ ANT difffiffioo fuuiiin/
1 Introduction
Diffusion based generative models [20, 66, 71] have accomplished remarkable achievements in vari- generative tasks, including image [8], video [21, 23], 3D shape [44, 54], and text generation [38]. In particular, they have shown excellent performance and flexibility in a wide range of image gener- ation settings, including unconditional [28, 47], class-conditional [22], and text-conditional image generation [1, 48, 55]. Consequently, improving diffusion models has garnered significant interest.
The framework of diffusion models [20, 66, 71] comprises gradually corrupting the data towards given noise distribution and its subsequent reverse process. A model optimized by minimizing the weighted sum of denoising score-matche matching losses across various noise levels [20, 69] for learning the reverse process. This can be interpreted as diffusion training aiming to train a single shared model to
*Co-first author
1,2k4,5 done while at Riiid
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

denoising its input across various noise levels. Therefore, diffusion training is inherently multi-task leamning (MTL) in nature, where each noise level represents a distinct denoising task.
However, analyzing and improving diffusion models from an MTL perspective remains under- explored. In particular, sharing one model between tasks may lead to competition between conflicting tasks, resulting in a phenomenon known as negative transfer [24, 25, 57, 78], leading to poorer performance compared to learning individual tasks with separate models. Negative transfer has been a critical issue in MTL research, and related works have demonstrated that the performance of multi-task models can be improved by remediating negative transfer [24, 25, 57, 78, 83]. Considering these, we argue that negative transfer should be investigated in diffusion models, and if present, addressing it a potential direction for improving diffusion models.
In this paper, we characterize how multi-task diffusion model is, and whether there exists negative transfer in denoising tasks. In particular, (01) first observe that task affinity [12, 78] between two denoising tasks is negatively correlated with the difference in noise levels, indicating that they may be less conflict as the noise levels become more similar [78]. This suggests that adjacent denoising tasks should be considered more harmonious tasks than non-adjacent tasks in terms of noise levels.
Next, (02) we observe the presence of negative transfer from diffusion model training. During sampling within a specific timestep interval, utilizing a model trained exclusively on denoising tasks within that interval generates higher-quality samples compared to model trained on all denoising tasks simultaneously. This finding implies that simultancously learning all denoising tasks can cause degraded denoising within a specific time interval, indicating the occurrence of negative transfer.
Based on these observations, we focus on improving diffusion models by addressing negative transfer. To achieve this, we first propose to leverage the existing multi-task learning techniques, such as dealing with issues of conflicting gradients [5, 83], differences in gradient magnitudes [42, 46, 64], and imbalanced loss scales [4, 16, 29]. However, unlike previous MTL studies that typically focused on small sets of tasks, the presence of a large number of denoising tasks (es thousands) in diffusion models makes it computationally expensive since MTL methods generally require calculating per-task loss or gradient in each iteration [4, 5, 16, 24, 29, 42, 46, 64, 78, 83].
To address this, we propose a strategy that first clusters the entire denoising tasks and then applies multi-task learning methods to the resulting clusters. Specifically, inspired by (O1), we formulate the interval clustering problem which groups denoising tasks by pairwise disjoint timestep intervals. Based on the interval clustering, we propose timesteps, signal-to-now ratios, and task affinity score- based interval clustering and show that these can be clustered by dynamic programming as [2, 76, 49]. Through our strategy, we can address the issue of negative transfer in diffusion models by allowing for efficient computation of multi-task learning methods.
We evaluated our proposed methods through extensive experiments on FFHQ [27]. CelebA-Hi [26]. and ImageNet [7]. For a comprehensive analysis, we employed various models, including Ablated Diffusion Model (ADM) [8], Latent Diffusion Model (LDM) [56]. and Diffusion Transformer (DiT) [52]. These models represent diverse diffusion architectures spanning pixel-space, space, and transformer-base
datasets:
paradigms. Our results underscore a significant enhancement in image generation quality, attributed to marked reduction in negative transfer. This affirms the merits of our clustering proposition and its synergistic integration with MTL techniques.
2 Related Work
Diffusion Models Diffusion models [20, 66, 71] are a family of generative models that generate samples from noise via a learned denoising process. Diffusson models beat other likelihood- based models, such as autoregressive models [62, 75]. flow models [9, 10], and variational autoencoders [32] in terms of sample quality, and sometimes outperform GANs [14] in certain cases [8]. Moreover, pre-trained diffusion models can be easily applied to downstream image synthesis tasks such as image editing [30, 45] and alug-and-play generation [13, 15]. From these advantages, several works have applied diffusion models for various domains [3, 23, 38, 44, 54] and large-scale models [48, 56, 58].
Several studies have focused on improving diffusion models various aspects, such as architecture [1, 8, 28, 52, 82), sampling speed [33, 60, 67], and training objectives [6, 17, 31, 70, 74]. Among these, the most closely related studies are improving training objectives, as we aim to enhance optimization between denoising tasks from the perspective of multi-task learning (MTL). Several works [31, 70, 74]

---

redesign training objectives to improve likelihood estimation. However, these objectives may lead to sample quality degradation and training instability and require additional techniques such as importance sampling [70, 74] and sophisticated parameterization [31] to be successfully applied. On the other hand, P2 [6] proposes a weighted training objective that prioritizes denoising tasks for certain noise levels, where the model is expected to learn perceptually rich features. Similar to P2, we aim to improve the sample quality of diffusion models from an MTL perspective, and we will show that our method is also beneficial to P2.
As a concurrent work, MinSNR [17] shares a common insight with us that diffusion training is essentially multi-task learning. However, their observation lacks a direct connection to negative transfer in terms of sample quality. They address the instability and inefficiency of multi-task learning optimization in diffusion models, mainly due to a large number of denoising tasks. In contrast, our work delves deeper into exploring negative transfer and task affinity, and we propose the application of MTL methods through task clustering to overcome the identified challenges in MinSNR.
Multi-Task Learning Multi-Task Learning (MTL) is an approach that trains a single model to perform multiple tasks simultancously [57]. Although sharing parameters between tasks can reduce the overall number of parameters, it may also result in a negative transfer, causing performance degradation because of conflicting tasks during training procedure [24, 25, 57, 78].
Prior works have tracked down three causes of negative transfer: (1) conflicting gradient, (2) the difference in gradiens magnitude, and (3) imbalanced loss scale. First, Confficting gradients among different tasks may negate each other, resulting in poorer updates for a subset of, or even for all tasks. PCgrad [83] and Graddrop [5] mitigate this by projecting conflicting parts of gradients and dropping elements of gradients based on the degree of conflict, respectively. Second, tasks with larger gradients may dominate tasks with smaller gradients due to differences in gradient magnitude across tasks. Different optimization schemes have been proposed equalize gradient magnitudes, including MGDA-UB [64], IMTL-G [42], and NashMTL [46]. Similarly, imbalanced loss scales may cause tasks with smaller losses to be dominated by those with larger losses. To balance task losses, uncertainty [29], task difficulty [16], and gradient norm [4] is exploited.
Adapting MTL methods and negative transfer formulation to diffusion models is challenging since these techniques are typically designed for scenarios with a small number of tasks and easily measurable individual task performance. Our goal is address this challenge and demonstrate that observing negative transfer in diffusion models and mitigating it can improve them.
3 Preliminaries and Observation
We first provide the necessary background information on diffusion models and their multi-task nature. Next, we conduct analyses that yield two important observations: (01) task affinity between two tasks is negatively correlated with the difference in noise levels, and (02) negative transfer indeed exists in diffusion training, i.e., the model is overburdened with different, potentially conflicting tasks.
3.1 Preliminaries
Diffusion model [20, 66, 71] consists of two processes: a forward process and a reverse process. The forward process gradually injects noise into a datapoint xo to obtain noisy latents {xi,..., XT} as: q(xe|xo) N (xxelaxx0.033 of1), ((xe|xx) = (xxx (o? - o)),, 1 15 < <t< T (I) where 0g, oe characterize the signal-to-noise ratio SNR(t) = of and atjs = a./as. Here, SNR(() decreases in such that by the designated final timestep ( = T. q(xx) R 10.
The reverse process a parameterized model trained to restore the original data from data corrupted during the forward process. The widely adopted training scheme uses simple noise-prediction objective [8, 20, 34, 56, 59] that trains the model to predict the noise component e of the latent X, XxX0 + oe, ~ (N.. I). More formally, the objective is follows:
Iisipple = [.,),.<(-+))
where I,, = ||s
1)
Let us denote by D' the denoising task at timestep t trained by minimizing the loss L, (Eq. 2). Then, since a diffusion model jointly learns multiple denoising tasks (Di]t=1.... using single shared model 6ø, it can be regarded as a multi-task learner. Also, we denote by eal the set of tasks (D), D(1-1 ..... .'..,D'a henceforth.
3

---

200
4000
800
1000
1.0
4.0
1.0
0641
1000
1.0
5.0 6.0
40
60
-.0
200
4.0
08
400
0.6
2.0
4,0
0.5
0,2
600
5,
600
04
0.0
0.4
02
A1 4,0
2.0
1000
9.0
1002
6.0
(a) ADM (timestep 1)
(b) ADM (log-SNB)
(c) LDM (timestep t)
(d) LDM (log-SNR)
Figure I: Task affinity scores plotted against timestep and log-SNR axes in ADM and LDM. As the timestep and SNR differences decrease, task affinity increases, implying more aligned gradient directions between denoising tasks and reduced negative impact their joint training.
0.25
N76
-0.25
0.50
-0.75
[1,200] [201]400] [401,600) 600] (800
[1,200]
[201,400] [401,600] [60], [,01, [801,
Denoising tasks DI
Denoising tasks D
(a) ADM
(b) LDM
Figure 2: Negative transfer gap (NTG) with FID score of ADM and LDM for denoising tasks D!:e), NTG is negative, -tainet model outperforms the entire denoising tasks-trained model in terms of denoising latent (xr hee---) showing the occurrence of negative transfer. Negative transfer occurs in both ADM and LDM.
3.2 Observation
By considering diffusion training as a form of multi task learning, we can analyze how the diffusion model learns the denoising task. We experimentally analyze diffusion models with two concepts in multi-task learning: 1) Task affinity [72, 12]: measuring which combinations of denoising tasks may yield more positive impact on performance. 2) Negative transfer [68, 24, 25, 57, 78, 83]: degradation in denoising tasks caused by multi-task learning. We use lightweight ADM [8] used in [6] and LDM [56] with FFHQ 256x256 dataset [27] for analyze diffusion models trained on both pixel and latent space.
(01) Task Affinity Analysis We first analyze how the denoising tasks D(1.2] relate to each other by measuring task affinities [72, 12]. In partieular, we adopt the gradient direction-based task affinity score [78]: for two given tasks D' and Di, we calculate the pairwise cosine similarity between gradients from each task loss, i.e., Vgli and Vol.j, then average the similarities across training iterations. Task affinity score assumes that cooperative (conflicting) tasks produce similar (conflicting) gradient directions, and it has been correlate with the MTL model's overall performance [78]. Although there have been attempts to divide diffusion model phases using signal-to-nois ratio [6] and trace of covariance training targets [81], we the first to provide an explicit and fine-graine ane-grained analysis of task affinities among denoising tasks.
In Fig. 1, visualize the task affinity scores among denoising tasks, for both ADM and LDM, with both timestep and log-SNR as axes. As can be seen Fig. 1, task affinity between two tasks D', Dí is high for neighboring tasks, i.e., R j, and decreases smoothly the difference in SNRs (or timesteps) increases. This suggests that tasks sharing temporal/noise-leve proximity can be cooperatively learned without significant conflict. Also, this result hints at the possibility that denoising tasks for vastly different SNRs (distant in timesteps) may potentially be conflicting.
(02) Negative Transfer Analysis Next, show that there exist negative transfers among different denoising tasks DİL.T], Negative transfer refers to multi-task learner's performance degradation due to task conflicts, and it can be identified by observing the performance gap between a multi-task leamner and specific-task learners. For case of observation, group up tasks by intervals, based on

---

the observation (01) that more neighboring tasks in timesteps have higher task affinity. Specifically, we investigate whether the task group Dit,. suffers negative impacts from the remaining tasks.
To quantify the negative transfer, we follow the procedure: First, we generate samples { using model trained on all denoising tasks D Next, we repeat the same sampling procedure, except we replace the model with model trained on Dita.fa) for the latent (xx)oe(t),t)) .t25 We denote the resulting samples by (x). If [x((,.22)) exhibits superior quality compared to {xo). it indicates that the model trained solely on Di performs better in denoising the latent {xt (xettt dta than the model trained on the entire denoising task. This suggests that Diti. suffers from negative transfer by learning other tasks. More formally, given a performance metric P. FID [18] in this paper, we define the negative transfer gap:
NTG == P([x) (alp)) - P((x))),
where NTG 0 indicates that negative transfer occurs. The relationship between the negative transfer gap in previous literature and our negative transfer gap is described in Appendix A.
We visualize the negative transfers among denoising tasks for both lightweight ADM [6, 8] and LDM [56] in Fig. 2. The results indicate that negative transfer occurs in three out of the five considered task groups for both models. Notably, negative transfers often have a significant impact, such 7.56 increase in FID for ADM in the worst case. Therefore, we hypothesize that there is room for improving the performance of diffusion models by mitigating negative transfer, which motivates us to leverage well-designed MTL methods for diffusion training.
Methodology
In Section 3.2, make two observations: (01) Denoising tasks with a larger difference in / and SNR(() exhibit lower task affinity, (O2) Negative transfer occurs in diffusion training. Inspired by these observations, we aim to remediate the negative transfer in diffusion by leveraging MTL methods. Although MTL methods are reported effective when there are only a few tasks, they are impractical for diffusion models with large number of denoising tasks since they require computing per-task gradients or loss at each iteration. In this section, to deal with challenges, we propose a strategy that first groups the denoising tasks task clusters and then applies the multi-task learning methods by regarding each task cluster as one distinct task.
4.1 Interval Clustering
Here, we first introduce scheme that groups all denoising tasks D'1,7] into small number of task clusters. This is a necessary step for applying well-established MTL methods, for they usually involve computationally expensive subroutines such as computing per-task gradients or loss in each training iteration. Our key idea is to enforce temporal proximity of denoising tasks within task clusters, given our observation (01) that task affinity is higher for tasks closer in timesteps. Therefore, we assign tasks in pairwise disjoint time intervals.
To obtain the disjoint time intervals, we leverage an interval clustering algorithm [2, 49] that op- timizes for various clustering costs. In our case, interval clustering assigns diffusion timesteps
X' = [1,..., , T} io contiguous intervals J,, J,...., Ta. with I J4 n X, where I[ denotes disjoint union. Let 1; = ra). 4 < r; for
I....,& k. then we have ( = 1, and 4+1 1
(i < k and Fж = T). The interval clustering proplem is defined as:
min
Lauuster (I: n nx).
where Laluster denotes the cluster cost.
Generally, it is known that an interval clustering problem of n data points with k intervals can be solved via dynamic programming in (n)) [49], where wn)) is the time required to calculate the one-cluster cost for Laluster If the size of each cluster is too small, it is challenging to learn the corresponding task cluster, so we add constraints on the cluster size for dynamic programming.
More details regarding the dynamic programming algorithm can be found in Appendix G.
remains to design the clustering cost function Taalatee to optimize for. We present three clustering cost functions: timestep-based SNR-based, and bradient-based.
5

---

1. Timestep-based Clustering Cost Intuitively, one simple clustering cost is based on timesteps. We use the absolute timestep difference for the clustering objective by setting Letuster (T, n X)) Dis,, II Illeenter - i|l in Eq. 4 where ( leenter denotes the center of interval Tg. The resulting intervals divide up the timesteps into & uniform intervals.
2. SNR-based Clustering Cost Another useful metric to characterize a denoising task is its signal-to-noise ratio (SNR). Indeed, it has been previously observed that a denoising task encounters perceptually different noisy inputs depending on its SNR [6]. Also, we already observed that denoising tasks with similar SNRs show high task affinity scores (see Section 3.2). Based on this, we use the absolute log-SNR difference for clustering cost. We define the clustering cost as Liiuster (I, 0 X) Ei=, log (N log (N.
3. Gradient- Clustering Cost Finally, we consider the gradient direction- task affinity scores (see Section 3.2 for definition) for clustering cost. Task affinity scores have been used as a metric to group cooperative tasks [78]. Based on a similar intuition, we design a clustering cost as follows: Leluster (4, O X) Ei. TAS(ttenter 1) where TAS(-) is the gradient-based task affinity score. While leveraging more fine-grained information regarding task affinities, this cost function requires computing and storing gradients throughout training.
4.2 Incorporating MTL Methods into Diffusion Model Training
After dividing the denoising tasks into task clusters via interval clustering, we apply multi-task learning methods to the resulting task clusters. As mentioned in Section 2, previous multi-task learning works have tracked down the following causes for negative transfer: (1) conflicting gradient, (2) difference in gradient magnitude, and (3) imbalanced loss scale. In this work, we leverage one representative method that tackles each of the causes mentioned above, namely, (1) PCgrad [83], (2) NashMTL [46], and (3) Uncertainty Weighting [29].
For each training step in diffusion modeling. we compute the noise prediction loss L' for the 1-th data within the minibatch. As shown in Eq 2, calculating L' involves sampling the timestep t!, in which case L' is a loss incurred on the denoising task D' We may then assign L' to the appropriate task cluster by considering the corresponding timestep. Subsequently, we may group up the losses as IL,.hs.,....s where L1, is the loss for the i-th task cluster. (More details in Appendix C)
1. PCgrad [83] In each iteration, PCgrad projects the gradient of a task onto the normal plane of the gradient of another task when there a conflict between their gradients. Specifically, PCgrad first calculates the per-interval gradient V. Then, if the other interval gradient VOL,, for i + has negative cosine similarity with VoLı,- projects VoLı, onto the normal plane of V PCgrad repeats this process with all of the other interval gradients for all interval gradients. resulting in a projected gradient per interval. Finally, model parameters are updated with the summation of projected gradients.
2. NashMTL [46] In NashMTL, the aggregation of per-task gradients is treated as a bargaining game. It aims to update model parameters with weighted summed gradients AO = Σi= a; VoLı, by obtaining the Nash bargaining solution to determine ag, where A0 is in the ball of radius centered zero, B., They define the utility function for each player as Uj = (Vo LI, A0), then the unique Nash bargaining solution can be obtained by arg maxAςEB, Σ, log(u,) u,). By denoting G as matrix whose columns contain the gradients Vol.,, Rt is the solution to GT 1/ox where 1/a is the element-wise reciprocal. To avoid the optimization to obtain a for each iteration, they update a once every few iterations.
3. Uncertainty Weighting (UW) [29] UW uses task-dependent (homoscedastic) uncertainty to weight task cluster losses. By utilizing observation noise parameter a; for i-th task clusters, the total loss function is Σ, /o² + log(o,). As the noise parameter for the i-th task clusters loss o; increases, the weight of L1, decreases, and vice versa. The o, is discouraged from increasing too much by regularizing with log(o,).
5 Experiments
In this section, we demonstrate the efficacy of our proposed method by addressing the negative transfer issue in diffusion training. First, we provide the comparative evaluation in Section 5.1, where

---

Table 1: Quantitative comparison vanilla training (Vanilla) on the unconditional generation. Integration of MTL methods using interval clustering consistently improves FID scores and generally
enhances precision compared vanilla training.
Dataset
Model
Clustering
Method
FPRQ [27]
CelebA-HQ 26]
FID (1)
Precision (7)
Recall (t)
FID (L)
Precision (f)
Recall (1)
Vanilla
24.95
0.5427
0.3996
22.27
0,5651
0.4328
PCgrad [83]
22.29
0,5566
0.4027
21.31
0,5610
0.4238
Timestep
NashMTL [46]
21.45
0,5510
0.4193
20.58
0.5724
0.4303
UW [29]
20.78
0.5995
0.3881
17.74
0.6323
0.4023
PCgrad [83]
20.60
0,5743
0.4026
20.47
0,5608
0.4298
ADM [8,
SNR
NashMTL [46]
23.09
0.5581
0.3971
20.11
0.5733
0.4388
UW
20.19
0.6297
0.3635
18.54
0.6060
0,4092
PCgrud [83]
23.07
0.5526
0,3962
20.43
0.5777
0,4348
Gradient
NashMTL [46]
22.36
0.5507
0.4126
21.18
0,5682
0.4369
[29]
21.38
0.5961
0.3685
18.23
0.6011
0,4130
Vanila
10.56
0.7198
0.4766
10.61
0,7049
0.4732
PCgrud [83]
9.599
0.7349
0,4845
9.817
0.7076
0.4951
Timestep
NashMTL [46]
9.400
0.7296
0,4877
9.247
0.7119
0.4945
UW [29]
9.386
0.7489
0,4811
9.220
0.7181
0.4939
PCgrad [83]
9.715
0.7262
0,4889
9.498
0.7071
0.5024
1.DM [56]
SNR
NashMTL [46]
10.33
0.7242
0,4710
9.429
0.7062
0.4883
UW [29]
9.734
0.7494
0.4797
9.030
0.7202
0.4938
PCgrad [83]
0.6954
0.4927
Gradient
NashMTL [46]
9.294
0,7234
0,4962
9.740
0.7051
0,5067
UW [29]
9.439
0.7499
0.4855
9.414
0.7199
0.4952
our method can boost the quality of generated samples significantly. Next, we compare previous loss weighting methods for diffusion models to UW with interval clustering in Section 5.2, verifying superior effectiveness to existing methods. Then, we analyze the behavior of adopted MTL methods, which serve to explain the effectiveness of our method in Section 5.3. Finally, we demonstrate that our method can be readily combined with more sophisticated training objectives to boost performance even further in Section 5.4. Extensive information all our experiments can be found in Appendix E.
5.1 Comparative Evaluation
Experimental Setup Here, we demonstrate that incorporating MTL methods into diffusion training improves the performance of diffusion models. For comparison, we consider unconditional and class-conditional image generation. For unconditional image generation, we used FFHQ [27] and CelebA-HQ [26] datasets, where all images were resized 256 x 256. For class-conditional image generation experiments, employed the ImageNet dataset [7], also resized to 256 x 256 resolution.
For architecture, we adopt widely recognized architectures for image generation. Specifically, use the lightweight ADM [6, 8] and LDM [56] for unconditional image generation, while employing DiT-S/2 [52] with classifier-free guidance [19] for class-conditional image generation. We train the model using our method: We consider every possible pair of (1) interval clustering (timestep-. SNR-, and gradient- based) and (2) MTL method (PCgrad, NashMTL, and Uncertainty Weighting (UW)), and report the results. We used k = 5 in interval clustering throughout experiments.
For evaluation metrics, recall [36] for assessing sample diversity and distribution coverage. IS [61] is additionally used for the evaluation metric in the class-conditional image generation setting. Finally, for sample generation, we use DDIM [67] sampler with 50 steps for unconditional generation and DDPM 250 steps for class conditional generation, and all evaluation metries are calculated using 10k generated samples.
FID [18] and precision [36] for measuring sample quality, and
Comparison Unconditional Generation As seen Table our method significantly improves performance upon conventionally trained diffusion models (denoted vanilla in the table). In particular, there is an improvement in FID all cases, and an improvement in precision scores in all but two cases, which highlights the efficacy of our method. Also, given strong results for both pixel- and latent-space models, we can reasonably infer that our method is generally applicable.
We also observe the distinct characteristics of each multi-task learning method considered. Uncertainty Weighting tends achieve higher improvements in sample quality compared to PCgrad and NashMTL. Indeed, UW achieves superior FID and Precision for ADM, while excelling in Precision for LDM.

---

0.60
83
60
0.55
70
Procision
Recall
40
0.48
30-
40
39
2.5 Guidance scale
20
3.0
15
300
1.
2.0
1.5
25 Guidance scale
2.0
3.0
Guidance scale
Guidance scale
Vanilla
UW-Grad
Nash-Time
Nash-5NR
PCgrad-Grad
UW Twme
UW SNR
Nash Grad
PCgradilime
PCgrad SNR
Figure 3: Quantitative comparison to vanilla training (Vanilla) ImageNet 256x 256 dataset with DiT-S/2 architecture and classifier-free guidance. Integration of MTL methods using interval clustering consistently improves FID, IS, and Precision compared to vanilla training.
Table 2:
Comparison between MinSNR and
Table 3: GPU memory usage and runtime com-
ANT-UW. DiT-1/2 is trained on ImageNet.
parison on FFHQ dataset LDM architecture.
Method
FID
IS
Precision
Recall
Method
GPU memory usage Iterations Sec
Vanilla 12.59
134.60
0.73
0.49
Vanilla
PCyral
28.160
1.523
MinSNR
9.58
179.98
0.78
0.47
NashMTT
38.914
2.011
ANT-UW
6.17
203.45
0,82
0.47
UW
34.350
2.103
However, UW sacrifices distribution coverage in exchange for sample quality, resulting in lower Recall compared to other methods. Meanwhile, NashMTII. scores higher in recall and lower in precision compared other methods, suggesting it has better distribution coverage while sacrificing sample quality. Finally, PCgrad tends to show a balanced performance terms of precision and recall. We further look into behaviors of different MTL methods in Section 5.3.
Due to space constraints, we provide a comprehensive collection of generated samples in Appendix In summary, diffusion models trained with method produce more realistic and high-fidelity images compared to conventionally trained diffusion models.
Comparison in Class-Conditional Generation We illustrate the results of quantitative comparison on caaascoondiiional generation in Fig. 3. The results show that our methods outperform vanilla training in FID, IS, and Precision. In particular, UW and Nash-MTL significantly boost these metrics, showing superior improvement in generation quality. These results further support the generalizability of MTL methods through the interval clustering class-conditional generation and the transformer-basee diffusion model.
5.2
Comparison Loss Weighting Methods
Since UW is a loss weighting method, validating the superiority of UW with interval clustering compared to previous loss weighting methods such as [6] and MinSNR [17] highlights the effectiveness of our method. We name UW by incorporating interval clustering as Addressing Negative Transfer (ANT)-UW. We trained D/T-L/2 with MinSNR and UW with
5 on the ImageNet across 400K iterations, using batch size of 240. All methods are trained by AdamW optimizer [43] with learning rate of le Table shows that ANT-UW dramatically outperforms MinSNR, emphasizing the effectiveness of our method. An essential note is that the computational cost of ANT-UW remains remarkably similar to vanilla training as shown Section 5.3, ensuring that enhanced performance does not come the expense of computational efficiency. Additionally, we refer to the results in [50]. showing that our ANT-UW outperforms P2 and MinSNR when DIT-L/2 trained on the FFHQ dataset.
5.3 Analysis
To provide a better understanding our method, we present various analysis results here. Specifically, we compare the memory and runtime of MTL methods, analyze the behavior of MTL methods adopted, provide a convergence analysis, and assess the extent to which negative transfer has been addressed.

---

Ig
fg
10
4
0.54
0.66
0.72
16
0.64
0.54
0,00
0.53
0.61
0.74
0.66
0.53
3.00
0.35
oogght
ooigle
0.80
0.72
0,61
0.35
0.00
Tamation
(a) Average conflict in PCgrad (b) Gradient weights of NashMTL
(c) Loss weight of UW
Figure 4: Behavior of multi-task learning methods across training iterations. (a): With increasing timestep difference, gradient conflicts between task clusters become more frequent in PCgrad. (b) and (c): Both UW and NashMTL allocate higher weights to task clusters that handle noisier inputs.
ADM
LDM
Vannila
50
16
UW-Time
UW-Gra
UW-SNR
FID
Nash-Time
Nash-
30
Nash- SNR
PCgrad
10
PCprad-Grad
20
PCgrad-SNR
0.6
0.8
Iterations
X10'
Iterations
x10*
Figure 5: Convergence analysis on FFHQ dataset. Compared to baselines, all methods exhibit fast convergence and achieve good final performance.
Memory and Runtime Comparison We first compared the memory usage and runtime between MTL methods and vanilla training for deeper understanding of their cost. We conducted measure- ments memory usage and runtime with and timestep-basee bbased clustering, and the results are shown in Table PCgrad has slower speed of 1.523 iterations/second compared to vanilla training, but its GPU memory usage is lower due to the partitioning of minibatch samples. Meanwhile, NashMTL has runtime of 2.011 iterations/second. Even though NashMTL uses more GPU memory, it has a better runtime than PCgrad because it com- putes per-interval gradients occasionally. Concurrently, UW shows similar runtime and GPU memory usage as vanilla training, which attributed to its use of weighted loss and a single backpropagation process.
on the FFHQ dataset using the LDM architecture
Behavior of MTL Methods We analyze the behavior of different multi-task learning methods during training. For PCgrad, calculate the average number of gradient conflicts between task clusters per iteration. For UW, we visualize the weights allocated to the task cluster losses over training iterations. Finally, for NashMTL, gradients over training iterations. We used LDM trained on FFHQ for our experiments. Although only report results for time-based interval clustering for conciseness, note that MTL methods exhibit similar behavior across different clustering methods. Results obtained using other clustering methods can be found in Appendix D.
visualize the weights allocated to per-task-cluster
The resulting visualizations are provided in Fig. 4. As depicted in Fig. 4a, the task pair that shows the most gradient conflicts is / and 15, namely, task clusters apart in timesteps. This result supports our hypothesis that temporally distant denoising tasks may be conflicting, and as seen Section 1. PCgrad seems to mitigate this issue. Also, depicted in Fig. 4b and 4b, both UW and NashMTL tend to allocate higher weights to task clusters that handle noisier inputs, namely, I4, J5. This result suggests that handling noisier inputs may be difficult task that is underrepresented in conventional diffusion training.
Faster Convergence In Fig. 5, we plot the trajectory of the FID score over training iterations, as observed while training on FFHQ. We can observe that all our methods enjoy faster convergence and better final performance compared to the conventionally trained model. Notably, for pixel space

---

Vanill
PCgrad PCandd SNR PCgrud-Gras
Nash Time Nasb-SNI Nasb-Graa
UW Time
UW-SNR
ADM WIG (IID)
6.24
2.70
[002
10 400]
[401, 600]
[,0 800]
[801, 1000]
LDM LDM ATG (FID)
0.60
0.22
-0.15
0.53
[1. 200]
120 400]
[401, 600]
[601, 800]
[801. 1000]
Figure 6: Negative transfer gap (NTG) comparison on the FFHQ dataset. Integration of MTL methods tends improve the negative transfer gap. Methods that fail to improve NTG areas where the baseline records low NTG tend to achieve lesser improvements in the baseline.
diffusion (ADM), UW converges extremely rapidly, while beating the vanilla method by large margin. Overall, these results show that our method may not only make diffusion training more effective but also more efficient.
Reduced Negative Transfer Gap We now demonstrate that proposed method indeed mitigates the negative transfer gap we observed in Section 3.2. We used the same procedure introduced in Section 3.2 to calculate the negative transfer gap for all methods considered, for the FFHQ dataset.
As shown Fig. 6 our methods improve upon negative transfer gaps. Specifically, for tasks that exhibit severe negative transfer gaps in the bascline (e.g., [601, 800], [801, 1000] for ADM, and [401, 600), [601, 800] for LDM), our methods mitigate the negative transfer gap for most cases, even practically removing it in certain reducing negative transfer (NashMTL-SNE for LDM and PCgrad-Grrs -rad for ADM) indeed show worse FID scores, which supports hypothesis that resolving negative transfer leads performance gain. We also note that even the worst-performing methods still beat the vanilla model.
Another interesting result note is that models less effective
5.4 Combining MTL Methods with Sophisticated Training Objectives
Finally, we show that our method is readily applicable on top more sophisticated training objectives proposed in the literature. Specifically, we train an LDM by applying both UW and PCgrad on top of the P2 objective [6] and evaluate the performance on the FFHQ dataset. We chose UW and PCgrad based on a previous finding that combining the two methods leads to performance gain [41]. Also, chose the gradient- based clustering method due to its effectiveness for LDM on FFHQ. As seen in Table 4, when combined with P2, our method improves the FID from
Table 4: Combining our method
with P2 on the FFHQ dataset. DDIM 200-step sampler is used.
Type
Method
FID-50K
GAN
PGin [63]
3.39
AR
VQGAN (1I]
9.6
D2C [65]
13.04
Diffusion (LDM)
Vanilla
9.1
7.21 to 5.84.
P2
P2 Ours
5.84
6 Conclusion
In this work, we studied the problem of better training diffusion models, with the distinction of reducing negative transfer between denoising tasks in a multi-task learning perspective. Our key contribution is to enable the application of existing multi-task learning techniques, such as PCgrad and NashMTL, that were challenging to implement due to the increasing computation costs associated with the number of tasks, by clustering the denoising tasks based on their various task affinity scores. Our experiments validated that the proposed method effectively mitigated negative transfer and improved image generation quality. Overall, our findings contribute to advancing diffusion models. Starting from our work, we believe that addressing and overcoming negative transfer can be the future direction to improve diffusion models.
10

---

References
[1] Yogesh Balaji, Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. edifh: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.0132 2022. 1,
[2] Richard Bellman. A note cluster analysis and dynamic programming. Mathematical Biosciences, 18 (3-4)311-3112, 1973. 2, 5,
[3] Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslie Rice, Mingjie Sun, and Zico Kolter. (certified!!) adversarial robustness for free! arXiv preprint arXiv:2206. 10550. 2022. 2
[4] Zhao Chen, Vijay Badrinarayanan, Chen-Yo Lee, and Andrew Rabinovich. Gradnorm: Gradient normal- ization for adaptive loss balancing in deep multitask networks. In International conference machine learning, pages 794-803. PMLR, 2018. 3
[5] Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguclov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039- 2-520000000, 2020. 2, 3
[6] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11472-114811 2022. 4,
10, 17.21
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009. 2, 21
[8] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021. 1, 2, 7, 17,
[9] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 14Xiv:/410.85//1 2014.
[10] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.000 088003, 2016. 2
[11] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high resolutcoottoon image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021. 10
[12] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503-27516,
2021. 2, 4, 20
[13] Hyojun Go. Yunsung Lee, Jin- Young Kim, Seunghyun Lee, Myeongho Jeong, Hyun Seung Lee, and Seungtaek Choi. Towards practical plug-and-play diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1962-1971, 2023. 2, 17
[14] Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
[15] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and and- play priors. Advances in Neural Information Processing Systems, 35:14715-14728, 2022. 2
[16] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning, In Proceedings of the European conference on computer vision (ECCV), pages +70-287 2018. 2,3
[17] Tiankai Hang. Shuyang Gu, Chen Li, Jianmin Bao. Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEL/CVF International Conference on Computer Vision (ICCV), pages 7441-7451, October 2023. 2, 3, 8
[18] Martin Hcusel, Hubert Ramsauer, Thomas Unterthiner, Berhhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 5, 7, 17, 11
[19] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 21
11

---

[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020. 1,
[21] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22
2022. 1
[22] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1-33, 2022. 1
[23] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, David Fleet. Video diffusion models. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 1,2
[24] Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask lcarning. In Interna- tional Conference on Learning Representations, 2022. 2, 3,
[25] Junguang Jiang. Baixu Chen, Junwei Pan, Ximei Wang. Liu Dapeng, Jie Jiang, and Mingsheng Long. Forkmerge: Overcoming negative transfer in multi-task learning. arXiv preprint arXiv:2301.12612 2023.
2, 3, 3,4,111 17
[26] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. 2. 7, 21
[27] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 4410, 2019. 2, 4, 7, 17, 20,
[28] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Flucidating the design space of diffusion- based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 1,2
[29] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018. 2, 3, 6, 7, 19, 20,
[30] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2426-2435, 2022. 2
[31] Diederik Kingma. Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021. 2,
[32] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013. 2
[33] Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicir Likelihood Models, 2021. 2
[34] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. 3
[35] Vitaly Kurin, Alessandro De Palma, llya Kostrikov, Shimon Whiteson, and Pawan K Mudigonda. In defense of the unitary scalarization for deep multi-task learning. Advances in Neural Information Processing Systems, 35:12169-12183, 2022. 18, 20
[36] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019. 7,
[37] Yunsung Lee, Jin- Young Kim, Hyojun Go, Myeongho Jeong, Shinhycok Oh, and Seungtack Choi. Multi- architecture multi-expert diffusion models. arXiv preprint arXiv:2306.0443 2023. 17
[38] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328 4343, 2022. 2.2
12

---

[39] Baijiong Lin and Yu Zhang. Libmtl: A python library for deep multi-task learning. Journul of Muchine Learning Rexearch, 24:1-7, 2023. 18
[40] Baijiong Lin, Feiyang YE, and Yu Zhang. A closer look at loss weighting in multi-task learning, 2022.
URI. https:://opereeeeee
18, 20
[41] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conffict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:1 8878-18890, 2021. 10
[42] Liyang Liu, Li, Zhanghui Kuang, J Xue. Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. Towards impartial multi task learning. International Conference on Learning Representations, 2021. 2,
[43] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentatinss 2019. URL
wwt. htt://oommmiie http//ooruiidd-Bkgeeee BigeeiiiidBBidigpir CqY7. 8.21
[44] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021. 1,2
[45] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. 2
[46] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as bargaining game. In Proceedings the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16428-16446. PMLR, 17-23 Jul 2022. URL https: ststpp://ppooeeddingg mll. www...reeeevivvi www.//////vvnaaaaaaaaaa htmm.. 2, 2,3, 6, 18, 21
[47] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162 -171- PMLR, 2021. 1,
[48] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew. Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804. PMLR, 2022. 1,2
[49] Frank Nielsen and Richard Nock. Optimal interval elustering: Application to bregman clustering and statistical mixture learning. IEEE Signal Processing Letters, 21(10):1289 1292, 2014. 2, 5, 22
[50] Byeongjun Park, Sangmin Woo, Hyojun Go, Yin-Young Kim, and Changick Kim. Denoising task routing for diffusion models. arXiv preprint arXiv:2310.07118, 2023.
[51] Gaurav Parmar, Richard Zhang, and Yun Zhu, On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 11110-11420 2022. 18, 29
[52] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023. 2, 21
[53] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.0 200000997./ 2023. 21
[54] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference Learning Representations, 2022. 1.2
[55] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar arXiv:2204.061 2022. 1
[56] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Partern Recognition, pages 10684-10695, 2022. 2, 3, 4, 5, 7, 17, 20, 21
[57] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint
2017. 2,3,4
[58] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- to image diffusion models with deep language understanding. Advances in Neural Information Processing Systems,
2022. 2
13

---

[59] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 3
[60] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations. 2021. 2
[61] Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016."
[62] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2016.
[63] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 74400---1749,, 2021. 10
[64] Ozan Sener and Vladlen Koltun. Multi-task learning as nulti-objecttie optimization. Advances in neural information processing systems, 31, 2018. 2,3
[65] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 14:12533-12548, 2021. 10
[66] Jascha Sohi-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages PMLR, 2015. 2,
[67] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020. 2, 7, 18, 19
[68] Xiaozhuang Song. Shun Zheng, Wei Cao, James Yu, and Jiang Bian. Efficient and effective multi-task grouping via meta learning on task combinations. In Advances in Neural Information Processing Systems, 2022. 20
[69] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[70] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415- -428, 2021. 2,3
[71] Yang Song, Jascha Sohl- Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. 1, 2,
[72] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should he learned together in multi-task learning? In International Conference on Machine Learning, pages 9120-9132, PMLR, 2020. 4, 20
[73] Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Patel, Dengxin Dai, and Luc Van Gool. Task switching network for multi-lask learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8291-8300 2021. 24
[74] Arash Vahdat, Karsten Kreis, and Jan Kautz. based generative modeling in latent space. Advances in Neural Information Processing Systems, 1132- 2021. 2,
[75] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016. 2
[76] Haizhou Wang and Mingzhou Song. Ckmeans. 1d. dp: optimal k-means clustering in one dimension by dynamic programming. The R journal, 3(2):29, 2011.. 2, 22
[77] Zirui Wang, Zihang Dai, Barnabás Póczos. and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1130211302, 2019. 17
[78] Zirui Wang. Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Conference on Learning Representations, 2021. 2,
14

---

[79] Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In International Conference Learning Representations, 2019. 17
[80] Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg. and Orhan Firat. Do current multi-task optimization methods in deep learning help? Advances in Neural Information Processing Systems. 35:13597-13609 2022. 18, 20
[81] Yilun Xu, Shangyuan Tong, and Tommi S. Jaakkola. Stable target field for reduced variance estimation in diffusion models. Eleventh International Conference on Learning Representations, 2023. 4
[82] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22552-
22562, 2023. 2
[83] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, Advances in Neural Information Processing Systems, 33:5824-500 5836, 2020. 2, 3, 4, 6, 19, 20, 21
15

---

Appendix
Contents
Relation to Negative Transfer Gap Previous Literature
17
Detailed Experimental Settings for Observational Study
17
Implementation Details for MTL methods
18
D Additional Experimental Results
18
D.I Visualization for the Behavior of MTL Methods with Other Clustering Methods
19
D.2 Analysis: The Number of Interval Clusters
19
D.3 Comparison Interval Clustering with Task Grouping Method
20
D.4 Comparison Random Loss Weighting and Linear Scalarization
20
Detailed Experimental Settings in Section 5
E.1
Detailed Settings of Comparative Evaluation and Analysis (Section 5.1 and 5.3)
21
E.2
Detailed Settings of Comparison Loss Weighting Methods (Section 5.2)
21
E.3
Detailed Settings of Combining MTL Methods with Sophisticated Training Objec- tives (Section 5.4)
21
Qualitative Results
G Dynamic Programming Algorithm for Interval Clustering
22
Broader Impacts
23
I Limitations
24
16

---

A Relation to Negative Transfer Gap in Previous Literature
Previous works on transfer and multi-task learning have explored measuring the negative transfer [25, 79, 77]. For the source task Tore and the target task Tigt. the negative transfer can be defined as the phenomenon that the source task negatively transfer the target task. Denote the model trained on both source and target task as e (Tigt, Tare) and the model only trained on the target task as O ( With performance measure for the model on Tegt. negative transfer can be quantified by utilizing negative transfer gap (NTG):
(Tgt, TTrc) = P(O(TTg)) - P( Tsre))-
(5)
For P. higher is better, NTG > indicates that negative transfer occurs, showing that additionally training on Tare negatively affects the learning of Tigt.
In our study of negative transfer in diffusion models, the target task involves denoising tasks within specific timestep interval as Tigt Dit,,al,, while the source task comprises the remaining denoising tasks as Tare- D(1.7] Dlt..ta).
However, since model trained only a subset of entire denoising tasks cannot generate samples properly, we cannot utilize the sample quality metrics (e.g. FID [18]) for P to measure ((O(T))t) )) Eq. 5 for arbitrary timestep intervals. This is a different point from typical MTI. setting, where the performance of each task can be measured.
Alternatively, we redefine NTG with the difference in sample quality resulting from denoising by different models, (Tige) and @ Tare), in the [t t2] interval. During the sampling procedure with a model trained on entire denoising tasks, we use (TTig) or ( Tare) in [tz, t2). Denote
the resulting samples with (Tett, Tsre) as ) and the resulting samples with (Tige) as (=
Then, by comparing the quality of these samples as Eq. 3, we can measure how much the denoising of [2] degrades in terms of sampling quality.
Furthermore, the success of multi-expert denoisers in prior studies [37, 13, 1] suggests the potential existence of negative transfer. By distinctly separating parameters for denoising tasks, they might mitigate this negative transfer. leading enhanced performance in their generation.
B Detailed Experimental Settings for Observational Study
In this section, we provide the details experimental settings in Section 3. The training details and the architectures used are the same as those in Section All experiments are conducted with a single A100 GPU and with FFHQ dataset [27].
For the pixel-space diffusion model, we use the lightweight ADM as same in [6]. inherits the architecture of ADM [8], but it uses fewer base channels, fewer residual blocks, and a self-attention with single resolution. Specifically, the model uses one residual block per resolution with 128 base channels and 16x 16 self-attention with 64 head channels. A linear schedule with T = 1000 is used for diffusion scheduling. We referenced the training scripts in the official code2 for implementation.
For the latent-space diffusion model, we use the LDM architecture as the same settings for FFHQ experiments in [56]. Specifically, an LDM-4-Q encoder and decoder are used, in which the resolution of latent vectors is reduced by four times compared to the original images and has a vector quantization layer with 8092 codewords. The denoising model has 224 base channels with multipliers for each resolution as 1, 2, 3, 4 and has two residual blocks per resolution. Self-attention with 32 head channels is used for 32, 16, and 8 resolutions. For diffusion scheduling, the linear schedule with T 1000 is used. We conducted experiments with the official code". In general, we utilized the pre-trained weights provided by LDM. However, if our retraining results demonstrated superior performance, we reported them.
Task Affinity Analysis To measure the task affinity score between denoising tasks, we first calculate VoL, for t = I,..., T every 10K iterations during training. The gradient is calculated with 1000 samples in the training dataset. Then, the pairwise cosine similarity of the gradient computed and
Phttps:// http://thth....coo/o///atattaaaa
17

---

weglt
4-
107
4
42
9.36
107
4-
8.65
0.52
4
5
4.
Beration
(a) Average conflict in PCgrad (b) Gradient weights of NashMTL
(c) Loss weight of UW
Figure 7: Behavior of multi-task learning methods through SNR-based interval clustering across training iterations. A similar trend in Fig. 3 is observed.
their cosine similarities calculated by every 10K iterations are averaged. Finally, we can plot the average cosine similarity against the timestep axis as in Fig. 1. For plotting them against the log-SNR axis, the values of the axis were adjusted, and the empty parts were filled with linear interpolation.
For ADM and LDM, the pairwise cosine similarity between gradients is calculated during 1M training iterations and 400K training iterations, respectively.
Negative Transfer Analysis To calculate the negative transfer gap in Eq. 3, need additionally train the model on denoising tasks within specific timestep interval [+1. tg]. Since we plot five intervals [1, 200]. [201, 400], [401, 600], [601, 800], and [801, 1000]. trained the model denoising tasks for each interval. Each model is trained for 600K iterations in ADM and 300K iterations in LDM on the FFHQ dataset. For the model trained
entire denoising tasks, used the trained model the same as Section 5.1. ADM is trained on 1M iterations and LDM is trained on 400K iterations. All of these models trained with the same batch size and learning rate as experiments in Section 5.1 (See Appendix E).
DDIM 50-step sampler [67] was used for the generation. FID is calculated with Clean-FID [51] by setting the entire 70K FFHQ dataset as reference images. Since the official code of Clean-FID* supports FID calculation with statistics from these reference images, used it and reported FID with 10k generated images.
C Implementation Details for MTL methods
We describe how MTL methods are applied in Section 4.2. To be more self-contained, we hereby present implementation details for MTL methods. For the implementation of MTL methods, we used the official code of LibMTL [39] NashMTL [46] supports practical speed-up by updating gradient weights every few iterations, not every iteration. We utilize this by updating every 25 training iterations.
D Additional Experimental Results
We present additional experimental results to supplement the empirical findings presented Section 5. In Section D.I, we provide visualizations of the behavior of MTL methods with other clustering meth- ods that were not covered in Section 5.3. Furthermore, we examine the impact of our hyperparameter, the number of clusters in Section D.2. To validate the effectiveness of interval clustering compared to other clustering methods, we present additional results in Section D.3. In Section D.4, we delve deeper into comparing the performance of stronger MTL baselines such Linear Scalarization (LS) [80, 35] and Random Loss Weighting (RLW) [40] with our proposed approach.
18

---

weggt.
0.72
6.79
0.29
6:00
0.49
0.54
6.73
16
0:57
6:49
0.00
0.55
6:64
107
0:22
0.35
0:00
644
6.00
Beration
(a) Average conflict in PCgrad (b) Gradient weights of NashMTL
(c) Loss weight of UW
Figure 8: Behavior of multi-task learning methods through gradient-based interval clustering across training iterations. A similar trend in Fig. 3 is observed.
Table 5: FID-10K scores of the LDM trained using a combination UW and PCgrad methods on the FFHQ dataset while varying the value of k. Notably, integrating MTL methods with two clusters significantly improves FID scores. Increasing k from 2 to 5 also enhances FID scores, but further increasing from 5 to shows similar results.
Number of clusters (k)
Clustering
Vanilla
k ==2
==5
==
Timestep SNR
9.563
9.151
9.083
10.56
9.606
9.410
9.367
Gradient
9.634
9.033
9.145
D.1 Visualization for the Behavior of MTL Methods with Other Clustering Methods
Due to space constraints in our main paper, we were unable to include the behavior analysis of MTL methods for SNR-based and gradient-based interval clustering. However, we present these results in Fig. 7 and 8, which show similar trends to the observations depicted in Fig. 4. These findings suggest valuable insights into the behavior of MTL methods, regardless of the clustering objectives.
Firstly, we observed a notable increase in the occurrence of conflicting gradients as the timestep difference between tasks increased. This observation suggests that the temporal distance between denoising tasks plays crucial role in determining the frequency of conflicting gradients.
Secondly, noted that both loss and gradient balancing methods assign higher weights to task clusters with higher levels of noise. This finding indicates that these methods allocate more importance to the noisier tasks.
D.2 Analysis: The Number of Interval Clusters
To understand the impacts of the number of clusters k we conducted experiments by varying k with 2, and 8. We trained model for timestep-based, SNR-based, and gradient-based clustering with each k, resulting in nine trained models. For MTI. methods, we used combined methods with UW [29] and PCgrad [83] as in Scction 5.4. All training configurations such as Icarning rate and training iterations are the same in Section 5.1. We evaluate 10K generated samples from the DDIM 50-step sampler [67] for all methods with the FID score [51, 18].
Table 5 shows the results. Notably, we made an intriguing observation regarding the integration of MTL methods with only two clusters, which resulted in a noteworthy enhancement in FID scores. Additionally, we found that increasing the number of clusters, denoted as k. from 2 to 5 also exhibited a positive impact on improving FID scores. However, our findings indicated that further increasing k from 5 to 8 did not yield significant improvements and resulted in similar outcomes. From these results, we conjecture that increasing the number of clusters to greater than five has no significant effect.
19

---

Table The results of Random Loss Weighting (RLW) and Linear Scalarization (LS) on the FFHQ dataset in ADM architecture.
Clustering
Method
FID
Precision
Recall
Vanilla
24.95
0.5427
0.3996
RLW
38.06
0,4634
0.3293
Timestep
LS
25.34
0.5443
0.3868
RLW
35.13
0.4675
0.3404
SNR
LS
25.69
0.5369
0.3843
RLW
36.19
0.4643
0.3392
Gradient
LS
26.12
0.5120
0.3878
D.3 Comparison Interval Clustering with Task Grouping Method
To show the effectiveness of interval clustering methods for denoising task grouping in diffusion models, we compare high-order approximation (HOA)-based grouping methods [72, 12].
For grouping N-tasks in deep neural networks, the early attempt [72] established two-stage procedure: (1) compute MTL performance gain for all task combinations and (2) search best groups for maximizing MTL performance gain across the groups. However, performing (1) requires huge computation since MTL performance gain should measured for all 2 combinations. Therefore, they reduce computation by HOA, which utilizes MTL gains on only pairwise task combinations. Also, the HOA scheme is inherited by the following work, task affinity grouping [12]. which uses their defined task affinity score instead of MTL gains. Different from these works, our interval clustering aims to group the tasks with interval constraints.
For a fair comparison, we use pairwise gradient similarity averaged across training iterations between denoising tasks for the objective of
Table 6: Comparison inter- val clustering and high or- der approximation-base based task grouping. DDIM-50 step sam- pler is used.
HOA-based grouping and interval clustering. In this case, the HOA- based grouping becomes cosine similarity grouping used in [12], and interval clustering becomes gradient-based clustering in our method. However, for HOA-based grouping. a solution of brute force searching with branch-and rodd-boun-liiee algorithm [72, 12] requires computational complexity of O(2 It incurs enormous costs diffusion with many denoising tasks. Therefore, we search scheme [68]. We set the number of clusters as 5 for both methods.
Clustering
FID-10k
HOA Interval
9.873
beam-
9.033
We apply the combined method with UW [29] and PCgrad [83] as Section 5.3 for the resulting clusters from both HOA-based grouping and interval clustering. We trained the model on the FFHQ dataset [27] and used LDM architecture [56]. All training configurations are the same as in Section 1. For evaluation metrics, we use FID and its configurations are the same as in Section 5.1.
Table shows the results, indicating that the interval clustering outperforms HOA-based task grouping.
D.4 Comparison to Random Loss Weighting and Linear Scalarization
Linear Scalarization (LS) [80, 35] and Random Loss Weighting (RLW) [40] can serve as strong baselines for MTL methods. Therefore, validating the superiority of our method compared to theirs emphasize the necessity of applying sophisticated MTL methods such UW, PCgrad, and NashMTL. Accordingly, we provide the results of comparative experiments for LS and RLW on the FFHQ dataset using ADM architecture in Table 7. We note that all experimental configuration is the same as in vanilla training in Section 5.1.
As shown in the results, LS achieves slightly worse performance than vanilla training, which suggests that simply re-framing the diffusion training task as an MTL task and applying LS not enough. Also, RLW achieves much worse performance compared to vanilla training. It appears that the randomness introduced by loss weighting interferes with diffusion training. These results indicate that sophisticated MTL methods are indeed responsible for significant performance gain.
20

---

E Detailed Experimental Settings in Section 5
In this section, we describe the details of experimental settings in Section 5. For validating the effectiveness in both pixel-space and latent-space diffusion models in unconditional generation, we used ADM [8] and LDM [56] as same in our observational study (refer to details of architecture in Appendix B).
E.1 Detailed Settings of Comparative Evaluation and Analysis (Section 5.1 and 5.3)
single A100 GPU is used for experiments in Section 5.1 and 5.3.
Setups for Unconditional Generation We trained the models on FFHQ [27] and CelebA-HQ [26] datasets. All training was performed with AdamW optimizer [43] with the learning rate le-4 or 2e-5, and better results were reported. For ADM, we trained 1M iteration with batch size 8 for the FFHQ dataset and trained 400K iterations with batch size 16 for the CelebA-HQ dataset. For LDM, we trained 400K iterations with batch size 30 for both FFHQ and CelebA-Ho -HQ datasets. We generate 10K samples with DDIM-50 step sampler and measure FID [18], Precision [36], and Recall [36] scores. For all evaluation metrics, we use all training data as reference data. FID is calculated with clean-FID [51], and Precision and Recall are computed with publicly available code 6, All analyses are conducted above trained models.
Setups for Class-Conditional Generation We trained the DIT-S/2 [52] on ImageNet dataset [7]. All training was performed with the Adam W optimizer [43] with the learning rate of 1e-4 or 2e 5. and better results were reported. As in DiT [52], we applied the classifier-free guidance [19] and trained 800K iterations with batch size of 50. All samples are generated by a DDPM 250-step
sampler. For evaluation metrics, we follow the evaluation protocol in ADM [8]. by using their evaluation code?. We used the cosine schedule [47] for noise scheduling and SD-XL VAE [53] for our VAE.
E,2 Detailed Settings of Comparison to Loss Weighting Methods (Section 5.2)
We trained the DiT-L/2 [52] on ImageNet dataset [7]. All training was performed with the Adam optimizer [43] with the learning rate of 1e-4. As in DiT [52], we applied the classifier-free guid- ance [19] and trained 400K iterations with batch size of 256. All samples are generated by DDPM 250-step sampler and classifier dlasiifer-guiddace scale of 1.5. We used the cosine schedule [47] for noise scheduling. For experiments, we used 8 A100 GPUs
E.3 Detailed Settings of Combining MTL Methods with Sophisticated Training Objectives (Section 5.4)
We trained three different models: vanilla LDM, vanilla LDM with P2 [6]. and vanilla LDM with P2. PCgrad [83], and UW [29] applied simultaneously. All training configurations are the same in Section 5.1 but we use 500K iterations. We generate 50K samples for evaluation with a DDIM 200-step sampler and evaluate FID.
F Qualitative Results
In this section, we provide qualitative comparison results, which were omitted from the main paper due to space constraints. In Figure 9, 10, and 12, we visualize the generated images by all models that are used for results in Table As shown in the results, MTL methods for diffusion training can improve the quality of generated images. One noteworthy observation is that UW [29] tends to generate higher-qualiy, images compared to NashMTL [46] and PCGrad [83]. This finding aligns with the results observed in Table 1.
observe that incorporating
Moreover, we plot the randomly selected samples from 50K generated data in Fig. 13. Despite being randomly selected, the majority the generated images exhibit remarkable fidelity.
hpt.ppp////ttaaoooo////onnnuuuog htgp:///hw....ooooooo.o.oooooooa lhttps://githul /t/p//////////////////////////// http:///wo.aaaaaaaaaaaaaaatatt/a
21

---

(a) Vanila
(b) PCgrad -Timestep
(c) PCgrad -SNR
(d) PCgrad -Gradient
(c) Nash
(i) Nash -SNR
(g) Nash -Gadient
(h) U/W Timestep
(i) UW -SNR
( UW Gradient
-Timestep
Figure Qualitative comparison of ADM trained on the FFHQ dataset.
(a) Vanila
PCgrad -imestep
(e) PCgrad SNR
(d) PCgrad GGaaieent
Nash Timessepp
(f) Nash -SNR
(g) Nash
(h) UW Timestep
UW -SNR
( UW GGadient
-Gradient
Figure 10: Qualitative comparison of LDM trained on the FFHQ dataset.
G Dynamic Programming Algorithm for Interval Clustering
In this section, we introduce the algorithm for optimizing the interval cluster and the implementation details. The optimal solution of interval clustering can be found using dynamic programming for a Latuster function [2, 49, 76]. The sub-problem
then defined as finding the minimum cost of clustering X2 (1. i)) into clusters. By saving the minimum cost of clustering Xi.; = (1,..., i} into m clusters to the matrix D(i, m)]. the value in D k] represents the minimum clustering costs for the original problem in Eq. 4. For some timestep S j S i, Díj 1, 1] must contain the minimum costs for clustering Xi,j-1 into (m- - 1) clusters [49, 76]. This establishes the optimal substructure for dynamic programming, which leads to the recurrence equation follows:
.)
Díi m]
min (Dij 1, 1] + Latuster (Rg,)); 1SiST, 1 < k.
(6)
mijSi
To obtain the optimal intervals 4...., L. we use Sli, m] to record the argmin solution of Eq. 6. Then, we backtrack the solution (k) time from S(T, k] by assigning 4m = S(1m+1 1, m] from m to m 1 by initializing 4 k).
Interval clustering with based or gradient-baae based objectives can produce unbalanced sizes of each interval, which causes unbalanced allocation of task clusters due to randomly sampled timestep Therefore, we add constraints on the size of each cluster to avoid seriously unbalanced task clusters. To add constraints on the size of each cluster n, = |7,
1,..., k. we define the lower and upper bounds of as m, and M, with mi, < Ri S M,. In Eq.. 6, the mith cluster (i.e.,
- 4 - 1 for
&
x².)) size R'm must range from my to M,, yielding i + 1 M1 j < <itl my. Furthermore, to
22

---

(a) Vanila
(b) PCgrad
(c) PCgrad SNR
(d) PCgrad Gradient
(e) Nash
(i) Nash -NR
(g) Nash Gradieet
(h) U' Timestep
(i) UW SNR
( UW Gradient
Timestep
Uniffrrm
Figure 11: Qualitative comparison of ADM trained on the CelebA-HQ dataset.
(a) Vanila
(b) PCgrad
(c) PCarad
(d) PCgrad Gradient
Nash Tinestep
(i) Nash SNR
(g) Nash Gradient
(h) UW Timestep
(i) UW SNR
UW Grudient
Tamestep
SNR
Figure 12: Qualitative comparison of LDM trained on the CelebA-HQ dataset.
satisfy the (m cl-csuss constraint, + (m 1)m; S.j Finally, Eq. 6 with constraints on the size of the cluster is derived follows:
D(i, max (1 (mm-mmmm)) 1)mm 11-
min
[D[j m - 1]+ Tnluster
s m S k.
jsit1-m;
Specifically, we assign
and
37 26
to my and M1, respectively.
2h
H Broader Impacts
Revisiting Diffusion Models through Multi-Task Learning Our work revisits diffusion model training from a Multi-Tast Learning aspect. We show that negative transfer still occurs in diffusion models and addressing it with MTL methods improve the diffusion models. Starting from our work, a better understanding of the multi-task learning characteristics in diffusion models can lead to further advancements in diffusion models.
Negative Societal Impacts Generative models, including diffusion models, have the potential to impact privacy in various ways. For instance, in the context of DeepFake applications, where generative models are used to create realistic synthetic media, the training data plays a critical role in shaping the model's behavior.
When the training data is biased or contains problematic content, the generative model can inherit these biases and potentially generate harmful misleading outputs. This highlights the importance
23

---

Figure 13: Randomly selected images from generated images of LDM with combined methods of UW, PCgrad, and P2 on the FFHQ dataset. DDIM 250-step sampler is used.
of carefully selecting and curating the training data for generative models, particularly when privacy and ethical considerations at stake.
I Limitations
Our work has two limitations that can be regarded as future works. Firstly, we have not yet completely resolved the issue of negative transfer in the training of diffusion models shown in Fig. 5. This indicates that learning entire denoising tasks still causes degradation in certain denoising tasks. By successfully addressing this degradation and enabling the model harmoniously learn entire denoising tasks, we anticipate significant improvements in the performance of the diffusion model.
Secondly, study does not delve into the architectural design aspects of multi-task learning methods. While our focus lies on model-agnostic approaches in MTL, it worthwhile to explore the possibilities of designing appropriate architectures within MTL framework. Previous works in diffusion models utilize timestep and noise level as input, which can be considered as using task embeddings scheme [73]. By revisiting these aspects, the architecture of the diffusion model can be further advanced in future works.


Reviews:
[{"review_id": "QF9y9qJeAH", "reviewer": "Reviewer_7mPc", "summary": "This work explores the phenomenon of negative transfer in the diffusion training procedure, where different time steps or signal-to-noise ratios may conflict with each other and degrade overall performance. The authors propose a solution to this problem by introducing internal clustering and implementing several multi-task learning (MTL) methods to mitigate the negative transfer effect. The proposed approach is evaluated through experiments on the FFHQ and CelebA datasets, and the results demonstrate its effectiveness in improving performance compared to existing methods.", "questions": "1. It would be beneficial to include the average conflict of the baseline model in Fig 3(a) to provide a clearer comparison.\n2. The choice of the number of clusters is an important parameter in the proposed internal clustering approach, and it would be beneficial to conduct further analysis and ablation studies to determine the optimal number of clusters.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. It is interesting and novel to approach the training of diffusion models from the perspective of multi-task learning (MTL). By recognizing the potential for negative transfer in the diffusion training procedure, the authors of this work are able to leverage the benefits of MTL to mitigate the negative effects of conflicting time steps or signal-to-noise ratios. \n2. The motivation sounds good and the preliminary experiment can demonstrate the negative transfer phenomenon in diffusion models.\n3. After addressing negative transfer through internal clustering and MTL methods, the final model achieve good performance on FFHQ and CelebA.\n4. The writing is good and easy to follow.\n", "weaknesses": "My main concern is that the experiments are only conducted on small datasets like FFHQ and CelebA. Now many standard benchmarks exist. e.g., ImageNet-64, ImageNet-256, CoCo. Experiments on more datasets are needed to prove the effectiveness of the proposed methods.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer 7mPc for providing constructive comments, which are very helpful in improving our work through experimental results. We will address all raised concerns by the reviewer and revise the paper accordingly. \n\n---\n\n### **W1: Experiments are only conducted on small datasets.**\n\nThank you for your valuable suggestion. We agree with your point that our method must be validated on a large-scale dataset. Accordingly, we conducted experiments on ImageNet 256 x 256 for the conditional generation task using the DiT-S model. We opted for a smaller architecture of DiT-S due to resource constraints.\n\nWe report evaluation metrics for generated samples under varying guidance scales in **Fig. A of the Global Response**. As shown in Fig. A, our method consistently improves upon vanilla training in terms of FID, IS, and Precision scores by a large margin.  Specifically, balancing methods such as UW and NashMTL are shown to be very effective under strong guidance. We also provide convergence analysis in **Fig. B of the Global Response**. As shown in Fig. B, we see that applying our method leads to faster convergence and better final performance, across all MTL methods considered.\n\nWith the additional results on ImageNet, we believe our work has gained stronger empirical results. We will include the detailed experimental setup and results in the final manuscript.\n\n---\n\n### **Q1: Comparison to the average conflict of the baseline in Fig. 3a.**\n\nWe thank the reviewer for this valuable insight. We report the average conflict of the baseline in **Fig. C of the Global Response**. As can be seen in Fig. C, the average conflict of the baseline does not significantly differ from that of PCgrad (Fig. 3a). This result is in line with the recent findings in [B], which show that MTL methods dealing with conflicting gradients do not actually reduce the *occurrence* of conflicting gradients. We will include this additional information for clarity and completeness in the final manuscript.\n\n---\n\n### **Q2: Ablation studies on the number of clusters.**\n\nWe apologize for not making explicit references in the main paper to the ablation experiment results in the Appendix. We would like to point the reviewer to Section D.2 of the Appendix, where we perform ablation on the number $k$ of clusters.\n\nAs shown in the result, our method is robust to the number of clusters and improves upon vanilla diffusion training in all cases considered. Specifically, we see that applying MTL methods with only two clusters ($k=2$) already leads to a significant performance boost, which confirms the effectiveness of our method. For clarity and completeness, we will provide a concise summary of the ablation experiments and make an explicit reference to the results in the main paper.\n\n---\n\n### **References**\n\n[A] Peebles et al., “Scalable Diffusion Models with Transformers”, arXiv 2022.\n\n[B] Shi, Guangyuan, et al., “Recon: Reducing Conflicting Gradients From the Root For Multi-Task Learning.”, ICLR 2023."}, {"review_id": "6rDdWwWB8Z", "reviewer": "Reviewer_8Ptq", "summary": "This paper analyzes diffusion training from an MTL perspective and observes the negative transfer in diffusion training. Several multi-task learning algorithms are employed to address the negative transfer problem, leading to improved performance.", "questions": "Have you considered utilizing the weights obtained by MTL approaches as the loss weights for training the model? For instance, using the weighted loss with the mean weight of each interval in Figure 3b. Would this approach yield comparable results?", "limitations": "See weakness.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a detailed analysis of task affinity and negative transfer across different diffusion steps, which is interesting.\n2. The incorporation of MTL methods has yielded a substantial improvement in the performance of the diffusion model.", "weaknesses": "1. The paper lacks a comparison of time and GPU memory usage between the vanilla and MTL approaches. Typically, MTL methods require more time and significantly higher GPU memory.\n2. It would be beneficial to include random weights and linear scalarization baselines. Several studies have suggested that linear scalarization and random weights can serve as strong baselines [1][2].\n\n[1] Xin et al., Do Current Multi-Task Optimization Methods in Deep Learning Even Help?, NeurIPS 2022\n\n[2] Lin et al., A closer look at loss weighting in multi-task learning, arXiv 2022\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the insightful comments by reviewer 8Ptq. The comments are very helpful in making our work more complete from an MTL perspective. We will address all raised concerns by the reviewer and revise the paper accordingly. \n\n---\n\n### **W1: Lack of comparison of time and GPU memory complexity.**\n\nThank you for your valuable suggestion. First, we point out that clustering cost is almost free compared to the cost of diffusion model training, with the exception of gradient-based clustering which requires training of a diffusion model until convergence. Next, in order to assess the additional resource requirements incurred by MTL, we measured GPU memory usage and runtime when training with the LDM architecture and timestep-based clustering on the FFHQ dataset. The results are summarized in **Table A in the General Response**.\n\nWe explain the results below:\n\n- PCgrad exhibits a runtime of 1.523 iterations/second, which is comparatively slower than vanilla training's pace of 2.108 iterations/second. This discrepancy primarily arises from the inherent requirement of PCgrad to compute per-interval gradients for every iteration. However, it is worth noting that PCgrad uses less GPU memory compared to vanilla training. This stems from the partitioning of minibatch samples across intervals for per-interval gradient calculation, resulting in a reduced number of samples employed for backpropagation compared to vanilla training.\n- NashMTL exhibits a marginal decrease in runtime with 2.011 iterations/second compared to vanilla training. Similar to PCgrad, this minor difference in runtime arises due to the requirement for per-interval gradients. However, NashMTL offers a practical speed-up strategy by computing per-interval gradients every few iterations, in contrast to PCgrad. This distinction results in NashMTL outperforming PCgrad significantly in terms of runtime. However, NashMTL uses more GPU memory, which can be attributed to the caching of the parts for calculating weights.\n- UW exhibits comparable runtime and GPU memory usage to vanilla training, owing to its utilization of weighted loss and efficient weight updates through a single backpropagation pass.\n\nWe also note that applying MTL methods only incurs additional compute during training, and does not affect the inference time. We will add this additional information to our final manuscript.\n\n---\n\n### **W2: Additional baselines.**\n\nWe are truly grateful for your valuable suggestion. Indeed, Linear Scalarization (LS) and Random Loss Weighting (RLW) should serve as simple and strong baselines, and comparison of our methods to them will establish the necessity of applying such sophisticated MTL methods as PCGrad, NashMTL, and Uncertainty Weighting. Accordingly, we provide results for LS and RLW on the FFHQ dataset using ADM architecture in **Table B of the Global Response**. We explain the results below:\n\n- As seen in the results, LS achieves slightly worse performance than vanilla training, which suggests that simply re-framing the diffusion training task as an MTL task and applying a naïve MTL loss is not enough.\n- Also, RLW achieves much worse performance compared to vanilla training. It appears that the randomness introduced by the weighting interferes with diffusion training. This indicates that sophisticated weighting schemes such as UW and NashMTL indeed are responsible for significant performance gain.\n\nOverall, these additional experiments support the effectiveness of the applied MTL methodologies. We will include these results in the final manuscript.\n\n---\n\n### **Q1: Utilizing weights obtained by MTL approaches.**\n\nThank you for the suggestion. We find this approach very intriguing. As suggested, we obtained the average weight assigned to each task interval (for both UW and NashMTL), upon finishing the training of LDM on FFHQ. We then re-trained the model with the fixed weights obtained above. The results are presented in **Table C in the Global Response**. For both NashMTL and UW, using fixed weights leads to a slight degradation in FID and Precision Score, which suggests that adaptive weight assignment using NashMTL and UW offers an advantage, albeit slightly in this particular case. It is also notable that despite this slight decrease, using weighted loss with fixed weights still surpasses vanilla training in terms of sample quality. We will include this result in the final manuscript."}, {"review_id": "9ZxELyuy2f", "reviewer": "Reviewer_sPtA", "summary": "This paper discusses the use of diffusion-based generative models for various generative tasks, such as image, video, 3D shape, and text generation. The authors argue that negative transfer, which refers to competition between conflicting tasks leading to decreased performance, should be investigated and addressed in diffusion models. They observe a negative correlation between task affinity and the difference in noise levels, suggesting that adjacent denoising tasks are more harmonious. They also find evidence of negative transfer in diffusion model training, where utilizing a model trained exclusively on denoising tasks within a specific interval generates higher-quality samples compared to a model trained on all tasks simultaneously. To address negative transfer, the authors propose leveraging existing multi-task learning techniques and clustering denoising tasks into intervals. They demonstrate the effectiveness of their approach through experiments on image datasets, showing improved quality in generated images.", "questions": "1, What is the proposed clustering approach for addressing negative transfer in diffusion models? How does it take into account noise levels and task affinity?\n2, What are the limitations of the proposed clustering approach in terms of scalability and computational complexity?\n3, How did the authors validate the effectiveness of their approach? Can you provide more details about the experimental setup and the results obtained?", "limitations": "As stated in \"Weakness\".", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1, The paper provides a comprehensive analysis of diffusion-based generative models, considering their performance and flexibility in various generative tasks, including image, video, 3D shape, and text generation. It highlights the achievements and potential areas for improvement in diffusion models.\n2, The authors identify and address the issue of negative transfer in diffusion models. They observe evidence of negative transfer during model training and propose a clustering approach to mitigate its impact on denoising tasks. This analysis adds valuable insights to the field of multi-task learning and diffusion models.", "weaknesses": "1, The experiments conducted in the paper are limited to specific datasets (FFHQ and CelebA-HQ) and specific types of diffusion models (Ablated Diffusion Model and Latent Diffusion Model). This limited scope may restrict the generalizability of the proposed approach to other datasets or types of diffusion models. The paper could have benefited from evaluating the method on a broader range of datasets and models.\n\n2, While the paper presents empirical results to support the effectiveness of the proposed approach, it lacks in-depth theoretical analysis or mathematical formulation of the clustering strategy. A more rigorous theoretical analysis would have provided a deeper understanding of the approach and its underlying principles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful feedback provided by reviewer sPtA. We will address the raised concerns and revise the paper accordingly, as these comments contribute significantly to improving our work.\n\n---\n\n### **W1: Lack of experiments on different datasets and diffusion models.**\n\nThank you for bringing up this point. We acknowledge the importance of validating our method across diverse models and datasets. In response, we extended our experiments to include the transformer-based DiT-S architecture on the ImageNet 256x256 dataset. The results are presented in **Fig. A-B in the Global Response**.\n\nAcross different architectures and larger-scale datasets like ImageNet, our method consistently enhances FID, IS, and Precision across MTL methods compared to vanilla training. Notably, methods (UW, NashMTL) excelling in balancing are particularly effective. Additionally, convergence analysis in Fig. B highlights that our method (especially UW and NashMTL) noticeably accelerates convergence compared to vanilla training, and eventually leads to superior final performance.\n\nThese experiments strongly affirm our method's generalizability. We will add these results to the final version of the manuscript.\n\n---\n\n### **Q1: Proposed clustering approach for addressing negative transfer.**\n\nThe intuition behind task clustering is:\n\n- The clustering of tasks is necessary to apply MTL methods to diffusion models. Treating each denoising task as an independent task and applying MTL methods is simply not feasible due to instability and large computational complexity.\n- By clustering the denoising tasks into task groups, we best reap the benefits of multi-task learning. In other words, by letting the model treat similar tasks assigned to a task cluster simultaneously, we may expect that it prevents negative transfer among such tasks. Meanwhile, we use various methods suggested in MTL literature to reduce the negative transfer between tasks belonging to different task clusters.\n\nConsidering these, we opt for the interval clustering method based on our observation that tasks close in timestep have higher task affinity scores. To demonstrate the effectiveness of interval clustering, we tried and applied MTL methods on top of task groups discovered by Higher Order Approximation (HOA)-based task grouping which does not constraint that tasks within a cluster must be neighboring in timesteps (See Appendix D.3). We found that interval clustering outperforms HOA, showing grouping tasks into disjoint intervals is a valid strategy.\n\nWe consider timestep, SNR, and task affinity scores as clustering criteria. By considering SNR-based clustering costs, our method encourages the grouping of tasks with similar noise levels, effectively accounting for noise levels. By using task affinity-based objectives, our approach enhances task affinity among clusters, allowing for the inclusion of task affinity in cluster considerations.\n\n---\n\n### **W2 & Q2: In-depth theoretical analysis and computational complexity.**\n\nWe appreciate your thoughtful feedback. Regarding the theoretical analysis of the clustering algorithm, we have built upon well-established principles outlined in [43, 65]. In Appendix G, we have detailed use of dynamic programming to optimize interval clusters. Let $n$ and $k$ denote the number of data points ($T$, in our case) and clusters, respectively. The interval clustering algorithm employs a matrix $D[i,m]$ to store minimum clustering costs for interval length $i$ and cluster count $m$, with a total memory cost of $O(nk)$. Each matrix element involves $O(n\\omega(n))$ computation, resulting in time complexity of $O(n^2k\\omega(n))$. For each of our three clustering cost functions:\n\n- **Timestep**: The calculation of cluster costs takes $O(n)$ time. Hence, this approach operates in $O(n^3k)$ time and uses $O(nk)$ memory.\n- **SNR:** Computation of SNR in constant time keeps its time complexity at $O(n^3k)$. Memory complexity is also $O(nk)$.\n- **Gradient:** Unlike the aforementioned methods, the gradient-based clustering capitalizes on precomputed task affinity scores $\\mathrm{TAS}(t_x,t_y)$ for $x,y\\in[1, T]$, resulting in a memory cost of $O(n^2)$. The clustering cost computation involves memory-referencing, resulting in time complexity of $O(n^3k)$ and memory $O(nk+n^2)$.\n\nFurthermore, it's worth noting that potential exists for further optimization, reducing time complexity to $O(n^2k)$, as shown in [A].\n\n---\n\n### **Q3: Evaluation of the Proposed Method**.\n\nFor fair comparison and analysis, our experimental setup follows that of prior research such as ADM [7], P2 [6], and LDM [47]. We provide a list of all our experiments and their corresponding experimental setup details below:\n\n- Section 5.1: Incorporating MTL methods with interval clustering dramatically improves diffusion training. Setups are in Section 5 and Appendix E.1.\n- Section 5.2: 1) MTL methods exhibits fast convergence. 2) Our methods improve negative transfer in diffusion models. Setups are presented in Section 5 and Appendix E.1.\n- Section 5.3: Our method is also effective on a more sophisticated training objective, P2. Setups are presented in Section 5 and Appendix E.1.\n- Appendix D.3: Interval clustering outperforms the HOA-based task grouping method in terms of incorporating MTL methods. Setups are presented in Appendix D.3.\n- Other experimental analyses: We conducted several empirical analyses about 1) the existence of negative transfer, 2) task affinity, 3) ablation on $k$, and 4) the behavior of MTL methods.\n\nIn the final version, we will clearly present the above experimental settings by making explicit references. Furthermore, we release experimental code for explicitly sharing our experimental settings.\n\n---\n\n### **References**\n\n[A] Grønlund, et al., “Fast exact k-means, k-medians and Bregman divergence clustering in 1D.”, arXiv 2017."}]

Author rebuttal:
# Global Response

Dear reviewers,

We sincerely thank you for dedicating time and effort to review our manuscript. In an attached PDF file, we have provided the results of all conducted experiments during the author response period for addressing concerns and questions raised by reviewers. In this response, we offer a concise explanation and setup of experiments in the attached PDF file.

---

## Contents

- **[Figure A and B]**: To address concerns raised by reviewers 7mPc & sPtA, we conducted experiments on the large-scale dataset, ImageNet 256x256, with transformer-based diffusion models, DiT-S [A]. All training was performed for 800K iterations with AdamW optimizer with a learning rate of 1e-4 or 2e-5, and better results were reported. Figure A illustrates FID, IS, Precision, and Recall scores according to the classifier-free guidance scale from 1.5 to 3.0. Figure B depicts the trajectory of the FID score over training iterations with the classifier-free guidance scale 2.0. All samples are generated by DDPM 250-step sampler.
- **[Figure C]**: To answer question 2 of reviewer 7mPc, we plot the average number of gradient conflicts in baseline training (referred to as ‘vanilla’ in the manuscript). The settings are the same as in Fig. 3a of the manuscript.
- **[Table A]**: To address concerns raised by reviewer 8Ptq, a comparison of GPU memory usage and runtime for all methods is presented in Table A. These measurements were conducted using the setting where the number of clusters $k$ was set to 5 on the FFHQ dataset with LDM and timestep-based clustering.
- **[Table B]**: For reflecting reviewer 8Ptq’s suggestion, the results of Linear Scalarization (LS) [B] and Random Loss Weighting (RLW) [C] on the FFHQ dataset with ADM architecture are provided in Table B. All settings are the same as in Table 1 of the manuscript.
- **[Table C]:** To respond to the question of reviewer 8Ptq, we applied the weighted loss with the mean weight of each interval of UW and NashMTL in Fig. 3b and Fig. 3c and report the result in Table C. All experimental settings are the same in Figure 3 of the manuscript.

---

### References

[A] Peebles et al., “Scalable Diffusion Models with Transformers”, arXiv 2022. 

[B] Xin et al., “Do Current Multi-Task Optimization Methods in Deep Learning Even Help?”, NeurIPS 2022

[C] Lin et al., “A closer look at loss weighting in multi-task learning”, arXiv 2022

Meta-review: All reviewers liked the paper and engaged well in the reviewing and discussion phase. Accept is recommended.
Decision: Accept (poster)