TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Authors: Shaobin Zhuang
Yiwei Guo
Yanbo Ding
Kunchang Li
Xinyuan Chen
Yaohui Wang
Fangyikang Wang
Ying Zhang
Chen Li
Yali Wang
Venue: ICML 2025 poster (2025)
Keywords: Diffusion
Generation
LoRA
Timestep
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: A new diffusion model fine-tuning paradigm, effectively learns diverse noise levels via an asymmetrical mixture of TimeStep LoRA experts. .

Abstract: Diffusion models have driven the advancement of vision generation over the past years. However, it is often difficult to apply these large models in downstream tasks, due to massive fine-tuning cost. Recently, Low-Rank Adaptation (LoRA) has been applied for efficient tuning of diffusion models. Unfortunately, the capabilities of LoRA-tuned diffusion models are limited, since the same LoRA is used for different timesteps of the diffusion process. To tackle this problem, we introduce a general and concise TimeStep Master (TSM) paradigm with two key fine-tuning stages. In the fostering stage (1-stage), we apply different LoRAs to fine-tune the diffusion model at different timestep intervals. This results in different TimeStep LoRA experts that can effectively capture different noise levels. In the assembling stage (2-stage), we design a novel asymmetrical mixture of TimeStep LoRA experts, via core-context collaboration of experts at multi-scale intervals. For each timestep, we leverage TimeStep LoRA expert within the smallest interval as the core expert without gating, and use experts within the bigger intervals as the context experts with time-dependent gating. Consequently, our TSM can effectively model the noise level via the expert in the finest interval, and adaptively integrate contexts from the experts of other scales, boosting the versatility of diffusion models. To show the effectiveness of our TSM paradigm, we conduct extensive experiments on three typical and popular LoRA-related tasks of diffusion models, including domain adaptation, post-pretraining, and model distillation. Our TSM achieves the state-of-the-art results on all these tasks, throughout various model structures (UNet, DiT and MM-DiT) and visual data modalities (Image, Video), showing its remarkable generalization capacity.

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Shaobin Zhuang"
Yiwei Guo
Yanbo Ding "2 Kunchang Li Xinyuan Chen
Yaohui Wang"
Fangyikang Wang* Ying Zhang Chen Li
Yali Wang
Abstract
tensive experiments three typical and popular LoRA-related tasks of diffusion models, including domain adaptation, post-pretraining, and model distillation. Our TSM achieves the state-of-the- art results on all these tasks, throughout various model structures (UNet, DiT and MM-DiT) and visual data modalities (Image and Video), show- ing its remarkable generalization capacity.
Diffusion models have driven the advancement vision generation over the past years. However, it is often difficult to apply these large models in downstream tasks, due to massive fine-tuning cost. Recently, Low-Rank Adaptation (LoRA) has been applied for efficient tuning of diffusion models. Unfortunately, the capabilities of LoRA- tuned diffusion models are limited, since the same LoRA is used for different timesteps of the dif- fusion process. To tackle this problem, we in- troduce general and concise TimeStep Master (TSM) paradigm with two key fine-tune fine-tuning stages. In the fostering stage (1-stage), we apply different LoRAs to fine-tune the diffusion model at differ- ent timestep intervals. This results in different TimeStep LoRA experts that can effectively cap- ture different noise levels. the assembling stage (2-stage), we design novel asymmetrical mix- ture of TimeStep LoRA experts, via core-context collaboration of experts multi-scale intervals. For each timestep, we leverage TimeStep LoRA expert within the smallest interval as the core ex- pert without gating, and use experts within the bigger intervals as the context experts with time- dependent gating. Consequently, our TSM can effectively model the noise level via the expert in the finest interval, and adaptively integrate con- texts from the experts of other scales, boosting the versatility of diffusion models. To show the ef- fectiveness of our TSM paradigm, we conduct ex-
1. Introduction
Diffusion models have shown remarkable success in vi- sion generation (Rombach al., 2022b; Podell et al., 2023; Singer et al., 2022; Ho et al., 2022a; Chen al., 2024d; Esser et al., 2024b; Chen et al., 2024a). Especially with the guidance of scaling law, they demonstrate the great power in generating images and videos from user prompts (Esser et al., 2024b; Liu et al., 2024a;c; Bao et al., 2024) owing to billions of model parameters. However, it is often difficult to deploy these diffusion models efficiently in various down- stream tasks, since fine-tuning such huge models resource- consuming. To fill this gap, Low-Rank Adaptation (LoRA) (Hu et al., 2021), 1), initially developed in NLP (Chowdhary & Chowdhary, 2020), has been applied to diffusion models for rapid adaptation and efficient visual generation (Luo et al., 2023a; Li et al., 2024b; Peng et al., 2024; Yin et al., 2024b).
However, we observe that the generative capability of LoRA- tuned diffusion models limited. For illustration, we take the well-known Prt (Chen al., 2024d) as an example, which is pre-trained on et al., 2024d) for image generation. As shown in Fig. (b) and (c), we perform LoRA on two typical fine-tuning set- tings. On one hand, we fine-tune this model with LoRA on new image data (e.g.. 222lCommBeench (Huang et al., 2023)). In this setting of downstream adaptation. the LoRA-tuned model makes similar errors as the pre-traines model, i.e.. they both fail to fit the target data distribution. On the other hand, we fine-tune this model with LoRA on the pretraining image data. In this setting of post-rret,a,ning LoRA-tunce model results prompt misalignment, which deteriorates the generative capacity of the pre-traine trained model. Based on these observations, there is a natural question: why does
10M (Chen
Eqqual contribution. Equual corresponding /f Work done as intern WeChat, Tencent Inc. 'Shanghai Jiao Tong Univer- sity. "Shenzhen Key Laboratory of Computer Vision and Pat- tem Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences. 'Shanghai Artificial intelli- gence Laboratory. 'Zhejiang University. 'WeChat, Tencent Inc.. Correspondence to: Yali Wang <yl.wang @ siat.ac.en>> Xinyuan Chen <chenxinyuanee warpuuann@pllligg...mmom pjlab.org.en: en.>, Yaohui Wang <wangyao- hui @jllaboorg.cm.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Standard Deviaton (n)) liddin Soae ve Timerap
Pre-tmined Model
w LokA
wi ISM
Poovitainad Model
w LoLKAA
w TSM
1e02
Badk2
and greck
D604
Trassao
rnd boot and yollve vatee Domren Adaptation
colorfit Sature world setgged thring aning she cloudr.
(8) Analysis
(c) Part-rrtmamy
Figure 1. Motivation Visualization. (a) During generation process, the hidden states in the same block of pre-trained PixArt-a changes significantly with timestep. (b) The pre-trained model and LoRA-tuned model incorrectly generate green bench and vase, while TSM corrects these enrors. (c) LoRA-tuned model generates degraded images, while TSM benefits visual quality and text alignment.
such deterioration appear in LoRA-tuned diffusion models? We generated 30K prompts from the COCO2014 (Lin et al., 2014) validation and observed hidden states during the inference process. As shown in Fig. 1 (a), when Diffusion models process input at different timesteps, the performance of even the same block varies greatly (Balaji et al., 2022; Xue et al., 2024; Hang et al., 2023; Go et al., 2023). We then hypothesis that it the low-rank characteristic of LoRA that makes it difficult to learn complex representations at dif- ferent timesteps. In the vanilla LoRA setting, only ONE LoRA is applied for fine-tuning diffusion models at DIF- FERENT timesteps. Thus, in the downstream adaptation case, it fails to fit the new target data just like the pre-trained model. In the post-prette post-pretraining case, such an inconsistent manner would reduce the capability of diffusion models to tackle different noise levels, especially with very limited parameters in LoRA (more evidence provided in Tab. I and 2).
well as adaptively integrate contexts from the experts of other scales, boosting the versatility and generalization ca- pacity of diffusion model.
To show the effectiveness of our TSM paradigm, con- duct extensive experiments on three typical and popular LoRA-related tasks of diffusion models, including domain adaptation, post-pretraining, and model distillation. Our TSM achieves the state-of-the-art results on all these tasks, throughout various model structures (UNet (Ronneberger et al., 2015), DIT (Peebles & Xie, 2023), MM-DIT (Esser et al., 2024a)) and visual data modalities (Image, Video), showing its remarkable generalization capacity. For the above three tasks. TSM achieves the best performance on T2I-CompBench, efficiently improves model performance after post- post-pretrainigg using only public datasets, and our 4-step model reaches the FID of 9.90 on COCO2014 with a very low resource consumption of 3.7 A100 days.
To alleviate this problem, we propose general and concise TimeStep Master (TSM) paradigm, with novel asymmet- rical mixture of TimeStep LoRA experts. Specifically, our TSM contains two distinct stages of fostering and assem- bling TimeStep LoRA experts, boosting the versatility and efficiency of tuning diffusion models in vision. In the fos- tering stage, divide the training procedure into several timestep intervals. For different intervals, introduce dif- ferent LoRA modules for fine-tuning the diffusion model, leading to different TimeStep LoRA A experts. This can effec- tively enhance the diffusion model to fit the data distribution under different noise levels. the assembling stage, we combine the TimeStep LoRA experts of multi-scale inter- vals further boost performance. Specifically, we introduce novel asymmetrical mixture of TimeStep LoRA experts, for core-context expert collaboration. For each timestep, we leverage TimeStep LoRA expert within the smallest interval as the core expert without gating, and use experts within the bigger intervals of other scales as the context experts with dinendependent gating. In this case, our TSM can effectively learn the noise level via the expert in the finest interval, as
2. Related Work
Diffusion models for visual synthesis. Recently, diffu- sion models (DMs) have swept across the realm of visual generation and have become the new statc-of-the-ar gener- ative models for text-to-image (Podell et al., 2023; Nichol et al., 2021: Li al., 2023; Saharia al., 2022; Chen et al., 2024d;b:c; Xue et al., 2024) and text-to-video (Ho et al., 2022b; Blattmann et al., 2023; Khachatryan et al., 2023; Luo et al., 2023b; Wang et al., 2023; Singer et al., 2022; Chen et al., 2023a; Zhuang et al., 2024). Stable Diffusion 1.5 (SD1.5) (Rombach et al., 2022b) operates in the la- tent space and can generate high-resolution images. The Pix series (Chen et al., 2024d;b;c) provide more acces- sibility in high-quality image generation by introducing efficient training and inference strategies. SD3 (Esser et al., 2024b) demonstrates even more astonishing generation re- sults with the MM-DiT architecture and scaled-up parame- ters. VideoCrafter2 (VC2) (Chen et al., 2024a) discovers the spatial-temporal relationships of the video diffusion model and further proposes an effective training paradigm for high-
2

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
0
t=T
Timesteps of Diffusion
A
A1
A
A
θ
θ
B
B
B
B
E [1.
t E 1 TI
E I +1, T
LE 1 +
1,7]
(a) Vanilla LoRA
(b) TimeStep LoRA Experts of n intervals
Figure 2. Fostering Stage: TimeStep LoRA Expert Construction. We divide all I' timesteps into intervals and fine-ture the diffusion
model with individual LoRA module for each interval.
quality video generation. MoE based methods (Park al., 2024; Lee et al., 2023; Balaji al., 2022; Zhang et al., 2024; Fei et al., 2024; Sun et al., 2024; Fang et al., 2024) try to use multiple experts to better learn complex data distribu- tions. However, the increasing number of parameters of the DMs also makes it difficult to directly transfer its powerful capabilities to other domains.
enhanced at a low fine-tuning cost.
3. Method
In this section, we introduce our TimeStep Master (TSM) paradigm in detail. First, we briefly review the diffusion model and LoRA as preliminaries. Then, we explain two key fine-tuning stages in TSM, i.e., expert fostering and assembling. in order to build asymmetrical mixture of TimeStep LoRA experts for efficient and versatile enhance- ment of the diffusion model.
Efficient tuning of difffusion models. To reduce the cost of full fine-tuning DMs in downstream tasks and retain generalization ability, prompt tuning is a common and effi- cient fine-tuning method. DMP (Ham et al., 2024) adapts DMs to different timesteps by simultaneously training gat- ing modules and prompt tokens, but performance still lags behind that of the A-ased (Hu et al., 2021) fine-tuning method. LoRA (Hu et al., 2021) is widely applied in DMs to efficiently train low-rank matrices (Zhang et al., 2023; Ye et al., 2023; Xie et al., 2023; Mou et al., 2024; Lin et al., 2024a; Xing et al., 2024; Ran et al., 2024; Gu et al., 2024; Lyu et al., 2024; Huang et al., 2023; Go et al., 2022: Choi et al., 2024). GORS (Huang et al., 2023) applies LoRA to fine-tune the DMs to the target domain. DMD (Yin et al., 2024b) supports the use of LoRA in model distillation for fast inference. ControlNeXt (Peng et al., 2024) employs LoRA for efficient and enhanced controllable generation. T2V-Turbo (Li et al., 2024b) injects LoRA into video diffu- sion model (Chen et al., 2024a) and optimizes with mixed rewards, achieving inference acceleration and quality im- provement. DeMe (Ma et al., 2024) makes DMs better fit the data distribution by weighting the parameters of LoRA trained for different timesteps, but simple weighting cannot fully develop the knowledge of multiple LoRA. However, as discussed earlier, the generation capabilities of LoRA- tuned DMs are limited. We tackle this with our TSM, which assigns TimeStep LoRA experts to learn the distribution within diverse noise levels, and assemble these experts for further information aggregation. Using TSM, the generative performance of pre-trained diffusion models is significantly
Diffusion Model. The diffusion model is designed to learn a data distribution by gradually denoising a normally- distributed variable (Song et al., 2021; Ho et al., 2020). has been widely used for image/video generation (Rom- bach et al., 2022b; Podell al., 2023; Singer al., 2022: Ho et al., 2022a; Chen et al., 2024d; Zhuang et al., 2024; Chen et al., 2024e). In the forward diffusion process, one should add Gaussian noise e ~ N () on the input Top in order to generate the noisy input 2, at each timestep, 2y = vazzo + vI - Tye. where f = 1,2, ,T, and T is the total number of timesteps in the forward process. o₁ is a parameter related to f. When approaches T, a. approaches 0. The training goal is minimize the loss for denoising,
L - Emorce, [||e E (Iz, t, c) [1 t [1, T], (I)
where ce is the output of neural network with model param- eters θ and e indicates the additional condition, e-g., text input. To achieve superior performance, the diffusion model is often designed with a large number of network parameters that are pre-trained on large-scale web data. Apparently, is computationally expensive to fine-tune such a big model for specific downstream tasks.
Low-Rank Adaptation (LoRA). To alleviate the above dif- ficulty, LoRA (Hu et al., 2021) has been recently applied for rapid fine-tuning diffusion models on target data (Ruiz et al., 2023; Huang et al., 2023). Specifically, LoRA introduces
3

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Timestep t T
Zt
Core Expert
Context Expert
14
11 2
A
A
A4
F
H
112 = 4
B
l
+
B
i2
B ig
Big
11 =8
C Output Feature 1 1 T n 1,2, ..., -,2,-,,,,,44
is divided as m 4 scales of intervals
Figure 3. Assembling Stage: Asymmetrical Mixture of TimeStep LoRA Experts. We divide T into 4 intervals, namely n1 =8, 122 ny=2, n4=1. The TimeStep LoRA expert within the smallest-scale interval plays the core role to model the noise level of with fine granularity. The core expert (red) is without gating; the context experts (blue, yellow and green) are with gating. The router is timestep-dependent, which adaptively weights the importance of context experts at L.
low-rank decomposition of an extra matrix,
We dub the fine-tuned diffusion model as TimeStep LoRA expert at interval i. Hence, we can obtain v. TimeStep LoRA experts for n intervals of timesteps. During inference, we first sample ær from Gaussian noise FT ~ N (0, 7). and then use these TimeStep LoRA experts iteratively denoise 27, i.e., when the timestep / iterates one certain interval, we use the corresponding TimeStep LoRA expert of this interval to estimate the noise of 24, where t = 1.
+ + = o BA,
(2)
where o E fa is the pretrained parameter matrix of diffusion model. AO E R is the extra parameter matrix that decomposed as the multiplication of two low-rank matrices A E xr** and B E Rdxr, where < d, k. To achieve parameter--fficient fine-tuning, one can simply freeze the pre-trained parameter O while only learning the low-rank matrices A and B on target data for computation cost reduction. However, the generation capabilities of these vanilla LoRA-tuner diffusion models are limited. The main reason is that, diffusion model exhibits different processing modes for the noisy inputs at different timesteps (Balaji al., 2022; Hang et al., 2023). Alternatively, LoRA applies the same low-rank matrices A and B for different timesteps. Such inconsistency would reduce the capacity of diffusion model to tackle different noise levels, especially with a very limited number of learnable parameters in A and B. To address this problem, we propose a TimeStep Master (TSM) paradigm with two important stages as follows.
It is worth mentioning that, there are two extreme cases with n = I and n = T. When n = 1, it refers to the vanilla LoRA setting that is limited to capture different noise levels at different timesteps. When n T, it refers to the setting where there is LoRA expert for each timestep. Apparently, this setting makes no sense since the noise levels are similar among the adjacent timesteps. Hence, it is unnecessary to equip a LoRA for each timestep. Especially T is often large in the diffusion model, such an extreme setting introduces too many LoRA parameters to learn. Consequently, we propose divide T in different numbers of intervals, i.e. n M1, 722, 7m- In this case, for each timestep t, there are TimeStep LoRA experts. In the following, we introduce novel asymmetrical mixture of these TimeStep LoRA experts, which can effectively and adaptively make them collaborate to further boost diffusion models via multi- scale noise modeling.
3.1. Fostering Stage: TimeStep LoRA Expert Construction
To learn different modes of the noisy inputs, we propose to introduce different LoRAs for different timesteps. Specifi- cally, we uniformly divide the timesteps of T into intervals. For the i-th interval, we introduce an individual LoRA,
3.2. Assembling Stage: Asymmetrical Mixture of TimeStep LoRA Experts
Via the multi-scale design of interval division above, one can obtain m TimeStep LoRA experts for each timestep t. Hence, the next question is how to assemble their power to model the noise level of this step. Naively, one can leverage the standard Mixture of Experts (MoE) (Riquelme et al., 2021; Chen et al., 2023b) without distinguishing the role of experts. But this is not the case for TimeStep LoRA experts. Apparently, for each timestep, the TimeStep LoRA expert within the smallest interval plays the core role in modeling the noise level of this step with fine granularity. When
+ +0 = θ - B,,,,
(3)
where A; E R and B; E Rdxr refer to low-rank ma- trices in the i-th interval. We optimize A; and B, by fine- tuning the diffusion model on the noisy inputs within the i-th interval,
L = EEr,cce [lle- CB.A..B,
(,,,t,c) [12]
i
(4)
t
T
+

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Method
Colort
Shapet
Texturet
Spatial
Non-Spatial
Complex
SD1.4 (Rombach et al., 2022b)
37.65
35.76
41 41.56
12.46
30.79
30.80
SD1.5 (Rombach et al., 2022b)
36,97
36.27
41.25
11.04
31,05
30.79
SD2 (Rombach et al., 2022b)
50.65
42.21
49.22
13.42
31.27
33.86
SD2 Composable (I.iu
2022)
40.63
32.99
36.45
8,00
29.80
28.98
SD2 + Structured (Yu et al., 2023)
49.90
42.18
49.00
13.86
31.11
33.55
SD2
Exct (Wang et al., 2024)
64.00
45.17
59.63
14.55
31.09
34.01
SD2 + GORS unbaised (Huang et al., 2023)
64.14
45.46
60.25
17.25
31.58
34.70
SDXL (Podcll al., 2023)
58.79
46.87
52.99
21.33
31.19
32.37
PixArt-a (Chen et al., 2024d)
41.70
37.96
45.27
19.89
30,74
33.43
PixArt-n-ft (Chen al., 2024d)
66.90
49.27
64.77
20.64
31.97
34.33
DALLE3 (Betker et al., 2023)
77.85
62.05
70.36
28.65
30.03
37.73
SD3 (Esser
2024b)
80.33
58.49
74.27
26.44
31.43
38.62
SDI.5 Vanilla LoRA (Hu al., 2021)
51.70
44.76
$2.68
15.45
31.69
32.83
SD2 + Vanilla LoRA (Hu et al., 2021)
66.03
47.85
62.87
18.15
31.93
33.28
PixArt-ax + Vanilla LoRA (Hu
2021)
46.53
43.75
53.37
23.08
30.97
34.75
SD3 + Vamilla LoRA (Hu et al., 2021)
82.41
62.32
77.27
31.87
31.72
38.41
SD1.5 + TSM (Ours)
57.12
46.65
58.16
18.80
31.83
32.94
SD2 + TSM (Ours)
75.93
53.34
67.44
18.34
31.47
34.20
PixArt-a TSM (Ours)
54.66
44.47
57.12
25.41
31.05
34.85
SD3 TSM (Ours)
83,45
63.16
78.18
34,50
31.81
38.71
Table Domain Adaptation on 12l-CompBench. Our TSM exhibits the strongest performance in all categorics 121-Compbench.
Image Modality Method
Colort
Shapet
Texturef
Spatialt
Non-Spatial"
Complex?
PixArt-c (Chen et al., 2024d)
41.70
37.96
45.27
19.89
30,74
33.43
LoRA (Hu al., 2021)
43.47 1.77
34.74 3.222
41.57 3.70
15.37 4.52
30.74
30.43 1 3.00 32.96 J 0.47
+ TSM (Ours)
48.86 1 7.16
37.97 0.01
47.31 2.04
21.55 1 1.66
31.13 0.39
Video Modality Method
1St
Action
Amplitude?
BLIP-BLEUT
Colort
Count
VC2 (Chen al., 2024a)
16.76
77.76
44.0
23.02
46.74
53.77
+ LoRA al., 2021)
15.06
1,70
3.91
46.0 2.0
21.89
41.30 5.44
27.89 J. 25.88
+ TSM (Ours)
18.08
1,32
80.77
3.01
54.0 1 10.0
24.26 1 1.24
60.87
14.13
60.38 1 6.61
Table 2. Image/Video Modality Post-Pretraining on T2I-CompBench/eralCrattee TSM performs much better than vanilla LoRA.
the interval is bigger, the granularity of noise modeling is getting bigger, i.e., the TimeStep LoRA experts within bigger intervals are getting more insensitive to noise levels.
asymmetrical mixture of TimeStep LoRA experts,
L Exe.c,,,t [||e ce, (Ai. & (ze, t,
Based on this analysis, we introduce novel and concise asymmetrical mixture of TimeStep LoRA experts for core- context expert collaboration. Specifically, for each timestep t, we leverage TimeStep LoRA expert within the smallest interval as the core expert without gating, and use the rest (m experts as the context ones with gating,
i, T n
where the timestep t simultancouse belongs to intervals of m scales, i.e., L = 1,2, .... T, and 1,.... m. Note that, the TimeStep LoRA experts have been trained in the fostering stage. Hence, we freeze them and only learn the pa- rameters of router G( t) in the assembling stage. Via such distinct paradigm, our TSM can further boost diffusion master noise modeling via TimeStep expert collaboration, as well as inherit the efficiency of LoRA for rapid adaption.
θ + AQ; + ()), @ [AO i2:
A
= + = + 1=2 E 9j 2500 Bi Aiy,
(5)
Note that, we design the router of gating G(2,t) t) E Rm to be timestep dependent, in order to adaptively weight contexts of the rest (m-1) experts according to the timestep. Specifically, we make G(=, !) as a transformation of the timestep t and the input feature 2 e Rkx! x of this step. For simplicity, we design it as the sum over FC layer of 2, and an embedding layer 6,
4. Experiments
We apply Timestep Master (TSM) to three typical fine- tuning tasks of diffusion model in visual generation: do- main adaptation, post-pretraining, and model distillation. Extensive results demonstrate TSM achieves the sta the-art performance on all these tasks, throughout different model structures and modalities. We also make detailed ablation and visualization to show its effectiveness.
G( t) G. = F()) EE(),
(6)
4.1. Domain Adaptation
where the embedding layer refers to learnable matrix with a size 7f x (m 1). and E (t) means that we extract the parameters in the t-th row as the embedding timestep t.
Problem Definition and Dataset. Domain adaptation (Fara- hani et al., 2021) refers to the task of adapting a model trained on a source domain to perform well on different
Finally, we minimize the diffusion loss function over this
5

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Family
Method
Resolution"
Yparams
Training Cost,
FID.
DALL-E (Ramesh ct al., 2021)
256
12.0B
2048 V100 x 3.4M steps
27.5
DALL-E 2 (Ramesh et al.. 2022)
256
6.5B
41667 A100 days
10.39
Make-A-Seene (Gafni et al., 2022)
256
4.0B
11.84
GLJID (Nichol et al., 2021)
256
5.0B
12.24
Unaccelerated Diffusion
LDM (Rombach al., 2022b)
256
1.45B
12.63
Imagen (Saharia el al., 2022)
256
7.9B
4755 TPUv4 days
7.27
eDiff-I (Balaji et al., 2022)
9.1B
256 A x 600K steps
6,95
SDI.5 (50 step, ofg=3, ODE)
512
860M
6250 A100 days
8.59
SD1.5 (200 step, cfg=2, SDE)
512
860M
6250 A100 days
7.21
DPM++ (I.u et al., 2022)
512
22.36
UniPC (4 step) (Zhao
2024)
512
19.57
LCM-LoRA (4 step) (Luo et al., 2023a)
512
67M
1.3 A100 days
23.62
InstaFlow-O.9B (Liu et al., 2023)
199 A100 days
13.10
SwiftBrush (Nguyen & Tran, 2024)
512
860M
4.1 A100 days
16.67
Accelerated Diffusion
HiPA (Zhang & Hooi, 2023)
512
3.3M
3.8 A100 days
13.91
UFOGen (Xu et al., 2024b)
512
860M
12.78
SLAM (4 step) (Xu et al., 2024a)
512
860M
6 A 100 days
10.06
DMD (Yin et al.. 2024b)
512
860M
108 A100 days
11.49
DMD2 (Yin al., 2024a)
512
860M
70 A100 days
8.35
DMD2 LoRA (Hu 2021)
512
67M
3.6 A.00 days
14.58
DMD2 + TSM (Ours)
68M
3.7 A100 days
9,90
Table Model Distillation on 30K prompts from COCO2014. Our TSM achieves comparable FID while lowering the training cost significantly. Rows marked in gray demonstrate the superiority of our TSM over the vanilla LoRA based on DMD2.
but related target domain. The goal is to fit the target domain distribution while preserving the strong generalization abil- ity of the pre-trained model. We conduct domain adaptation experiments on T2I-CompBench (Huang al., 2023), open-world text-to-image generation benchmark which con- tains six domains. Each domain includes domain-specific training and testing prompts (700:300) and employs spe- cialized models to evaluate generated test images and we convert all scores into percentile for ease of reading.
trained model on a general dataset. The goal is to further improve the general performance of the model. We conduct experiments on post-pretraining tasks in both image and video modalities. For image modality, evaluate our post- trained model on 121-CompBench (Huang et al., 2023) as in Sec. 4.1. For video modality, we use EvalCrafter (Liu et al., 2024b), a public benchmark for text-to-video generation using 700 diverse prompts. Specifically, we adopt Inception Score (IS) for video quality assessment. For motion quality, we consider Action Recognition (Action) and Amplitude Classification Score (Amplitude). We evaluate text-video alignment with Text-Text Consistency (BLIP-BLPP and Object and Attributes Consistency (Color and Count).
Implementation Details. Following (Huang et al., 2023), generate 90 distinct 5 512x5 12 resolution images per train- ing prompt for adaptation. We conduct both vanilla LoRA (Hu et al., 2021) and TSM experiments based on the pre- trained models of SD1.5 (Rombach et al., 2022a), SD2 (Rombach et al., 2022a), PixArt-a (Chen et al., 2024d) and Stable Diffusion (SD3) (Esser al., 2024b). Regarding the configuration of model training, for the sake of sim- plicity, adopted the default configuration in the PEFT (Mangrulkar et al., 2022) library. More details in Sup. A.
Implementation Details. For image modality, we conduct both vanilla LoRA (Hu et al., 2021) and TSM experiments based on the pre-trained model Pix (Chen al., 2024d) and the training dataset SAM-LLaVA-Caption 10M (Chen et al., 2024d). For video modality, conduct experiments based the pre-trained VideoCrafter2 (Chen et al., 2024a) and use a 70k subset of Open Vid- 1M (Nan et al., 2024) for post post-pretraining. More details can be seen in Sup. A.
As shown in Tab. 1, TSM achieves state-of-the-ar results on 12l-CompBench and is far ahead in domains of color, shape, texture, and spatial. For complex domain, which contains more complex prompts and metrics than others, the performance of the model deteriorates after employing vanilla LoRA for domain adaptation. However, TSM can still improve the model performance.
As shown in Tab. 2, the performance of models using vanilla LoRA for post-pree pest-pretraiinn drops significantly while TSM improves model performance with the same training data. In Sup. C. we also provide the TSM post-pretrained PixArt- a evaluation result on COC02014 (Lin et al., 2014).
4.2. Post-Pretraining
4.3. Model Distillation
Problem Definition and Dataset. Post-pretraining (Luo et al., 2022) refers to the task of continuing to train pre-
Problem Definition and Dataset. Model distillation (Gou et al., 2021) refers to the task of training a simplified and
6

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Model
FT Method
Colort
Shapet
Texture
Spatialt
Non-Spatialf
Complexi
Vanilla LoRA
51.57
44.76
52.68
15,45
31.69
32.83
5D1.5
UNet
TSM s-stage
56.48*4.0
45.911 1.15
57.081 5.12
18.011 2.56
31.77f 0.08
32.791 0.04
TSM 2-stage
57.12 5.55
46.65t 1.89
58.161 5.48
18,80 3.35
31.831 0.14
32.941 0.11
PixArt-a DIT
Vanilla LoRA
46.53
43.75
53.37
23.08
30.97
34.75
TSM 1-stage
52.84
6.31
43.921
0.17
54.071 0.7
25.351 2.27
31.031 0.06
35,041 0.29
TSM 2-stage
54.66*
8.13
44.471 0.72
57.121 3.75
25.411 2.33
31.051 0.08
34,851 0.10
SD3 MM-DiT
Vanilla LoRA
82,41
62.32
77.27
31,87
31.72
38.41
TSM 1-stage
82.52*
0.11
62.941
0.62
77.551
0.28
1.21 33.081
31.74 0.02
38 38.541 0.13
TSM 2-stage
83.45 1.04
63.161 0.84
78.181 0.91
34.501
2.63
31.81 0.09
38.711 0.30
Table 4. Domain Adaptation Ablation on T2I- BeecCh.
IMG Model
FT Method
Colort
Shapet
Texturet
Spatialt
Non-Spatial;
Complext
Vanilla LoRA
43.47
34.74
41.57
15.37
30.74
30.43
PixArt-o
TSM 1-stage
45.66 1 2.19 37.06 2.32 45.42 f 3.85
22.32 1 6.95
31.03 f 0.29
32.65 f 2.22
TSM 2-stage
48.86 1 5.39
37.97 1323 47.31 f 5.74
16.18 t 1.66
31.13 f 0.39
32.96 1 2.53
VID Model
FT Method
1St
Action
Amplitudet
BLIP--BEEU
Colort
County
Vanilla LoRA
15.06
73.85
46.0
21.89
41.30
27.89
VC2
TSM 1-stage
16.71 f 1.65 79.07 t 5.22 50.0 1 4.0
23.99 1 2.10
55.48 1 27.59 60.87 19.57 60.38 1 32.49
56.52
15.22
TSM 2-stage
18.08 1 3.02 80.77 6.92 54.0 1 8.0
24.26 1 2.37
Table 5.
Image and Video Post-Pretraining Ablation on 121-CompBench and EvalCrafter.
Metric
FT Method
Value
Model
4/ t
Colort
Shapet
Texture
Spatialf
Non-Spatial
Complex
Vanilla LoRA
14.58
/
/
57.12
46.65
58.16
18.80
31.83
32,94
FID
TSM I-stage
9.92
J. 4.66
SD1.5 UNet
x /
51.42
44.09
53.46
13.56
31.75
33.45
TSM 2-stage
9.90
J 4.68
/ x
53.64
46.24
55.08
16.31
31.72
33.42
-FID
Patch
TSM Isstage
Vanilla LoRA
15.43
11.88
3,355
PixArt-a DIT
V
x
V
V
45.37
$4.66
44,47
42,49
57.12
52.09
24.84
25.41
31,05
30.99
34.83
34.85
TSM 2-stage
11.82
4,3.61
V
x
47.23
44.69
54.30
25.25
30.99
34,86
CLIP- Scoret
Vanilla LoRA
0.3176
SD3 MM-DIT
V
V
83.45
63.16
78.18
34,50
31.81
38.71
TSM 1-stage
0.3208 1 %1.01
80.99
60.38
74.62
31.87
31.61
38.53
TSM 2-stage
0.3212 1 %1..1
/
x
82.55
62.25
76.68
31.53
31.74
38.87
Table Model Distillation Ablation based on Table 7. Gating Ablation on I2I-CompBench. The model performance is optimal DMD2.
when the router input has both 2, and t.
efficient model to replicate the behavior of complex one. Since LoRA is widely used in model distillation, we explore the capabilities of TSM in this task. We conduct experi- ments on 30K prompts from COCO2014 (Lin et al., 2014) validation set. Following DMD2 (Yin et al., 2024a), we gen- erate images from these prompts and calculate the Fréchet Inception Distance (FID) (Heusel al., 2017) from these images with 40,504 real images from validation dataset.
efficiency of our TSM in model distillation.
4.4. Ablation Studies
Overall Design. We conduct two-stage ablation experi- ments on domain adaptation, post-pretraining, and model distillation. As shown in Tab. 4, in domain adaptation, our TSM significantly outperforms the vanilla LoRA on three main generative model architectures (UNet, DiT, and MM-DiT), verifying the generalization of TSM on model architecture. The model and training settings of SD1.5, PixArt-o and SD3 can be seen in Sup. A. As shown in Tab. 5, in post-pretraining, TSM achieves huge improvements over vanilla LoRA on two modalities (image and video), verifying the generalization of TSM on visual modality. The experimental settings are same as Sec. 4.2. As shown in Tab. 6, in model distillation, TSM outperforms the vanilla LoRA on FID, Patch-FID (Lin et al., 2024b; Chai et al., 2022), and CLIP score (Radford et al., 2021b) on 30K prompts from COC02014, demonstrating the generality of our TSM throughout various tasks. We also compare the trainable pa- rameters in Tab. 9. In TSM I-stage, the training parameters are the same as vanilla LoRA A and only 1/4 in the 2-stage.
Implementation Details. We distill a 4-step (i.e., 999, 749, 499, 249) generator from 1000 steps of SD1.5 (Rom- bach et al., 2022b). Following DMD2, we first train the model without GAN loss, and then with the GAN loss on 500K real images from LeshecAst (Schuhmann al., 2022). More implementation details are in Sup. A.
Tab. 3 shows the SOTA comparison on model distillation, where N Npurams refers to the trainable parameters and Training Cost is calculated based on a single A 100 GPU. Notably, our TSM far outperforms LoRA (FID 9.90 us. 14.58) with an increase of less than 1M trainable parameters and 0.1 A 100 days gain of training cost. Although we could not achieve the lowest FID due our limited training resources, we obtain competitive result while significantly reducing the training cost. This demonstrates the effectiveness and
7

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Model
step
Colt
Shat
Tex
Spat
NST
Com
Model
Reore
Colt
Shat
Text
Spat
NSt
Comt
w/o fine-tuning
36.97
36.27
41.25
11.04
31.05
30.79
4
55.85
46.45
58.06
18.32
31.77 32.95
4
4000
49.10
44.62
53.62
14.00
31.69
33.02
1,4
56.42
45.77 56.59
17.17 31.76 32.66
32
4000
51.70
44.76
52.68
15.45
31.69
32.83
56.93
46.92 57.95
18.02
31.79
32.71
SD UNet
2
4
4
32000
4000
51.86
52.02
43.61
44.74
55.74
55.43
15.70
16.35
31.74
31.70
29.84
33.13
4
56.84
46.70 57.70
17.86
31.75
32.80
2
4
16000
54.30
45.25
57.26
17.05
31.79
31.50
SD1.5 UNct
4
8
56.96
46.12
59.00
18.43
31.74
32.76
4
4
4000
54.24
45.78
56.61
17.97
31.73
33.13
8
$6.48
45.91
$7.08
18.01
31.77
32.79
4
4
8000
55.85
46.45
58.06
18.32
31.77
32.95
1.8
54.56
45.52
56.30
17.90
31,78
33.27
8
4
4000
56.48
45.91
57.08
18.01
31.77
32.79
16
4
2000
54.20
45.61
56.39
14.56
31.71
33.16
8
57.12
46.65
58.16
18.70
31.83
32.94
56.20
46.58
58.04
18.17
31.78
32.91
w/o fine-turing
41.70
37 96
45.27
19.89
30.74
33.43
4000
46.26
42.58
$2.01
23.00
30.88
34.58
56.63
46.70
58.80
18.84
31.77
32.69
32
4000
46.53
43.75
53.37
23.08
30.97
34.75
8
1,2,4
57.59
46.18
57.69
17.91
31,82
32.78
32000
52.55
43,47
53.20
22.95
31.00
33.67
4
52.77
43.77
55.48
25.64
31.06
34.68
PixArt-o DiT
2
4
4000
50.68
43.69
54.57
24.41
30,96
34.76
1,4
53.24
43.79
54.70
25.63
31,06
35.02
16000
53.00
55.08
24.95
31.02
34.63
4 4
4000
51.96
43.42
53.38
24.76
31.02
34.98
53.57
44.29
56,26
25.55
31,04
34.58
4
8000
52.77
43.77
55.48
25.64
31.06
34.68
2
53.54
44.02
56.02
26.17
31.08
34,41
8
4
4000
52.84
43.92
54.07
25.35
31.03
35.04
PixArt-c DIT
8
52,70
43.66
55.62
25.37
31.06
34.68
16
4
2000
51.98
43.54
53.98
25.22
30,93
34.95
8
52.84
43.92
54.07
25.35
31.03
35.04
w/o fune-tuning
80.33
58.49
74.27
26,44
31.43
38.62
1.8
51.93
43.87
54.00
25.67
31.03
35.08
4000
81.28
61.31
76.65
31.28
31.70
38.55
32
4000
82.41
62.32
77.27
31.87
31.72
38,41
8
$4.66
44.47
57.12
25.41
31.05
34.85
32000
81.82
62.53
76.81
32.94
31.73
38.97
54.33
44.10
55.75
25.82
31.05
34,80
SD3
2
4
4000
81.74
61.2
76.68
32.01
31.73
38.44
8
54.03
43.73
54.72
26.06
31.03
34.83
MM-DIT
4
16000
82.60
62.71
77.80
32.98
31.79
38.61
8
1,2,4
54.80
44.26
56,30
26.00
31.05
34.78
4
4
4000
82.24
62.00
77.11
32.20
31.79
38,35
4
8000
82.76
62.77
77.57
33.01
31.75
38.54
4
82.76
62.77
77.57
33.01
31.75
38,54
8
4
4000
82.52
62.94
77.55
33,08
31.74
38,54
1,4
81.38
62.73 77.19
33.65
31.69
38.65
16
2000
82.34 62.66
76.90
32.73
31.74
38.60
83.47
63.00 77.92
34.18
31.80
38.66
Table 8. TSM 1-Stage Ablation.
and step represent the
4
2
83.14 63.09 77.87
34.36
31.81
38.63
number, rank and fine-tuning steps of Tiee experts. Values in red and blue represent the optimal and suboptimal respectively. When r==1, TSM I-stage is equal to vanilla LoRA; when 1>11, it significantly outperforms vanilla LORA.
SD3 MM-DIT
4
8
83.30
62,94 78.02
34.37
31.80
38.66
8
82,52 62.94
77.55
33.08
31.74
38.54
1,8
82.84 62.60 76.11
34.21
31.75
38.67
83.45
63.16 78.18
34.50
31.81
38.71
82.89
62.90 77.58
34.30
31.80
38.68
8
82.78
62.99
77.71
34.15
31.79
38.68
Model
SD1.5
Vanilla LoRA
Method
Trainsblle (%
0.09604
SD1.5 36.97
Cal
Sha? Spar
36.27 41.25 11.04 31,05 30.79
Text
NS1 Comft
8
1,2,4 83.02
62.97 77.83 34.12 31.79
38.60
UNet
MM I-stage
0,0960
54,63
46.66 55.35
13,23 31.66 31.84
Table 11. TSM 2-Stage Ablation on T2I-CompBench, Plcore and Trootes refer to the number of core experts and context experts respectively. Values in green represent the improved performance compared to the 1 1-stage model with the same core experts, while gray indicate the decreased. The results show that the design of asymmetric TimeStep LoRA experts assembly is better than the symmetric case or without assembly, and n1 n1 n1=1 can achieve stable performance improvement.
TM 2-stage
042222
5644
45.91 57.08
18.11 31.77 32.79
PiaAnt-
Vanilla LoRA
0.06764
16
57.66
44.99
58.13
14.11 31.82 33.20
DiT
TM I-stage
0.06764
64
59.37 46.267 58.99
15.40 3186 33.59
TM 2-stage
0.01344
Table 10. TSM 1-Stage LoRA
SD3
Vanilla LoRA,
0.05635
MM-DIT
IM 1-stage
0.05633
Rank r Ablation. We choose
SD1.5 as the pre-train model and of set n=8 for I-stage. It is obvi- Trainable Parameters. W ous that, as inercases, the per- and R represent the trainable formance of the model also in- LoRA and Router in TSM. creases significantly.
TM 2-Arage
Table Comparison
Assembling Stage. We conduct TSM 2-stage ablation exper- iments on T21-CompBench, based on TSM 1 I-stage model with r=4. The training settings are same as Fostering Stage ablation. As shown in Tab. 11, we ablate the core expert and context expert. It shows that TSM 2-stage can improve model performance in most cases compared to TSM I I-stage. But surprisingly, the number of context LoRA and the per- formance in 1 I-stage have little impact on the performance in 2-stage. This is why we use the simplest n of context LoRA in the experimental settings in Sec. 4.1, 4.2, 4.3. We also study on the symmetry of the TimeStep experts without core LoRA in Tab. 11, all the TimeStep experts are context LoRA. The experiment results show that the 2-stage performance of the symmetrical pattern is often worse than the asymmetrical pattern. Finally, as shown in Tab. 7, we conduct ablation experiments on the router's 's input, and the results show that is necessary for the router to receive both feature 2, and timestep as inputs. Ablation study on TSM
Fostering Stage. We conduct TSM I-stage ablation ex- periments for TimeStep experts' n, r, and fine-tuning step on 121-CompBench, based SD1.5, PixArt-o, and SD3.
More details can be seen in Sup. A. Notably, when n-1, TSM I-stage degenerates to vanilla LoRA. As shown in Tab. 8, regardless of whether we train each LoRA for the same steps, introduce equivalent training costs (n (tx=ttp=32 or the same amount of additional parameters, all n=2, 4, 8, 16 configurations significantly outperform vanilla LoRA. This highlights that the TSM I-stage surpasses vanilla LoRA. Moreover, we can find that the performance of n and n=8 is similar and better than n=166 Therefore, we believe that n= is enough for the division of the overall timesteps. In addition, we also conduct ablation experiments on LoRA rank r in Itsag in Tab. 10, and the results show that the performance of the model will also improve as r increases.
8

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
and MoE LoRA (Li et al., 2024a) can be seen in Sup. C.2.
S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Visualization. As shown in Fig. 1 (b), in the domain adap- tation task, the TSM fine-tuned model revises the incorrect images generated by the pre-trained model, while LoRA could not. As shown in Fig. 1 (c), in the post-pretrainig task, the TSM fine-tuned model improves the alignment between images/videos and text without degrading visual quality, while the LoRA fine-tuned model exhibits a signifi- cant decline in both visual quality and vision-text alignment. More visualization results can be seen in Sup. B.
2023.
Chai, L., Gharbi, M., Shechtman, E., Isola, P., and Zhang, R. Any-resolution training for high-resolution image synthesis. In European Conference on Computer Vision, pp. 170-188. Springer, 2022.
Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang. S., Xing, J., Liu, Y., Chen, Q.. Wang, X., et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023a.
5. Conclusion
We introduce the TimeStep Master (TSM) paradigm for dif- fusion model fine-tuning. TSM employs different LoRAs on different timestep intervals. Through the fostering and assembling stages, TSM effectively learns diverse noise lev- els via an asymmetrical mixture of TimeStep LoRA experts. Extensive experiments show that TSM outperforms exist- ing approaches in domain adaptation, posttrretra,nin, and model distillation. TSM demonstrates strong generaliza- tion across various urchitectures and modalities, marking significant advancement in efficient diffusion model tuning.
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang. X., Weng, C., and Shan, Y. Videocrafter2: Overcoming data limitations for nigh-quality video diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 7310-7320, 2024a.
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z.,, Luo, P., Lu, H., and Li, 7. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024b.
Impact Statement
Chen, J., Wu, Y., Luo, S., Xie, E., Paul, S., Luo, P., Zhao, H., and Li, Z. Pixart-0: Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.052555 2024c.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
Chen, J., Yu, J., Ge, C., Yao, L.,, Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Rep- resentations, 2024d.
Acknowledgement
This work is supported by the National Key R&D Program of China(N0.2022200110000))
References
Chen, T. Chen, X., Du, X., Rashwan, A., Yang, F., Chen, H., Wang, Z., and Li, Y. Adamy-moe: Adaptive multi- task vision mixture-of-experts. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 17346-17357, 2023b.
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., et al. ediff- Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint ar arXiv:2211.01324 2022.
Chen, X., Wang, Y., Zhang, L.,, Zhuang, S., Ma, X., Yu, J., Wang. Y., Lin, D.,, Qiao, Y., and Liu, Z. Seine: Short- to-long video diffusion model for generative transition and prediction. In International Conference on Machine Learning, 2024e.
Bao, F, Xiang, C., Yue, G., He, G., Zhu, H., Zheng, K., Zhao, M., Liu, S., Wang, Y., and Zhu, J. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint 240v:2405.042233 2024.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for con- trastive language-iman learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L.., Ouyang, L., Zhuang. J., Lee, J.. Guo, Y. et al. Improving image generation with better captions. Computer Science. 2(3):8, 2023. Blatmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim,

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Choi, J. Y., Park, J. R., Park, I., Cho, J., No, A., and Ryu, E. K. Simple drop-in lora conditioning atten- tion layers will improve your diffusion model. ArXiv, abs/2405.03958 2024.
Ham, S., Woo, S., Kim, J.-Y., Go, H., Park, B., and Kim, C. Diffusion model patching via mixture-of-promps. ArXiv, abs/2405.17822 2024.
Hang, T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H.,, Geng, X., and Guo, B. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Chowdhary. K. and Chowdhary, K. Natural language pro- cessing. Fundamentals of artificial intelligence, pp. 603- 649, 2020.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024a.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. In Conference and Workshop on Neural Information Processing Systems, 2020.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., al. Scaling rectified flow transformers for high- resolutooo image synthesis. In International Conference on Machine Learning, 2024b.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300, 2022a.
Fang, G., Ma. X., and Wang, X. Remix-dit: Mixing dif- fusion transformers for multi-expert denoising. ArXiv, abs/2412.05628, 2024.
Farahani, A., Voghoei, S., Rasheed, K., and Arabnia, H. R. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pp. 877-894, 2021.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8644 2022b.
Fei, Z., Fan, M., Yu, C., Li, D., and Huang, J. Scaling diffusion transformers to 16 billion parameters. ArXiv, abs/2407.1166 11633, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 102.20000099... 2021.
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., and Taigman, Y. Make-a-scene: Scene-based text- to-image generation with human priors. In European Conference on Computer Vision, pp. 89-106. Springer, 2022.
Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i- compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neu- ral Information Processing Systems, 36:78723-78747, 2023.
Go, H., Lee, Y., Kim, J.-Y., Lee, S., Jeong, M., Lee, H. S., and Choi, Towards practical plug-and-play diffusion models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1962-1971, 2022.
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., and Shi, H. Text2video- zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 15954 15964, 2023.
Go, H., Kim, J., Lee, Y., Lee, S., Oh, S., Moon, H., and Choi, Addressing negative transfer in diffusion models. ArXiv, abs/2306.000 000554 2023.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey. International Journal of Computer 2021.
Lee, Y., Kim, J.-Y., Go, H., Jeong, M., Oh, S., and Choi, S. Multi-architecture multi-expert diffusson models. ArXiv, Abs/2306.04990 2023.
Vision,
Gu, Y., Wang, X., Wu, J. 7., Shi, Y., Chen, Y., Fan, Z Xiao, W., Zhao, R., Chang, S., Wu, W., et al. Mix- of-show: Decentralized low-rank adaptation for multi- concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Li, D., Ma, Y., Wang, N., Ye, Z.,, Cheng, Z., Tang, Y., Zhang. Y., Duan, L., Zuo, J., Yang, C., and Tang, M. Mixlora: Enhancing large language models fine- tuning with lora-based mixture of experts. arXiv preprint arXiv:2404, 15159, 2024a.
10

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Li, J., Feng, W., Fu, T.-.,, Wang. X., Basu, S., Chen, W., and Wang, W. Y. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arXiv preprint arXiv:2405. 18750, 2024b.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint urXiv:2211.01093 2022.
Li, Y.. Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., and Lee, J. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511- 22521, 2023.
Luo, H.,, Ji, L.,, Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing. 508:293-304, 2022.
Luo, S., Tan, Y., Patil, S., Gu, D., von Platen, P., Passos, A., Huang, L., Li, J., and Zhao, H. Lem-lora: A univer- sal stable-diffusion acceleration module. arXiv preprint arXiv:231 1.05566, 2023a.
Lin, H., Cho, J., Zala, A., and Bansal, M. Ctrl-adapter: An efficient and versatile framework for adapting di- verse controls to any diffusion model. arXiv preprint arXiv:2404.09967, 2024a.
Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, I.,, Shen, Y., Zhao, D., Zhou, J., and Tan, T. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint S.Xiv:22333.83222 2023b.
Lin, S., Wang, A., and Yang, X. Sdxl-lightning: Progres- sive adversarial diffusion distillation. arXiv preprint arXiv:2402. 13929, 2024b.
Lyu, M., Yang, Y., Hong, H., Chen, H., Jin, X., He, Y., Xue, H., Han, J., and Ding, G. One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 7559-7568, 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P.. and Zitmick, C. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12,22 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Ma, Q.. Ning, X., Liu, D., Niu, L., and Zhang. L. Decouple- then-merge: Towards better training for diffusion models. ArXiv, (ils/2240.066644 2024.
Liu, B., Akhgari, E., Visheratin, A., Kamko, A., Xu, L.. Shrirao, S., Souza, J., Doshi, S., and Li, D. Playground v3: Improving text-to-image alignment with deep-fusion large language models. arXiv preprint arXiv:2409.1063 2024a.
Mangrulkar, S., Gugger, S., Debut, L.,, Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art parameter- efficient fine-tuning methods. https:: //tpt/ggitt... 2022.
com conn/huggingface//pe
Liu, N., Li. S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable dif- fusion models. In European Conference on Computer Vision, pp. 423-439. Springer, 2022.
Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., and Shan, T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 38, pp. 4296 4304, 2024.
Liu, X., Zhang, X., Ma, J., Peng, J., et al. Instaflow: One step is enough for high-quality diffusion-basett text-to- image generation. In The Twelfth International Confer- ence on Learning Representations, 2023.
Nan, K., Xie, R., Zhou, P., Fan, T., Yang, Z., Chen, Z., Li, X., Yang, J., and Tai, Y. Openvid- Im: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02377/ 2024.
Liu, Y., Cun, X., Liu, X., Wang. X., Zhang, Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalerafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22139- 22149, 2024b.
Nguyen, T. H. and Tran, A. Swiftbrush: One-step text-to- image diffusion model with variational score distillation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 7807-7816, 2024.
Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint 17X11:2402.17177, 2024c.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-d diffusion models. arXiv preprint 12, 2021.

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Park, B., Go, H., Kim, J.-Y., Woo, S., Ham, S., and Kim, C. Switch diffusion transformer: Synergizing denois- ing tasks with sparse mixture-of-experte In European Conference on Computer Vision, 2024.
Ran, L., Cun, X., Liu, J.-W., Zhao, R., Zijie, S., Wang, X., Keppo, J., and Shou, M. Z.. X-adapter: Adding universal compatibility of plugins for upgraded diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8775-8784, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pp. 4195-4205, 2023.
Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D., and Houlsby, N. Scaling vision with sparse mixture of experts. Ad- vances in Neural Information Processing Systems, 34: 2021.
Peebles, W. S. and Xie, S. Scalable diffusion mod- els with transformers. 2023 IEEF/CVF Interna- tional Conference Computer Vision, pp. 4172-4182, 2022. URL
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Peng, B., Wang, J., Zhang, Y., Li, W. Yang, M.-C., and Jia, J. Controlnext: Powerful and efficient control for image and video generation. arXiv preprint Siti::2200.0607000 2024.
Rombach. R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022b.
Podell, D., English, Z., Lacey, K.,, Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assiste Intervention-MICC. 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 241. Springer, 2015.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning trans- ferable visual models from natural language supervi- sion. In International Conference on Machine Learning, 2021a. URL https: wwtpp.ssmmatiicccheeooooo org/Corpus ID: (D:315911
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuming text-to-image dif- fusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A.. Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8768 PMLR, 2021b.
Saharia, C., Chan, W., Saxena. S., Li, L., Whang, Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to- -o-imaae dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Ex- ploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-677, 2020. URL pett://jmml. org/pappers/211/ html.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 15278-25299 2022.
Ramesh, A., Pavloy, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831. Pmlr, 2021.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. Make-a- video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792 2022.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical cext-conditional image generation with clip latents. arXiv preprint arXiv:2204. 06/25, 1(2):3, 2022.
12

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-hased generative modeling through stochastic differential equations. In International Conference on Machine Learning, 2021.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F,, Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024b.
Sun, H., Lei, T., Zhang, B., Li, Y., Huang, H., Pang, R., Dai, B., and Du, N. Ec-dit: Scaling diffusion transformers with adaptive expert-choice routing. ArXiv, abs/2410.02098,, 2024.
Yu, J., Wang. Y., Zhao, C.,, Ghanem, B., and Zhang, J. Free- dom: Training- free energy-guidee conditional diffussin model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23174-23184, 2023.
Wang, J., Yuan, H., Chen, D., Zhang, Y. Wang, X., and Zhang, Modelscope text-to-video technical report. arXiv preprint orXiv:2308.0633 2023.
Zhang, H., Lu, Y., Alkhouri, I., Ravishankar, S., Song, D., and Qu, Q. Improving training efficiency of diffusion models via multi-stage framework and tailored multi- decoder architecture. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7372-7381, 2024.
Wang, Z., Sha, Z., Ding, 7., Wang, Y., and Tu, Z. To- kencompose: Text-to-image diffusion with token-level supervision. In Proceedings of the IEEE/CVF Confer- Computer Vision and Pattern Recognition, pp. 8553-8564, 2024.
Zhang, L.., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Xie, E., Yao, L., Shi, H., Liu, Z., Zhou, D., Liu, Z., Li, J., and Li, Z. Difffit: Unlocking transferability of large diffu- sion models via simple parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4230-4239) 2023.
Zhang, Y. and Hooi, B. Hipa: Enabling one-step text-to- image diffusion models via high-firqquency-proooiii adaptation. arXiv preprint 1rX 2023.
Xing, Z., Dai, Q., Hu, H., Wu, Z., and Jiang, Y.-G. Simda: Simple diffusion adapter for efficient video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7827-7839, 2024.
Zhao, W., Bai, L., Rao, Y., Zhou, J., and Lu, J. Unipe: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Xu, C., Song, T., Feng, W., Li, X., Ge, T. Zheng, B., and Wang, L. Accelerating image generation with sub-path linear approximation model. arXiv preprini arXiv:2404.1390 2024a.
Zhuang, S., Li, K., Chen, X., Wang, Y., Liu, Z.,, Qiao, Y., and Wang, Y. Vlogger: Make your dream vlog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8806-8817, 2024.
Xu, Y., Zhao, Y., Xiao, Z., and Hou, T. Ufogen: You forward once large scale text-to-image generation via diffuuson gans. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8196-8206, 2024b.
Xue, Z., Song, G., Guo, Q., Liu, B., Zong, Z., Liu, Y., and Luo, Raphael: Text-to-image generation via large mix- ture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.067. 06721,, 2023.
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Du- rand, F., and Freeman, W. T. Improved distribution match- ing distillation for fast image synthesis. arXiv preprint arXiv:2405. /4867, 2024a.
13

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
A. More Implementation Details
A.1. LoRA and TSM Training Configuration on Domain Adaptation
A.1.I. STABLE DIFFUSION 1.5 (ROMBACH ET AL., 2022A)
For SD1.5, in vanilla LoRA and TSM 1-stage, we employ LoRA on the 1o-q. to_k, to_v and to_out.0 modules of the UNet and q-proj and v-proj modules of CLIP text encoders. We set LoRA r, 0=4, and employ zero initialization for all matrix B. The learning rate is le-4 and the weight decay is 1e-2 for both UNet and text encoder. At TSM assembling stage (2-stage), we add router the module which is equipped with LoRA and set TimeStep experts nh =8, 12=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the Adam optimizer with $1 =0.9, 0 For both UNet and text encoder, the learning rate is set to le-4 and the weight decay to 1e-2.
A.1.2. STABLE DIFFUSION 2 (ROMBACH ET AL.. 2022A)
For SD2, in vanilla LoRA and TSM 1-stage, we employ LoRA on the ro-q, to to.k, to_v and to_out, modules of the UNet and q proj and v proro modules of CLIP text encoders. We set LoRA r, a=4, and employ zero initialization for all matrix B. The learning rate is le-4 and the weight decay 1e-2 for both UNet and text encoder. At TSM assembling stage (2-stage), we add router to the module which is equipped with LoRA and set TimeStep experts n1 =8, ny=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the Adam W optimizer with 31 0190.9, 32 0.999. For both UNet and text encoder, the learning rate is set to le-4 and the weight decay to 1e-2.
A.1.3. PIXART-a (CIIEN ET AL., 2024D)
In vanilla LoRA and TSM I-stage, we employ LoRA on the to q. to ko.k, to v and to oo_ modules of the DiT (Peebles & Xie, 2022) and q.v modules of T5 text encoder (Raffel et al., 2020). We set LoRA r, a=4, and employ zero initialization for all matrix B. The learning rate is 2e-5 and the weight decay is 1e-2 for both DiT and text encoder. At TSM assembling stage (2-stage), add router to the module which is equipped with LoRA and set TimeStep experts n1 88 12=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the Adam optimizer with 3190.9, 3.-0.9 For both DiT and text encoder, the learning rate is set to 2 and the weight decay to 1c-2.
A.1.4. STABLE DIFFUSION 3 (ESSER BT AL., 2024B)
For SD3, in vanilla LoRA and TSM fostering stage (I-stage), we employ LoRA on the to-q, to_k, to_v and to_out. 0 modules of the MM-DiT and q_proj. k_proj. v-proj and out_proj modules of two CLIP text encoders (Radford et al., 2021a; Cherti et al., 2023). We set LoRA r, a=4, and employ zero initialization for all matrix B. At TSM assembling stage (2-stage), we add router the module which is equipped with LoRA and set TimeStep experts no =8, 12=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the AdamW optimizer with 3 00.9, 32 0.999. For MM-DiT, the learning rate is set to le-5 and the weight decay to le-4. For text encoder, the learning rate is set to 5e-6 and the weight decay to le-3.
A.2. LoRA and TSM Training Configuration on Post Pretraining
A.2.1. PIXART-a (CHEN ET AL., 2024D)
In vanilla LoRA and TSM Itsagge we employ LoRA on the to q, to k, to v and to out. modules of the DiT (Peebles & Xie, 2022) and q.v modules of T5 text encoder (Raffel et al., 2020). We set LoRA r, 4=4, and employ zero initialization for all matrix B. The learning rate is 2e-5 and the weight decay is 1e-2 for both DiT and text encoder. At TSM assembling stage (2-stage), we add router to the module which is equipped with LoRA and set TimeStep experts n 8, 11 We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the AdamW optimizer with =0=0.9 0.99. For both DiT and text encoder, the learning rate is set to 2e-5 and the weight decay to le-2.
A.2.2. VIDEOCRAFTER2 (CHEN ET AL., 2024A)
In vanilla LoRA and TSM 1-stage, we inject LoRA on the k, modules in both spatial and temporal layers of the 3D-UNet and proj module of OpenCIIP (Cherti et al., 2023) text encoder. We set LoRA r, a= and adopt lora-dropout only in the 3D-UNet. In TSM 2-stage we add router to the module where LoRA is injected and set TimeStep experts ni m 44 We train 5K steps for vanilla LoRA and two stages of TSM. The global batch size is 32. We use the same
14

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
optimizer setting as in image modality. The learning rate is 2e-4 and the weight decay is le-2 for both UNet and text encoder.
A.3. LoRA and TSM Training Configuration for Stable Diffusion 1.5 (Rombach al., 2022a) on Model Distillation
We employ LoRA with r=64, a=8 on 10 to_k, to_v, to_out.O, proj_in, proj_out, ff.net. O.pjj fnnet.2, convl, conv2, conv-shortcut, downsamplers. O.con,, upsamplers. w.wplerrs.O.ory. and time_emb proj modules of UNet. In vanilla LoRA, we train for 40K steps without GAN loss and 5K steps with it. In TSM 1-stage, we train the experts at 999 and 749 timesteps for 20K steps without GAN loss and 5K steps with it. At 499 and 249 timesteps, we reduce training without GAN loss to 5K steps and increase training with real image guidance to 20K and 40K steps respectively. In TSM 2-stage, we train the router and freeze other modules with n =4, 1 TimeStep experts. We only train it for 2K steps with GAN loss, due to the little N params (<c IM). The batch size is 32 without GAN loss and 16 with it (4 times for vanilla LoRA). Other settings are consistent with DMD2.
A.4. LoRA and TSM Training Configuration on Overrall Design Ablation
A.4.1. STABLE DIFFUSION 1.5 (ROMBACH ET AL., 2022A)
For SD1.5, in vanilla LoRA and TSM 1-stage, we employ LoRA on the ro-q, to.k, to_v and to.our. modules of the UNet and q-proj and v-proj modules of CLIP text encoders. We set LoRA r, a=4, and employ zero initialization for all matrix B. The learning rate is le-4 and the weight decay 1e-2 for both UNet and text encoder. At TSM assembling stage (2-stage), we add router to the module which is equipped with LoRA and set TimeStep experts no =8, 11=11. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the Adam optimizer with $1 0099, 0.999.. For both UNet and text encoder, the learning rate is set to le-4 and the weight decay to 1e-2.
A.4.2. PIXART-Q (CHEN ET AL., 2024D)
In vanilla LoRA and TSM 1-stage, we employ LoRA on the to q, to k, to.v and to oot modules of the DiT (Peebles & Xie, 2022) and v.v modules of TS text encoder (Raffel et al., 2020). We set LoRA r, a=4, and employ zero initialization for all matrix B. The learning rate is 2e-5 and the weight decay is 1e-2 for both DiT and text encoder. At TSM assembling stage (2-stage), add router to the module which is equipped with LoRA and set TimeStep experts n1 88 12=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the Adam optimizer with 31= 0.999 For both DiT and text encoder, the learning rate is set to 2e-5 and the weight decay to le-2.
A.4.3. STABLE DIFFUSION 3 (EsSER ET AL., 2024B)
For SD3, in vanilla LoRA and TSM fostering stage (I-stage), we employ LoRA on the to q. to k, to v and to out O modules of the MM-DiT and q-proj, k.proj, v-proj and out-proj modules of two CLIP text encoders (Radford et al., 2021a; Cherti et al., 2023). We set LoRA r, a=4, and employ zero initialization for all matrix B. At TSM assembling stage (2-stage), we add router the module which is equipped with LoRA and set TimeStep experts no =8, 12=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the AdamW optimizer with 31 =099, 32.0.999. For DÎT, the learning rate is set to le-5 and the weight decay to le-4. For text encoder, the learning rate is set to 5e-6 and the weight decay to 1e-3.
A.4.4. STABLE DIFFUSION 1.5 (ROMBACH ET AL., 2022
For SD1.5, in vanilla LoRA and TSM 1-stage, we employ LoRA on the 1o-q, to_k, to_v and to_out.0 modules of the UNet and proroj and v v-proj modules of CLIP text encoders. We set LoRA r, a=4, and employ zero initialization for all matrix B. The learning rate is le-4 and the weight decay is le-2 for both UNet and text encoder.
A.4.5. PIXART-a (CHEN ET AL., 2024D)
In vanilla LoRA and TSM I-stage, we employ LoRA on the to q. to k, to v and to out.0 modules of the DiT (Peebles & Xie, 2022) and v.v modules of TS text encoder (Raffel et al., 2020). We set LoRA r, 0=4, and employ zero initialization for all matrix B. The learning rate is 2e-5 and the weight decay is le-2 for both DiT and text encoder.
15

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
VC2
w/ LoRA
w/ TSM
Two birds flew around person, in the style of Sci-Fi.
A red dog playing with green frisbee in the park.
Figure 4. Comparison on Video Modality. The videos generated by the LoRA-tuned model are not aligned with the prompts, while our TSM facilitates high-quality and consistent video generation.
A.4.6. STABLE DIFFUSION (EsSER ET AL., 2024B)
For SD3, in vanilla LoRA and TSM fostering stage (1-stage), we employ LoRA on the to_g, to_k, 1o_v and to_ouf.0 modules of the MM-DiT and q q-proj, k proj, v proj and out puoj modules two CLIP text encoders (Radford et al., 2021a; Cherti al., 2023). We LoRA r, a=4, and employ zero initialization for all matrix B.
B. More Visualization result
Teacher: SDLS
w LoRA
W! TSM
Pye
a woman stands over sheet cake sitting on top a table
batter preparing swing at pitch during major game
a smart phone with an image of person on screen
a black and white picture of television set
a green train is coming down the track
giraffe walking across lush green field
Figure 5. Comparison Model Distillation. The images generated by TSM better align with the prompts, outperforming the vanilla
LoRA, and even surpassing the teacher SD1.5 some cases.
16

---

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Method
FID
PixArt-o:
23.15
PixArt-a + TSM
16.78
Table 12. Post-pretraining Evaluation on COC02014.
Method
Colort
Shapet
Texture?
Spatialt
Non-Spatialf
Complext
SDI.5 (Rombach et al., 2022b)
36.97
36.27
41.25
11.04
31.05
30.79
SD1.5 + MoE LoRA (Li et al., 2024a)
52.26
39.83
51.38
11.71
31.34
32.00
SD1 T 1-stage
56.48
34.91
57.08
18.01
31.77
32.79
SDL.5+TSM 2-stage (Symmetrical)
54.56
45.52
56.30
17.9
31.78
33.27
SDI55+TMM 2-stage (Asymmetrical)
57.59
46.18
57.69
17.91
31.82
32.78
Table 133. TSM and MoE LoRA Ablation on T21-CompBench.
B.1. Post-Pretraining
As shown Fig. (c), in the post- pest-rretrannin task, the TSM fine-tuned model improves the alignment between images/videos and text without degrading visual quality, while the LoRA fine-tuned model exhibits significant decline in both visual quality and vision-text alignment.
B.2. Model Distillation
As shown in Fig. 5, in model distillation task, the TSM fine-tuned model is more aligned with the prompts, outperforming LoRA.
C. More Experiments
C.1. Post-Pretraining Model FID Evaluation
Benchmark. We conduct experiments on 30K prompts from COC02014 (Lin et al., 2014) validation set. We generate images from these prompts and compare these images with 40,504 real images from the same validation set calculate the Fréchet Inception Distance (FID) (Heusel et al., 2017).
Post-pretraining Configuration. We perform post-pretraiing based the pre-trained PixArt-a (Chen et al., 2024d). In vanilla LoRA and TSM I-stage, we employ LoRA on the to_q, to.k, fo_v and to_out. modules of the DiT (Peebles & Xie, 2022) and q.V modules of T5 text encoder (Raffel et al., 2020). We set LoRA r, a=4, and employ zero initialization for all matrix B. The learning rate is 2e-5 and the weight decay is 1e-2 for both DiT and text encoder. At TSM assembling stage (2-stage), we add router to the module which is equipped with LoRA and set TimeStep experts n =1 n2=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the AdamW optimizer with =1=0.9, 0.99.9 For both DiT and text encoder, the learning rate is set to 2e-5 and the weight decay to le-2.
As shown in Tab. 12, after using TSM to fine-tune the original pre-training model on the original pre-training dataset, the zero-shot generation capability of the pre- pre-training model can still be improved.
C.2. Ablation Experiments on TSM and MoE LoRA
Benchmark. We conduct TSM and MoE LoRA ablation experiments T21-CompBench (Huang et al., 2023) based on pre-trained Stable Diffusion 1.5 (Rombach et al., 2022a) for domain adaptation task. We set up four experiments with different settings for ablation: MoE LoRA, TSM 1 1-stage, using the traditional symmetric MoE architecture to ensemble multiple LoRAs obtained from TSM 1-stage, and TSM 2-stage.
Domain Adaptation Configuration. We conduct experiments on the 1o.4, to k, lo_v and to_out.0 modules the UNet and q-proj and v-proj modules of CLIP text encoders. We set LoRA r, a=4, and employ zero initialization for all matrix B, The learning rate is le-4 and the weight decay is 1e-2 for both UNet and text encoder. At TSM assembling stage (2-stage), we add router to the module which is equipped with LoRA and set TimeStep experts ni =8, 11=1. We train 4K steps for vanilla LoRA and two stages of TSM. The global batch size is 64. We use the Adam W optimizer with 8 =0.9, =0.999 For both UNet and text encoder, the learning rate is set to le-4 and the weight decay to le-2.
The results of the ablation experiment show that the effect of our TSM is far better than that of MoE LoRA, and the asymmetric MoE architecture also has obvious advantages compared to the symmetric model.
17


Reviews:
[{"review_id": "XPWePIGmAV", "reviewer": "Reviewer_4dgm", "summary": "This paper introduces TimeStep Master (TSM), a method that employs multiple LoRA experts, each specialized in specific timestep regions. The authors empirically analyze the degradation caused by sharing LoRA parameters across all timesteps, which motivates their proposal of expert LoRAs tailored to distinct timestep ranges. Specifically, they construct specialized LoRAs by evenly dividing timesteps (or noise levels) and further refining them through multi-scale partitioning to capture coarse-to-fine timestep variations. These LoRA parameters are then dynamically routed in a mixture-of-experts (MoE) fashion, enabling effective ensembling.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer 4dgm:\n\nThank you for your thorough review of our paper and your valuable suggestions. Below are our point-by-point responses:\n\n**Question1:** Some closely related works that explore similar approaches but are not cited include DMP and Decouple-Then-Merge.\n\n**Answer1:** We sincerely appreciate your reminder.\n1. We were previously unaware of the DMP work, which shares similarities with our method. We will elaborate on this comparison in **Question4**.\n2. Decouple-Then-Merge was published in CVPR 2025, with its release date preceding the ICML 2025 submission deadline. Unfortunately, we could not access this work during our submission period. However, we will elaborate on this comparison in **Question4** and we will include citations to both works in the final version.\n---\n**Question2:** Essential References Not Discussed: Multi-Experts [2,3,4,5,6]: These approaches employ specialized parameters to mitigate conflicts between denoising tasks across different timesteps, as discussed in [1]. Mixture-of-Experts (MoE) [7,8,9]: This technique is used in their method specifically for ensembling LoRA experts.\n\n**Answer2:** Thank you for highlighting the need for broader discussion.  We deeply appreciate your additional references.\nWe will discuss the differences between us in detail below and these will be incorporated into the final version.\n\n|Method|Limitations|TSM Advantages|\n|-|-|-|\n|[2] Multi-Architecture Multi-Expert Diffusion Models. [3] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers. [6] Improving training efficiency of diffusion models via multi-stage framework and tailored multi-decoder architecture.|Require training multiple large models (storage/inference challenges)|Uses LoRA for **efficient** timestep interval tuning in 1-stage and ensembles knowledge in 2-stage|\n|[4] Towards Practical Plug-and-Play Diffusion Models.|Applies LoRA only to guidance models|Directly integrates LoRA into both text encoder and diffusion models|\n|[5] Mixture of efficient diffusion experts through automatic interval and sub-network selection|Focuses on timestep redundancy for inference speed|Achieves **superior performance** via two-stage specialization and ensembling|\n|[7] Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts. [8] Scaling Diffusion Transformers to 16 Billion Parameters. [9] EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing.|Need to train the model from scratch, which consumes resources.|TSM is compatible with the existing diffusion model and continues to improve model performance.|\n---\n**Question3:** The discussion on similar approaches in the diffusion model literature feels somewhat lacking. Multi-expert methods [2,3,4,5,6] have proposed a conceptually similar approach by assigning specialized parameters to different timestep regions, thereby resolving conflicts between denoising tasks across timesteps.\n\n**Answer3:** Thank you for your extensive research on our article. We have supplemented **Question2** with detailed discussions and will include them in the final version.\n\n---\n**Question4:** DMP [7] and Decouple-Then-Merge [8] explicitly adopt a strategy of dividing timesteps and merging them later, making them particularly relevant.\n\n**Answer4:** We give a detailed comparison below and all experiments use the same training and evaluation settings\n1. DMP\n- Trains gating modules and prompt tokens simultaneously\n- TSM uses **Two-Stage training**:\n  - Stage 1: Specializes experts per timestep interval\n  - Stage 2: Freezes experts and learns gating modules\n\nLAION-5B Results (FID↓):\n| Model|FID|\n|-|-|\n| SD 1.5|47.18|\n| SD1.5 + DMP|35.44|\n| SD1.5 + TSM (Stage1)|33.45|\n| SD1.5 + TSM (Stage2)|**32.73**|\n\n2. Decouple-Then-Merge (DeMe)\n- Merges parameters via weighted averaging\n- TSM uses **gating mechanisms** (See Table 7 for gating mechanism ablation) to preserve timestep-specific knowledge\n\nMSCOCO Results (FID↓):\n| Model|FID|\n|-|-|\n| SD 1.5|13.42|\n| SD1.5 + DeMe|13.06|\n| SD1.5 + TSM (1-stage)|11.98|\n| SD1.5 + TSM (2-stage)|**11.92**|\n\n---\n**Question5:** Additional Training Costs for Multi-Expert Convergence. A major drawback of multi-expert methods is the increased training cost. Since TimeStep Master (TSM) follows a similar paradigm by specializing LoRA experts for different timesteps, it inherently inherits this issue.\n\n**Answer5:** We rigorously **evaluated computational efficiency** in **Table 8**:\n- **Equal Cost Settings**: Training multi-experts (e.g., `n=4, r=4, step=8k`) outperforms single-expert training (`n=1, r=4, step=32k`) under the same training cost (4✖️8k==1✖️32k)\n- **Scaling Benefits**: Performance improves consistently with more training steps (e.g., `step=8k` vs. `step=4k` for `n=1, r=4`)\n---\nWe sincerely thank you again for your insightful feedback. All suggestions and experimental results will be integrated into the final version. Please let us know if further clarifications are needed.", "claims_and_evidence": "I think most claims are clear, as the effectiveness of using specialized parameters for divided timesteps has already been demonstrated.", "methods_and_evaluation_criteria": "The evaluation metrics and datasets appear reasonable, but the comparison is primarily against LoRA, leaving out several baselines. I believe their setup should include comparisons with multi-expert methods and MoE-like diffusion models.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I think the training costs reported in Table 3 are somewhat exaggerated, but it is not a critical issue.", "relation_to_broader_scientific_literature": "In this paper, the authors primarily focus on enhancing LoRA for efficient fine-tuning of diffusion models. Previous works on leveraging LoRA for efficient tuning have explored three main directions: (1) dataset design, (2) distillation objectives (e.g., reinforcement learning), and (3) LoRA architecture. This work falls under the third category, LoRA architecture, aiming to improve it by incorporating timestep division, MoE, multi-expert modeling, and multi-task learning—concepts that are extensively studied in the diffusion model literature.\n\nNotably, some closely related works that explore similar approaches but are not cited include DMP and Decouple-Then-Merge (but not cited):\n- Diffusion Model Patching via Mixture-of-Prompts (DMP), AAAI 2025\n- Decouple-Then-Merge: Fine-Tuning Diffusion Models as Multi-Task Learning, CVPR 2025\nThese works share similarities with the proposed method and should be considered in the discussion.", "essential_references_not_discussed": "As mentioned in Relation to Broader Scientific Literature, this work is closely related to diffusion timestep division. Therefore, it would be beneficial for the paper to explicitly discuss this topic.\n\nMulti-Experts [2, 3, 4, 5, 6]: These approaches employ specialized parameters to mitigate conflicts between denoising tasks across different timesteps, as discussed in [1].\nMixture-of-Experts (MoE) [7, 8, 9]: This technique is used in their method specifically for ensembling LoRA experts.\nGiven these connections, it would be useful to compare the proposed approach with these existing methods. The key novelty of this work lies in multi-scale timestep division, which enables the model to leverage both broadly trained LoRA experts that capture overall denoising patterns and fine-grained LoRA experts that specialize in specific timesteps. This flexibility makes the method more adaptable to different granularity levels of denoising tasks.\n\nAs long as the additional computational cost remains manageable, I believe this is a promising approach.\n\n### Reference\n- [1] Addressing Negative Transfer in Diffusion Models, Neurips 2023,\n- [2] Multi-Architecture Multi-Expert Diffusion Models, AAAI 2024\n- [3] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers\n- [4] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023\n- [5] Mixture of efficient diffusion experts through automatic interval and sub-network selection, ECCV 2024\n- [6] Improving training efficiency of diffusion models via multi-stage framework and tailored multi-decoder architecture, CVPR 2024.\n- [7] Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts, ECCV 2024\n- [8] Scaling Diffusion Transformers to 16 Billion Parameters,\n- [9] EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing, ICLR 2025.", "other_strengths_and_weaknesses": "## Strength \n\n1. The paper is clearly written and well-structured, making it easy to follow the motivation, methodology, and results.\n\n2. The authors conduct a thorough experimental evaluation, demonstrating the effectiveness of their approach with various settings.\n\n3. The concept of multi-scale timestep division for LoRA is an interesting and novel aspect. \n\n## Weakness \n1. Lack of discussions with timestep division in the diffusion model literature\n   - This paper considers LoRA specialization based on timestep division as its core idea. However, the discussion on similar approaches in the diffusion model literature feels somewhat lacking.\n\n   - Multi-expert methods [2, 3, 4, 5, 6] have proposed a conceptually similar approach by assigning specialized parameters to different timestep regions, thereby resolving conflicts between denoising tasks across timesteps. These works have demonstrated the effectiveness of such strategies [1], making them highly relevant to this paper’s methodology. Given the strong similarity between these approaches and the timestep-specialized LoRA concept, a more detailed discussion would be beneficial.\n\n   - Furthermore, prior works have already explored timestep division for fine-tuning diffusion models in setups very similar to the one proposed in this paper. Notably, DMP (Diffusion Model Patching via Mixture-of-Prompts) [7] and Decouple-Then-Merge [8] explicitly adopt a strategy of dividing timesteps and merging them later, making them particularly relevant. Given the conceptual overlap, an empirical comparison with these works would provide a clearer positioning of this paper’s contributions.\n\n2. Additional Training Costs for Multi-Expert Convergence\n\n   - A major drawback of multi-expert methods is the increased training cost due to the need for each expert to converge on its respective timestep region. Since TimeStep Master (TSM) follows a similar paradigm by specializing LoRA experts for different timesteps, it inherently inherits this issue. However, the paper does not provide sufficient validation on how well the model mitigates this problem. To address this, it would be beneficial to include an analysis of training efficiency. One possible way to validate this is by presenting a training step vs. evaluation metric curve, similar to iteration-based learning curves, to show the additional computational overhead and how performance scales with training.\n\n\n### Reference \n- [1] Addressing Negative Transfer in Diffusion Models, NeurIPS 2023\n- [2] Multi-Architecture Multi-Expert Diffusion Models, AAAI 2024\n- [3] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers\n- [4] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023\n- [5] Mixture of Efficient Diffusion Experts through Automatic Interval and Sub-Network Selection, ECCV 2024\n- [6] Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture, CVPR 2024\n- [7] Diffusion Model Patching via Mixture-of-Prompts (DMP), AAAI 2025\n- [8] Decouple-Then-Merge: Fine-Tuning Diffusion Models as Multi-Task Learning, CVPR 2025", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AIJ8BC7mhS", "reviewer": "Reviewer_whQF", "summary": "This article addresses the issue of limited model performance during the fine-tuning process of diffusion models, which arises from the use of the same LoRA across different time steps. We propose the TimeStep Master method, which employs different LoRAs at varying time step intervals to fine-tune the diffusion model. This allows different TimeStep LoRA experts to effectively capture varying noise levels. However, the article does not sufficiently elaborate on the scientific issues it addresses. For example, in the diffusion model, why is it necessary to learn different LoRAs during the fine-tuning process when the same set of U-Net network parameters can identify different noise levels? Besides, what issue can be illustrated by subfigure “a” of Figure 1? What does the hidden state of each block represent? According to the scientific problem proposed in the article, the model parameters should differ for different time steps, but the parameters are locked during the inference process. How can subfigure a be used to demonstrate the existence of this problem?", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer whQF:\n\nWe sincerely appreciate the time and care you invested in reviewing our manuscript. Your insightful comments and suggestions have been extremely valuable, and we are grateful for the opportunity to clarify these points. Below are our detailed responses:\n\n**Question1:** What does the hidden state of each block represent?\n\n**Answer1:** In Figure 1(a), the term “hidden state” refers to the output produced by a module after the input has been processed. For example, the curve labeled “Block26” represents the variable obtained after the input has passed through the first 26 modules of the model (noting that each transformer block typically comprises both an attention module and a feed-forward network module). We hope this explanation clarifies your query.\n\n**Question2:** What issue can be illustrated by subfigure “a” of Figure 1? How can subfigure a be used to demonstrate the existence of this problem?\n\n**Answer2:** In Figure 1(a), the curves show the progression of hidden states across various timesteps within each module. This visualization reveals that even within a single module, hidden state representations vary substantially over time. This significant variation underscores the challenge of using a single set of model parameters to capture such diverse representations consistently. Our observations are in line with similar findings discussed in previous works [1–8].\n\n**Question3:** Why is it necessary to learn different LoRAs during the fine-tuning process when the same set of U-Net network parameters can identify different noise levels?\n\n**Answer3:** Addressing the challenge mentioned in **Answer2**—namely, that a single set of network parameters may struggle to adequately model the data distribution across different timesteps—it becomes a natural choice to introduce distinct parameters for various timestep intervals. Directly augmenting the full model with new parameters for each interval, however, would lead to prohibitively high computational costs. To balance efficiency and effectiveness, we employ LoRA (Low-Rank Adaptation) during fine-tuning, which allows us to efficiently adapt the network across multiple timestep intervals without incurring excessive cost.\n\n**Question4:** The model parameters should differ for different time steps, but the parameters are locked during the inference process.\n\n**Answer4:** Thank you for this excellent observation. To reconcile this, our approach incorporates a gating mechanism within the TSM 2-stage framework. Although the underlying network parameters remain fixed during the inference process, the gating mechanism dynamically adjusts the contribution (or weights) of multiple experts. This adaptive activation allows the model to effectively tailor its response to different timesteps. We have provided a detailed demonstration of the effectiveness of this experts ensemble using the 2-stage gating approach in Tables 4, 5, 6, and 11, and we further analyze the gating structure in Table 7.\n\n**References**  \n[1] Multi-Architecture Multi-Expert Diffusion Models, AAAI 2024.  \n[2] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers  \n[3] Towards Practical Plug-and-Play Diffusion Models, CVPR 2023.  \n[4] Mixture of efficient diffusion experts through automatic interval and sub-network selection, ECCV 2024.  \n[5] Improving training efficiency of diffusion models via multi-stage framework and tailored multi-decoder architecture, CVPR 2024.  \n[6] Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts, ECCV 2024.  \n[7] Denoising Task Routing for Diffusion Models, ICLR 2024.  \n[8] Addressing Negative Transfer in Diffusion Models, NeurIPS 2023.\n\n---\nThank you once again for your thoughtful and constructive feedback. Your insightful suggestions have greatly enhanced our manuscript, and we have carefully integrated the corresponding experimental details and clarifications into the final version. We sincerely hope these revisions have addressed your concerns, and if you feel they have been satisfactorily resolved, we would be most grateful if you could consider revising your evaluation score accordingly. Please do not hesitate to let us know if you need any further explanations.", "claims_and_evidence": "subfigure “a” of Figure 1 is not clear. How can the variance changes of the hidden states of blocks at different time steps be used to demonstrate the existence of the scientific problem that this paper aims to address?", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria make sense for the introduced problem.", "theoretical_claims": "“We then hypothesis that it is the low-rank characteristic of LoRA that makes it difficult to learn complex representations at different timesteps.” The article lacks necessary evidence to support the validity of this hypothesis; simply stating that the variance of the hidden states of blocks at different time steps is large is insufficient to illustrate the problem.", "experimental_designs_or_analyses": "The experimental data is sufficient, with comparisons made to the current state-of-the-art (SOTA). Ablation experiments were conducted for the two important stages of the proposed method. Additionally, generalization validation was performed for tasks related to LoRA.", "relation_to_broader_scientific_literature": "From a performance perspective, it has improved the current performance of fine-tuning based on diffusion models and contributed to the advancement of LoRA. However, from a theoretical standpoint, the expansion of existing research theories is still insufficient, particularly in providing robust evidence for the hypothesis proposed in the article.", "essential_references_not_discussed": "Not yet", "other_strengths_and_weaknesses": "The article excels in the sufficiency of the experiments and the quality of writing and figures, which are significant highlights. However, it is somewhat lacking in theoretical exposition, particularly in the articulation of the scientific problem.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UiWFiQmAGU", "reviewer": "Reviewer_zVmL", "summary": "This paper introduces TimeStep Master (TSM), a diffusion fine-tuning framework using an asymmetrical mixture of timestep LoRA experts. Rather than applying a single LoRA module across all timesteps, which limits the adaptability of different noise levels in the diffusion process, TSM introduces a timestep-specific fine-tuning approach. Specifically, the authors present an asymmetric mixture of LoRA experts, dividing the entire timestep into different intervals, then using the smallest-scale interval as the core expert and the rest as context experts. Core experts address fine-grained noise modeling while context experts are adaptively weighted based on the timesteps. Extensive experiments verify the effectiveness of TSM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer zVmL:\n\nThank you for your careful reading and analysis of our article, and for providing valuable feedback. Below are our responses to your comments:\n\n**Question1:** Recent studies have provided in-depth analyses of timestep modeling in diffusion training, including approaches that divide timesteps into finer intervals to capture inter-task relationships between denoising tasks [1, 2] and methods that enhance timestep conditioning in attention layers [3,4]. Building on these analyses, some works have proposed more detailed fine-tuning strategies based on timesteps [5,6].\n\n**Answer1:**  We sincerely appreciate your reminder. We will supplement references to these works in the final version and provide detailed comparisons:\n- *Denoising Task Routing for Diffusion Models [1]* introduces channel mask strategies for different timesteps during training to inject temporal priors.  \n- *Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts [2]* employs timestep-conditioned gating mechanisms to regulate expert activation.  \n- *Diffit: Diffusion vision transformers for image generation [3]* adapts ViT architectures for image generation.  \n- *Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Model [4]* designs multi-expert ensembles for class labels and timesteps. \n\nAbove methods are typically used in pretraining.\n\nMethods like *Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising [5]* (learnable coefficients for timestep-specific fine-tuning) and *Diffusion Model Patching via Mixture-of-Prompts [6]* (gating modules with prompt tokens) focus on efficient timestep-aware fine-tuning. Compared with previous efficient timestep-aware fine-tuning methods, our TSM has a wider range of applicability (3 architectures, 2 modalities and 3 tasks, see **Table1,2 and 3** in our paper) and more advanced performance (see in **Question4**).\n\n---\n**Question2:** The authors should support their claim by constructing some experiments to confirm compatibility or logical connections with prior timestep-based training and fine-tuning approaches.\n\n**Answer2:**  Thanks for your reminder and we added comparisons with state-of-the-art fine-tuning methods in **Question4**.  We will include these comparative experiments in the final version of the article.\n\n---\n**Question3:** The authors should either design an experimental setup with a similar computational cost to vanilla LoRA or conduct comparative experiments against other fine-tuning methods.\n\n**Answer3:** This is an important question, and one we explore rigorously in our paper. We address this in Table 8 by comparing performance under **equal computational budgets** (e.g., `n=1, r=4, step=32k` vs. `n=4, r=4, step=8k` where `1✖️32k==4✖️8k`). Results show:\n- Training 4–8 experts outperforms single-expert setups under equal costs.\n- Performance improves with increased training steps (e.g., `step=8k` vs. `4k` for `n=1, r=4`).\n\nWhat's more, Table 13 further demonstrates TSM's superiority over MoE LoRA under identical budgets.\n\n---\n**Question4:** Its superiority over existing methods remains unproven. However, the paper lacks sufficient justification for the proposed method and comparative experiments to validate its superiority. The claim would be more robust if the authors could validate that insights and observations from previous works are compatible with TSM.\n\n**Answer4:**  Thanks for your reminder. We added comparisons with state-of-the-art fine-tuning methods  and all comparative experiments use the same training and evaluation settings as the comparative methods.\n\n1. vs. **Diffusion Model Patching via Mixture-of-Prompts, AAAI 2025. (DMP)** on LAION-5B:\n|Model|FID|\n|-|-|\n|SD1.5|47.18|\n|SD1.5+DMP|35.44|\n| SD1.5+TSM (1-stage)|33.45|\n| SD1.5+TSM (2-stage)|**32.73**|\n\n2. vs. **Decouple-Then-Merge: Fine-Tuning Diffusion Models as Multi-Task Learning, CVPR 2025. (DeMe)** on MSCOCO:\n|Model|FID|\n|-|-|\n|SD1.5|13.42|\n|SD1.5+DeMe|13.06|\n|SD1.5+TSM (1-stage)|11.98|\n|SD1.5+TSM (2-stage)|**11.92**|\n\n---\n**Question5:** Does TSM exhibit any scaling properties by increasing the number of intervals m?\n\n**Answer5:** This is a very important question for TSM, as we describe experimental phenomena related to scaling in our paper. We will discuss this in two aspects.\n1. **Timestep intervals (n):** Table 8 shows optimal scaling at `n=4~8` under fixed budgets, with performance improving as training steps increase.\n2. **Model size scaling:** Experiments in Tables 1,4,7,8,9 and 11 confirm TSM's effectiveness across model sizes (up to SD3 with 2B parameters).\n\n---\nWe sincerely appreciate your insightful comments. Please do not hesitate to let us know if you require any further information or additional explanations. If you find that our revisions have satisfactorily addressed your concerns, we would be most grateful if you could kindly consider an improved score. Thank you once again for your valuable input.", "claims_and_evidence": "The authors build on existing evidence that diffusion models are trained to represent different data distributions at different timesteps, a concept previously framed as a multi-task learning problem. While this motivation is somewhat not novel, they strengthen their claim by constructing extensive experiments for various diffusion fine-tuning tasks.", "methods_and_evaluation_criteria": "The authors validate the effectiveness of TSM across various diffusion fine-tuning tasks, but the analysis of its design choices is insufficient. Recent studies have provided in-depth analyses of timestep modeling in diffusion training, including approaches that divide timesteps into finer intervals to capture inter-task relationships between denoising tasks [1, 2] and methods that enhance timestep conditioning in attention layers [3, 4]. Building on these analyses, some works have proposed more detailed fine-tuning strategies based on timesteps [5, 6]. Therefore, the authors should support their claim by constructing some experiments to confirm compatibility or logical connections with prior timestep-based training and fine-tuning approaches.\n\n[1] Park et al., Denoising Task Routing for Diffusion Models, ICLR 2024. \\\n[2] Park et al., Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts, ECCV 2024. \\\n[3] Hatamizadeh et al., Diffit: Diffusion vision transformers for image generation, ECCV2024. \\\n[4] Choi et al., Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Model, TMLR 2024. \\\n[5] Fang et al., Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising, NeurIPS 2024. \\\n[6] Ham et al., Diffusion Model Patching via Mixture-of-Prompts, AAAI 2025.", "theoretical_claims": "No theoretical claims and proofs.", "experimental_designs_or_analyses": "The authors use vanilla LoRA as the primary baseline to validate the effectiveness of TSM. However, since TSM incurs higher computational costs, its improved performance may be expected. To properly demonstrate the benefits of the asymmetric mixture of LoRA experts, the authors should either design an experimental setup with a similar computational cost to vanilla LoRA or conduct comparative experiments against other fine-tuning methods.\n\nOne potential experiment could involve extending the ablation studies on MoE LoRA to include other timestep-based LoRA conditioning methods [4] and alternative fine-tuning approaches [5, 6] (as referenced above).", "relation_to_broader_scientific_literature": "Extensive experiments confirm that their approach aligns with observations in previous work and demonstrates effectiveness across various diffusion fine-tuning tasks. However, its superiority over existing methods remains unproven.", "essential_references_not_discussed": "Many other timestep-based diffusion training [1, 2, 3, 4] and fine-tuning methods [5, 6] are missing.\n\n[1] Park et al., Denoising Task Routing for Diffusion Models, ICLR 2024. \\\n[2] Park et al., Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts, ECCV 2024. \\\n[3] Hatamizadeh et al., Diffit: Diffusion vision transformers for image generation, ECCV2024. \\\n[4] Choi et al., Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Model, TMLR 2024. \\\n[5] Fang et al., Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising, NeurIPS 2024. \\\n[6] Ham et al., Diffusion Model Patching via Mixture-of-Prompts, AAAI 2025.", "other_strengths_and_weaknesses": "* **Strengths**: The authors provide comprehensive experiments to verify the effectiveness of TSM. Additionally, the asymmetric mixture of LoRA experts has a strong potential for scalable diffusion fine-tuning by increasing the number of context experts.\n\n* **Weaknesses**: However, the paper lacks sufficient justification for the proposed method and comparative experiments to validate its superiority.", "other_comments_or_suggestions": "The claim would be more robust if the authors could validate that insights and observations from previous works are compatible with TSM.", "questions_for_authors": "Does TSM exhibit any scaling properties by increasing the number of intervals m? Many MoE-based diffusion training methods [1, 2] analyze their scaling behavior, so it would be valuable if the authors could provide insights into this aspect.\n\n[1] Fei et al., Scaling Diffusion Transformers to 16 Billion Parameters, arXiv 2024. \\\n[2] Sun et al., EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing, ICLR 2025.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RWZNiI5q9m", "reviewer": "Reviewer_eNpx", "summary": "The paper introduces TimeStep Master (TSM), a novel approach for fine-tuning diffusion models efficiently. Unlike traditional Low-Rank Adaptation (LoRA), which applies the same tuning across all timesteps, TSM uses TimeStep LoRA experts specialized for different noise levels. It consists of two stages: fostering, where different LoRAs are trained for specific timestep intervals, and assembling, which combines these experts asymmetrically using a core-context mechanism. This method improves adaptability and achieves state-of-the-art results in domain adaptation, post-pretraining, and model distillation, demonstrating strong generalization across architectures and visual modalities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer eNpx:\n\nWe sincerely appreciate your thoughtful and detailed review. Your insightful comments have been invaluable in guiding us to improve our manuscript. Below we provide our point-by-point responses, and we hope our clarifications and planned enhancements address your concerns.\n\n**Question1:** : The paper lacks sufficient visualization results, which are particularly critical for a study focusing on fine-tuning diffusion models.\n\n**Answer1:** Thank you for your reminder. We realize that visualization is very important for generation tasks. Due to space constraints, we had to place some visualizations in the supplementary material, only showing a comparison example with traditional methods in the main paper teaser. We will add more visualizations to the main paper in the final version, including comparison results between TSM 1-stage and 2-stage to further demonstrate TSM's effectiveness and superiority, as well as visualization of intermediate layer outputs and generation results at different training steps to further illustrate the effectiveness of our method.\n\n---\n\n**Question2:** The novelty is limited. While the proposed method appears to be relatively straightforward, essentially extending the original LoRA to different timesteps and assembling them. The lack of more complex or novel mechanisms may limit the perceived novelty of the approach. The authors should clarify how this simplicity contributes to the method’s practicality and scalability, or discuss potential directions for further enhancement.\n\n**Answer2:** Thank you for this perceptive comment. We appreciate the opportunity to elaborate on the innovative aspects of our approach. Despite its straightforward formulation, TSM includes several key contributions that we believe represent a significant advancement:\n1. **Ensemble Strategy**: We cleverly ensemble LoRAs trained with different timestep intervals via gating mechanism to leverage complementary knowledge across intervals.\n2. **Comprehensive Validation**: We comprehensively validate TSM across three architectures (UNet, DiT, MM-DiT), two modalities (image/video), and three task types (domain adaptation, post-pretraining, model distillation).\n3. **Model Scalability**: We verify TSM's effectiveness on pretrained models of varying scales, from PixArt (0.6B) to SD3 (2B).  \nWe plan to extend experiments to more tasks like text-to-image generation, image colorization, and image restoration.\n\nFurthermore, we are excited to extend our experimental evaluations to include tasks such as text-to-image generation, image colorization, and image restoration. We truly appreciate your suggestion, which has motivated us to further clarify how the simplicity of our method contributes to both its practicality and scalability.\n\n---\n\n**Question3:** The performance difference between TSM-Stage1 and TSM-Stage2 in Table 6 is very small.\n\n**Answer3:** Thank you for your keen observation. The performance difference observed in Table 6 specifically corresponds to distillation tasks. As indicated in Tables 4 and 5, the two-stage approach shows a more pronounced improvement for domain adaptation and post-pretraining tasks. We hypothesize that the relatively modest improvement in distillation may be due to early performance saturation after the first stage, leaving less room for further gains. We appreciate your detailed attention and will provide additional discussion on this aspect in the final revision.\n\n---\n\n**Question4:** Including visual comparisons that highlight the contributions of each stage would help validate the effectiveness and necessity of the two-stage design. This would also provide a clearer understanding of how the assembling stage (Stage 2) augments the fostering stage (Stage 1).\n\n**Answer4:** We completely agree that visual comparisons can effectively highlight the contributions of each stage. In response, as mentioned in our answer to Question 1, we will include clear and side-by-side visual examples comparing TSM 1-stage and TSM 2-stage. This will better illustrate the overall impact and the incremental benefits brought by the second stage. Thank you for this excellent suggestion.\n\n---\nOnce again, we are truly grateful for your encouraging and incisive feedback. Your comments have inspired us to refine our manuscript further, and we hope that the planned revisions will enhance the clarity and impact of our work. Please do not hesitate to let us know if there are any additional details or clarifications that would be helpful.\n\nThank you for your time and consideration.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper advances the field of LORA fine-tuning by introducing the TimeStep Master (TSM) paradigm, which optimizes the application of diffusion models in tasks such as text-to-image/video generation (e.g., Flux and Sora). It underscores the necessity for efficient fine-tuning methods like LoRA to manage the extensive number of parameters involved, thereby enhancing both performance and computational efficiency.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n1. This paper introduces a general and concise TimeStep Master paradigm with two key fine-tuning stages. The fostering stage (1-stage), apply different LoRAs to fine-tune the diffusion model at different timestep intervals.  The assembling stage (2-stage) use a novel asymmetrical mixture of TimeStep LoRA experts, via core-context collaboration of experts at multi-scale intervals.\n2. The extensive experiments are conducted on three typical and popular LoRA-related tasks of diffusion models, including domain\nadaptation, post-pretraining, and model distillation. \n\nWeakness: \n1. Insufficient Visualization Results: The paper lacks sufficient visualization results, which are particularly critical for a study focusing on fine-tuning diffusion models. Adding more visual examples, such as qualitative comparisons, intermediate outputs, or generated results, would provide a clearer and more intuitive demonstration of the effectiveness of the proposed TSM paradigm. Visualizations should ideally be included in the main paper to ensure direct evidence of the method’s performance is presented upfront, rather than being relegated to the supplementary material.\n2. The novelty is limited. While the proposed method appears to be relatively straightforward, essentially extending the original LoRA to different timesteps and assembling them. The lack of more complex or novel mechanisms may limit the perceived novelty of the approach. The authors should clarify how this simplicity contributes to the method’s practicality and scalability, or discuss potential directions for further enhancement.\n3. The performance difference between TSM-Stage1 and TSM-Stage2 in Table 6 is very small. Including visual comparisons that highlight the contributions of each stage would help validate the effectiveness and necessity of the two-stage design. This would also provide a clearer understanding of how the assembling stage (Stage 2) augments the fostering stage (Stage 1).", "other_comments_or_suggestions": "Too many tables are included in the main paper. I suggest moving in the supplementary material or simplifying some of them to free up space for more visualization results.", "questions_for_authors": "See weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces TimeStep Master (TSM), which is a two-stage fine-tuning approach for diffusion models using LoRA. TSM applies different LoRA adapters at different timesteps and combines them through a core-context mixture strategy. This improves noise handling and adaptability, achieving state-of-the-art results across tasks like domain adaptation, post-pretraining, and model distillation on various model types and data modalities. Four reviewers checked this paper and they have slightly different opinions about this work in the final recommendations. Notably, one reviewer raised the score after rebuttal and believes this work is good enough to be accepted. Based on the comments, recommendations, and post-rebuttal discussion, the authors’ point-to-point rebuttal addressed most of these concerns so that no significant issues remain. This paper is acceptable in its current form.
Decision: Accept (poster)