ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
Authors: Xiaomeng Yang
Lei Lu
Qihui Fan
Changdi Yang
Juyi Lin
Yanzhi Wang
Xuan Zhang
Shangqian Gao
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion
Pruning
Mixture of Experts
Efficiency
Primary area: applications
TL;DR: An all-in-one framework that significantly speeds up diffusion models by dynamically creating pruned temporal experts during finetuning.

Abstract: Diffusion models have demonstrated exceptional capabilities in generating high-fidelity images. However, their iterative denoising process results in significant computational overhead during inference, limiting their practical deployment in resource-constrained environments. 
Existing acceleration methods often adopt uniform strategies that fail to capture the temporal variations during diffusion generation, while the commonly adopted sequential $\textit{pruning-then-fine-tuning strategy}$ suffers from sub-optimality due to the misalignment between pruning decisions made on pretrained weights and the model’s final parameters. To address these limitations, we introduce $\textbf{ALTER}$: $\textbf{A}$ll-in-One $\textbf{L}$ayer Pruning and $\textbf{T}$emporal $\textbf{E}$xpoert $\textbf{R}$outing, a unified framework that transforms diffusion models into a mixture of efficient temporal experts.
ALTER achieves a single-stage optimization that unifies layer pruning, expert routing, and model fine-tuning by employing a trainable hypernetwork, which dynamically generates layer pruning decisions and manages timestep routing to specialized, pruned expert sub-networks throughout the ongoing fine-tuning of the UNet. This unified co-optimization strategy enables significant efficiency gains while preserving high generative quality. Specifically, ALTER achieves same-level visual fidelity to the original 50-step Stable Diffusion v2.1 model while utilizing only 25.9\% of its total MACs with just 20 inference steps and delivering a 3.64$\times$ speedup through 35\% sparsity.

ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
Xiaomeng Yang"", Lei Lu" Qihui Fan', Changdi Yang Juyi Lin', Yanzhi Wang's Xuan Zhang', Shangqian Gao
Northeastern University FFlorida State University
(yang. xiaome, lu 1ei1, fan. qih, yang. changgd lin. juy, yanz. wang, Psgaoêcs edu
xuan. (nornnnnnttteeatteete
Abstract
Diffusion models have demonstrated exceptional capabilities in generating high- fidelity images. However, their iterative denoising process results in significant computational overhead during inference, limiting their practical deployment sesource-constraiiee environments. Existing acceleration methods often adopt uniform strategies that fail to capture the temporal variations during diffusion gen- eration, while the commonly adopted sequential pruming-then-fire thee-ffne-tuningg strategy suffers from sub-optimality due to the misalignment between pruning decisions made on pretrained weights and the model's final parameters. To address these lim- itations, we introduce ALTER: All-in-One Layer Pruning and Temporal Expoert Routing, unified framework that transforms diffusion models into mixture of efficient temporal experts. ALTER achieves a single-stage optimization that unifies layer pruning, expert routing, and model fine-tuning by employing trainable hypernetwork, which dynamically generates layer pruning decisions and manages timestep routing specialized, pruned expert sub-networks throughout the ongoing fine-tuning of the UNet. This unified co-optimization strategy enables significant efficiency gains while preserving high generative quality. Specifically, ALTER achieves same-level visual fidelity to the original 50-step Stable Diffusion v2.1 model while utilizing only 25.9% of its total MACs with just 20 inference steps and delivering a 3.64x speedup through 35% sparsity.
Introduction
Diffusion models 2 3 emerged as a Icading class of generative models, achieving quality results in diverse tasks such image generation 4 6 77, image editing 9 and personalized content generation [10 T However, they face application limits due to costly inference: repeated full- network demoising causes latency and memory use, restricting real-time and low-resource deployment.
To mitigate the high computational cost of diffusion models, two primary strategies have emerged: one focuses reducing the number of denoising steps in the temporal dimension, and the other explores model-level compression to reduce the per-step computational burden. Extensive research has been devoted to the former, including improved samplers [12 13 14]. distillation methods (T5) and feature caching [16]. this work, we focus on the latter: how model compression can be further improved enhance efficiency without sacrificing performance? Pruning has been a long-standing approach for reducing model complexity, with recent advances focusing on structured pruning to enable hardware-friendy. acceleration. Among various structured pruning schemes, layer-wise
Equal Contribution Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0-0-0
Samplewwise
/
prompe
-
pronge
Dynamic Pruning
T::1
ALTER
proupr
(ours)
promppt
(-T-1
Denoising Timesteps
Figure 1: Comparison of model utilization in dynamic pruning. Sample-wis wise pruning can only use a static part of the model for one-specific image generation, while ALTER aims to achieve the full utilization of model capacity according to the necessity of each timestep.
pruning is the most practical in deployment, as it achieves speedup by discarding full layers with minimal modification existing model execution pipelines. However, such coarse-grained pruning, especially when applied statically across all input prompts and denoising timesteps, often results in significant performance degradation due both the complete removal of functional operators and the lack of flexibility to adapt to the varying computational demands.
To improve flexibility while retaining the deployment efficiency of layer-wise pruning, it is natural to combine dynamic pruning with coarse-grained structural removal, enabling different smaller models to be activated conditionally during inference. Existing efforts explore sample-wise dynamic strategy, where a unique subnetwork is selected for each input [8]. While this improves input adaptivity, it severely limits parameter utilization: for given sample, only a small subset of the model is used throughow the entire denoising trajectory. To further unlock the potential of layer-wise pruning. leverage the temporal asymmetry of diffusion, where different timesteps contribute unevenly to generation, to adaptively activate pruned substructures conditioned on the timestep, as shown in Fig. [ This enables full model utilization across the trajectory. while suppressing redundant computation at each specific time step. While üimestep-conditionn overall parameter utilization, each substructure still requires fine-tuning recover performance. Prior works often separate pruning and fine-tuning into distinct stages [19, 20), leading to a mismatch between fixed pruning decisions and evolving model weights. To address this, we propose to jointly optimize the pruning configuration and model parameters, allowing the substructures to co-adapt with the backbone during training.
pruning improves
To this end, we introduce the All-in-One Layer Pruning and Temporal Expert Routing (ALTER) framework, a unified and one-stage optimization approach that seamlessly integrates layer pruning with temporal expert routing and model fine-tuning. It effectively transforms the standard diffusion model into a mixture of efficient temporal experts, where each expert specialized pruned sub- network of the original model, tailored for distinct phases of the generation process. This dynamic configuration is achieved by identifying the optimal substructure for these different expert sub- networks and intelligently routing denoising timesteps to them throughout the entire finetuning phase. More specifically, we employ hypernetwork to generate layer pruning decisions based on the updated model weights continuously, while managing the routing of timesteps to the appropriate experts at the same time. The impact of these pruning and routing decisions is simulated during training via a layer skipping mechanism in the forward pass. At inference time, the finalized hypernetwork allows for the selection of the most suitable expert and the skipping of designated layers for each timestep, thereby minimizing the temporal computational redundancy.
Our contributions can be summarized as follows:
(1) We propose ALTER: A novel framework that transforms diffusion models into a mixture of efficient temporal experts by unifying layer pruning, expert routing, and fine-tuning into single-stage optimization process.
(2) Temporal Expert Routing: We introduce a hypernetwork- mechanism that dynamically generates layer pruning decisions and manages timestep routing to specialized experts. Meanwhile, the shared denoising backbone is fully utilized by temporal experts to maximize parameter utilization.

---

(3) Strong Empirical Results: We conduct extensive experiments to show that ALTER significantly reduces computational costs (e.g.. 25.9% FLOPs of 50-step Stable Diffusion v2.1) and accelerates inference (e-g., 3.64x speedup with 20 steps) thanks to the nature of layer-wise pruning while maintaining the same-level generative fidelity.
Related Work
2.1
Efficient Diffusion Models
Efforts to enhance the efficiency of diffusion models primarily explore two avenues: (1) sampling
acceleration, such fast samplers 21
22
13
14
T2
23
24
step distillation [I5] 25 26 27 28
29] and feature caching [16
30 31
32
33
(2) model compression via techniques like pruning.
quantization 34 (35,, efficient attention mechanisms B6 And work focus on pruning to
compress the structure. Pruning aims to reduce model size and inference latency with minimal
performance loss 33
38
39
40
41
42
43.. Unstructured pruning [44 45
46 47
removes
individual weights, leading to sparse patterns challenging to accelerate on common hardware. In
contrast, structured pruning 48
49
30
52] removes entire components (e.g., channels, layers),
yielding more readily deployable speedups. Within diffusion models, early structured pruning efforts often applied static pruning decisions [53]. For instance. LAPTOP-DitE -iff [19] and LD-Pruner [20] identify and discard less important UNet layers, offering coarse- grained but highly efficient static pruning. However, the optimal model structure can vary significantly across the iterative denoising process. More recent advances have explored dynamic pruning [54], where the model learns to prune in data- or mentext-dependentt manner for flexibility resource-constrannet inference scenarios.
APTP [8]. for example, incorporates prompt-routing model that learns and adaptively routes inputs to appropriate sub-architectures. However, although APTP's sample-wise approach improves adaptivity over static pruning, it typically commits single sub-model for the entire denoising trajectory, which can lead to under-utilization of the full model capacity. ALTER addresses this by introducing dynamism the timestep level.
2.2 Mixture-of-Experse
conditionally activate only subset of parameters during inference, offering promising way to scale computation-expensiee neural networks. While exten- sively utilized in large language models 57 58 the application MoE principles to enhance the efficiency of diffusion models remain less explored (59 60). APTP [18] highlights the potential of MoE interpretation in diffusion models by using a prompt router to convert each pruned model expert specializing in the assigned prompt. However, its reliance on routing each prompt to fixed expert sub-model throughout the entire diffusion loop restricts model utilization and introduces parameter redundancy due to maintaining multiple expert copies. In light of this, propose ALTER, where each expert is specialized for a specific range of timesteps and dynamically selected based on timestep embedding routing. Our method uniquely combines timestep-awant routing and structural pruning into a single-stage optimization framework for diffusion acceleration.
Mixture-of- Exxpeetsss-xxrssse---ee (MoE) models (55 56
3 Method
ALTER restructures a standard diffusion UNet into dynamic ensemble of temporal experts, each defined by distinct layer-wise pruning configuration applied to the shared backbone. As shown in Fig. 2 hypernetwork generates these binary pruning masks, and a lightweight router assigns each denoising timestep to appropriate expert. All components are jointly optimized in single-stage training process. During inference, the router dynamically activates expert-specific sub-networks conditioned on the timestep.
3.1 Preliminaries
Diffusion Process Diffusion models 2 12
capture the data distribution by learning to reverse a predefined noising process applied to the input data. The forward process of the standard diffusion gradually adds Gaussian Noise to the clean input Ti over T timesteps, generating a sequence of increasingly noisy latents {Ø, 1T with the assumption that æт approximates pure noise. The reverse
3

---

Hypemeeworrk H
Li
Ny Experts
Timestop Embedding (o:)
0010100
Expert G Generator
Temporal Router
total
T
UNet Eg
by
Figure 2: Overview of the ALTER framework. ALTER is a temporal-adaptive prrnning framework for diffusion models, where hypernetwork generates layer-wise pruning confi gurations for expert subnetworks and assigns each denoising timestep to a corresponding expert.
process is modeled by a neural network ca(2,, t, conditioned on the noisy sample 2t, the current timestep and optional context e (e.g., embeddings). The network trained denoise 2, into Tt-1 by predicting the added noise at the (-th timestep, following the training objective:
Leenoise N.0... |||c co( c))))
where Ty Vārzo + VI- agt, and ār are predefined noise schedule coefficients. To enable the denoising network adapt its behavior across different noise levels, the discrete timestep is converted into a continuous vector embedding E IR Damb, where Dems is the embedding dimension. This often achieved using sinusoidal positional encodings 61 or learned embeddings. The timestep embedding e is then integrated into various layers of the neural network by adding it to the hidden activations within blocks.
Layer-wise Pruning for Denoising Network. The denoising network eø is commonly UNet 622 631, featuring an encoder-decoder structure with skip connections. Layer-wise pruning in the UNet architecture refers to the selective omission of entire pre-defined computational blocks, i.e, residual layers or transformer modules. To formalize the definition of pruning, we represent the configuration of UNet with Nr, prunable layers using binary mask vector indicates that the i-th layer is pruned, and m; = indicates that it is retained.
{0, N,,, where m,
3.2 Temporal Expert Construction
ALTER employs a trainable hypernetwork Ha to generate layer-wise pruning masks for shared UNet backbone. These masks define N, expert configurations, where each expert is equivalent to a layer-wise-pruned sub-network. H3 consists two trainable components whose trainable parameters jointly form 4: (I) an Expert Generator G and (2) Temporal Router R.
Expert Generator. To generate Ne expert configurations, we employ Expert Generator G to produce a set of pruning masks {m,) E [o. 1] NoNN, where each m, [0. N serves as the configuration for the i-th expert,
(1....,Ne) .N). The generator takes as input set of frozen embeddings Z E IR N. NN X Pnpul, initialized with orthogonal vectors, and processes them through several multi-layer perceptrons (MLPs) to produce a matrix expert layer logits Lexpens E R R.N.NNG N2, To approximate binomial distributions while retaining differentiability during training, we apply the Sigmoll function (4 combined with the Straight-Througe Estimator (ST-GS) as the final layer on Texpents to obtain {m.).
Temporal Router:
To select an appropriate expert for the t-th timestep in the diffusion process, employ Temporal Router R which shares a similar network structure with the expert configuration generator. The temporal router takes the corresponding dre-defined timestep embeddings 6, R. Damc from the target UNet's 's timestep embedding mechanism as input, and maps it to the routing logits L mating Ne over the N₂ expert candidates. Similar to the expert configuration generator, apply

---

the Gumbel-Softmal function with the Straight-Through Estimator to the routing logits to obtain the final categorical selection vector {
By routing based on distinct timestep embeddings, the router enables expert specialization across different phases of the denoising trajectory, allowing the model to adaptively assign computation to the most suitable sub-network at each step.
Differentiable Simulation of Layer-wise Pruning. To simulate the effect of timestep-specific layer-wise pruning on the diffusion UNet during training, we modify the forward computation of each prunable layer 1 {1...., NE} as follows:
Yout
(1 (m²)i) 2in + (m²)) fi(z)),
(2)
where Tin is the input to layer I, fi(-) the original layer computation, and (mee): E [0, 1] denotes the pruning decision for layer / at timestep t. When (nxx): - 0, the layer is effectively skipped; when (m,), = 1, it remains fully active. This formulation allows faithful simulation of pruning behavior while preserving gradient flow, enabling end-to-end optimization. The use of the Straight-Through Estimator (STE) in conjunction with Gumbel- Sigmood and Gumbel-Softman ensures differentiable sampling for both the pruning masks and the routing selection vector. During inference, apply hard pruning by performing actual layer skipping whenever (m+)) = 0, thereby realizing computational efficiency through expert-specific subnetwork execution.
3.3 Optimization Strategy
ALTER, consisting of the shared UNet eo with parameters 0 and the hypernetork Ha with parame- ters 4, is jointly optimized via single-stage alternating training scheme. As detailed in Algorithm [ each training step alternates between the following two optimization phases:
Shared UNet Backbone Optimization Given the Ne expert contigurations M generated by the
current hypernetwork Ho. the goal of shared UNet optimization is to ensure each expert substructure achieves strong denoising performance on its assigned group of timesteps. Thus, the primary training objective the standard denoising loss Ldenoise computed over masked subnetworks that simulate the active expert at each diffusion timestep. To further improve performance and stabilize training under layer-wise pruning, we optionally employ knowledge distillation from the frozen pretrained teacher model éT. Specifically, we introduce an output-level distillation loss LoWKD, which encourages the student UNet cs = 6e to produce similar denoising outputs as the teacher, and a feature-level distillation loss CraaKD. which aligns intermediate representations between teacher and student. These auxiliary losses are defined as:
CouIDD [EI,, ts( LEeaKD E|
|| || (z,,
(x,,,),
(3)
where f+z and fs denote the k-th block feature activations from the teacher and student models, respectively. The total UNet loss combines the denoising objective and the distillation terms:
Lu Aenoise ddnnoise +
AOunDD + CteakD-CeaKKD
where AoatkD and AgenKD are hyperparameters that control the influence of each distillation term.
Hypernetwork Optimization. Based on the current shared UNet backbone, the updated hypernet- work must satisfy three principles: (1) Performance Preservation: the temporal expert selected for given denoising timestep should preserve the denoising capability depicted by the performance loss Epet mirrors the UNet objective Ly. is evaluated using the masks mi from the trainable H applied to the frozen UNet, ensuring H generates effective masks maintaining good generation performance.
(2) Structure Sparsity: the temporal sparsity specialization throughout the diffusion trajectory should approach the user-defined computational reduction target. A sparsity regularization loss Catio (aaio (ml) guides Ha to achieve a target overall pruning ratio p. This term utilizes a log-ratio matching loss to penalize deviations of the current effective sparsity ((m{) from p. The precise method for calculating
((m() based layer costs and mask activations is detailed in Appendix B.2 S(m,) is measure of
the active portion of the network's 's FLOPs under mask m/.- The loss defined as:
max(S(m{).p)) min((S(m))
Cratio (m)) = log
(5)
5

---

Algorithm ALTER Training Algorithm
1: Input: Training dataset Drain: UNet eq; Hypernetwork Ho; Total training steps T; Hypernetwork
training end step Tend' Loss coefficients A Abalance> A A
2: Initialization: Initialize UNet parameters 0; Initialize Hypernetwork parameters & (c.g., to
output all-active masks).
3: M < Hp-cval=True)
Initial mask configuration from H.p
4: for t= 1 to T do
5: Sample mini-batch from Deraini
6: if I Tend then
D Hypernetwork Update P Differentiable masks for batch 86
7:
mi f- Hs, train=True)
8:
Lpert +- --tenoise (#b( {m({)) + LouKD + AtemKD feautD Compute (ma)) and Lhaalance (m/);
D frozen
9:
LH 5 -pett= + Pratofrau + bbalacce
11:
Update & with V CHi
12:
M + Hg (eval-True)
D Update mask from new Ha
13:
end if
14:
Di UNet Update
15:
Select masks {me} for st from M:
16:
CLNE & Ldeeoise (ColiseeC((o (co(Sb, {m+])) + AOuKD + LreakD
> O frozen
Update with V VoLuNer
18:
end for
Output: A fine-tuned UNet eg and temporal mask decisions {m²)-
where is a small constant for numerical stability.
(3) Expert diversity: The router should promote diverse expert selection across timesteps to prevent mode collapse. A router balance loss Dralance encourages diverse utilization Ne experts [65] 66];
Dalance N
F.P.
(6)
where F; = [ Σ,,e T(argmax, (Las
routing ) i) is the fraction of samples in batch B assigned to
expert i, and P. = B E. (Sofimax L.surinn)): ; is the average router probability for expert i.
The total hypernetwork objective is
Lh Epett + Cratio Abulance Coalance
where Aratio and Analance are scalar coefficients. The differentiability of mi achieved by ST-GS and ST-GSmax, whose implementation details are discussed in Appendix B.3 After updating the hypernet, the mask configuration My for subsequent UNet training steps refreshed using the updated Hq. This alternating optimization strategy allows the UNet to adapt to the dynamic architectures from H4. while Ha learns to generate efficient and effective configurations. The hypernetwork would be trained until Tend is reached.
Such alternating training strategy enables the co-adaptation within ALTER: the UNet progressively recovers its denoising performance under various configurations of layer-wise sparsity, while the hypernetwork keeps refining the configurations and routing of specialized expert configurations tailored to the diffusion process.
4 Experiment
4.1 Experimental Setup
Implementation Details. We experiment on official pretrained diffusion model SDv2.1 4 to
demonstrate the effectiveness of our method. For training, we utilize a randomly sampled 0.3M subset of the LAION-Aestheics V2 (6.5+) (67. We prune the models 65% and 10 temporal experts with the loss weight Ayatio = 5. The bi-level optimization takes 32k steps with global batch size 64, where the hypernetwork is trained for epochs. All experiments are conducted on 2 A100 GPUs.

---

Table Comparison with BK-SDM and APTP on CC3M and MS- AIS-COCO for SD-v2.1. The images are generated at the default resolution 768 and then downsampled to 256.
CC3M
MS-COCO
Method
Steps
MACs
Latency (sX))
FID
CLIP
CMMD
MACs
Latency (8)()
FID
CLIP
CMMD
(G)LL
(0)
(1)
(4)
(GXL)
(4)
(1)
(U)
SDv2.1
4
25
1384,2
4.0
17.08
30.96
0.361
1384.2
4.0
14.29
32.17
0,537
SDv2. 4
20
1384.2
3.2
17.37
30.89
0.374
1384.2
3.2
14.46
32.08
0.532
BK-SDM-v2 (Base)
53
25
876.5
2.5
17.53
29.77
0.486
876.5
2.5
19.06
30.67
0.654
BK-SDM-v2 (Small) 53
25
863.2
2.5
18.78
29.89
0.453
863.2
2.5
20.58
30.75
0.645
APTP (0.85)
[8
25
1182.8
3.4
36.77
30.84
0.675
1076.6
3.1
22.60
31.32
0,569
APTP (0.66) 18
25
916.3
2.6
60.04
28.64
1.094
890.0
2.5
39.12
29.98
0.867
ALTTR (0.65) (Ours)
25
899.7
2.6
16.74
30.94
0.393
899.7
2.6
13.70
32.17
0.536
ALTER (0.65) (Ours)
20
899.7
2.1
17.14
30.92
0.392
899.7
2.1
13.89
32.18
0,533
ALTER (0.60) (Ours)
20
830.5
1.9
17.87
30.88
0,479
830.5
1.9
14.24
32.17
0.596
Datasets. We evaluate our propased method on two widely-used diffusion benchmarks: Conceptual Captions 3M (68) (CC3M) and MS-COCO [69]. We evaluate pruning methods using 14k samples in the validation set of CC3M and 30k samples from the MS-COCO 2014's validation split. We sample the images at the default 768 resolution and then resize them 256 for calculating the metries with the PNDM [22] sampler following previous works [53] 18]. To compare with previous cache method, we also experiment on the COCO 2017's 5k validation dataset.
Evaluation Metrics. We report FID (L) 70 and CMMD (L) [7] to assess image quality, CLIP (1) score (72 measure text-image alignment, the number of muutiply-accumullan operations (MACs) (L) as a proxy for computational cost, and wall-clock inference latency in seconds (Latency) (J). We also report the speedup of ALTER compared with the original model. The MACs and latency are measured with batch size in the A100 GPU platform.
4.2 Comparison Results
Comparison with Static and Sample-wise Dynamic Pruning Method. We benchmark ALTER against two representative pruning baselines: BK-SDM-v2
55 which employs static pruning, and APTP 18] which utilizes sample-wise -xxturs (MoE) approach conditioned on input prompts. As detailed in Table ALTER consistently demonstrates superior generative quality and competitive efficiency. Specifically, during a 25-step inference on MS-COCO, ALTER achieves leading FID of 13.70 and CLIP score of 32.17, which outperforms BK-SDM-v2 (Small) and APTP (0.66), while operating at comparable computational costs. Similar advantages for ALTER observed on the CC3M dataset. Remarkably, ALTER's 's efficiency allows for even stronger performance with fewer steps. With only 20 inference steps, ALTER still surpasses the 25-step performance of both BK-SDM-v2 and APTP, highlighting its superior trade-off between speed and quality. Furthermore, ALTER's ' performance at both 25 and even 20 steps is not only competitive with but even exceeds that of the original unpruned SDv2.1 model. These comprehensive results confirm that ALTER's 's strategy of routing masks by timestep, rather than relying on a single global pruning or solely on prompt-based cues, allocates model capacity more precisely to the varying demands of the diffusion process. Qualitative comparisons presented in Figure 3 visually affirm these findings, showcasing that images generated by ALTER exhibit a quality comparable to the unpruned SD-v2.1 model and display fewer artifacts than those produced by BK-SDM-v2
Comparison with Cache-Based Method. We further evaluate ALTER against DiP-GO [30], leading cache-based acceleration technique, on the MS-COCO 2017 validation set, with results detailed in Table 2 Cache-based methods primarily aim to reduce computation by reusing features from previous timesteps. While effective, they often operate on the original model architecture and may require careful tuning of parameters like cache rate [16] for optimal performance with different inference schedules. In contrast, ALTER's 's temporal pruning- schedule-agnostic model. Once the hypernetoor has determined the temporal expert structures and routing, the resulting dynamically pruned UNet can be deployed with any number of denoising steps without retraining specific adjustments, offering significant operational flexibility.
approach results an inference
7

---

out loving of blanket woar
stock of trusks and laggage wall
niew a wst lighthone
reno flacred pletare A browa roddy boar jerra whoel sitting the raddle moad
von diersed up and pusing for
v2.1
(4.0s)
BK-SDM-v2
(Small)
(2.5s)
ALTER (ours)
(2.1s)
Figure 3: A qualitative comparison with original SDv2.1 and BK-SDM-Small. SDv2.1 and BK- SDM-Small adopt the 25-step PNDM while our method adopts the 20-step inference.
Table 2: Comparison with DiP-GO on the MS-COCO 2017 validation set on SD-2.1. MACs here is the total MACs for all steps.
Method
MACs(L)
Speedup(1)
CLIP (1)
FID-5K (L)
SDv2.1 (50 steps) 4
38,04T
1.00x
31.55
27.29
SDv2.1 (20 steps) 4
15.21T
2.49
31.53
27.83
DiP-GO (0.7) 30
11.42T
3.02
31.50
25.98
ALTER (0.65) (Ours) (20steps)
9.89T
3.64x
31.62
25.25
SDv2.1 (15 steps)
11.41T
3.23x
30.99
27.46
DiP-GO (0.8) 30
7.6IT
3.81
30.92
27.69
ALTER (0.65) (Ours) (15steps)
7.42T
4.37
31.19
26.58
For 20-step inference process, ALTER significantly reduces total MACs to 9.89T, which is substantial improvement over the original SDv2.1 and achieves a 3.64x speedup compared the 50-step SDv2.1 baseline, while delivering generative quality that is comparable or even superior (e.g., FID-5K 25.25 for ALTER vs. 27.83 for SDv2.1 20-steps and 27.29 for SDv2. 50-steps). ALTER (20 steps) is also more computationally efficient than DiP-GO (0.7) and demonstrates superior performance scores. When the inference step further reduced to steps, ALTER maintains strong generative quality and achieves better performance compared with the I 15-step SDv2. It again outperforms DiP-GO (0.8) in both efficiency and generative quality. This robust ability to adapt to various inference steps without re-tuning demonstrates ALTER's suitability for real-time applications.
4.3 Abalation Study
Ablation Study on ALTER's Key Components. To evaluate the contributions of ALTER's key design components, the usage of multiple specialized experts, the temporal router and the single-stage joint optimization strategy, we conduct comprehensive ablation study. As shown in Table we compare the full ALTER framework against three variants: (1) "Static", which employs static global pruning pattern across all timesteps; (2) "Manual", which introduces multiple experts but assigns them to predefined fixed timestep intervals instead of employing learnable router; (3) "TwoStage", which incorporates ALTER's 's architecture but adopts two-stage optimization approach where the hypernetwork is trained first, followed by fine-tuning the UNet.
The results clearly demonstrate the benefits of each component. The "Static" variant, lacking specialized temporal handling, expectedly exhibits the worst performance. Transitioning to the "Manual" approach, which introduces multiple experts for different fixed temporal intervals, yields significant improvement (e.g., FID drops from 19.03 to 17.91 CC3M). This demonstrates the importance of having specialized experts for different phases of the denoising process. Further introducing the learnable timestep router in ALTER provides additional performance boost over

---

Table 3: Ablation study on pruning and optimization strategies.
Variant
Multi Expert
Timestep Router
Joint Training
CC3M
MS-COCO
FID (1)
CLIP (1) CMMD (4)
FID (U)
CLIP (t)
CMMD (L)
Static
19.03
30.75
0.397
15.35
32.01
0.544
Manual
x
v
17.91
30.82
0.427
14.13
32.08
0,571
TwoStage
x
17.65
30.88
0.401
14.45
32.02
0.539
ALTER
17.14
30.92
0.392
13.89
32.18
0.533
Table 4: Comparison with TinyFusion and DyDiT for DiT-XL/2 on ImageNet of 256x256 resolution.
Method
Speedup
IS(
FTD(L)
sFID(L)
Precision(()
Recall(
DiT-XL/2 7 (28-layer)
1.00x
277.00
2.27
4.60
0.83
0.57
DyDiT-XL S4) (A=0.5)
1.72x
248.03
2.07
4.56
0.61
TinyFusion 7 (14-layer)
1.96x
234.50
2.86
4.75
0.82
0.55
ALTER (Ours) (avg. 14- l4ylry
1.92x
254.29
2.36
4.63
0.82
0.58
Experts Pruning Visualization
4TD
Dowm
Ma
Nandor Tltpett
Layees
MS-COCO
Router Visualtaatio: Steps Rosted to Expens
Vimamrape
(6)
Figure 4: ALTER (0.65)'s temporal experts and router behavior. (a) Visualization of pruning patterns for N₂ = 10 experts. (b) Visualization of timestep-to-expen routing dynamics. (c) Ablation results for the number of experts CC3M and MS-COCO 2014.
the "Manual" variant's coarse-grained, fixed assignments, reducing FID) to 17.14 on CC3M and 13.89 MS-COCO, which underscores the benefits of dynamically learned, fine-grained temporal routing mechanism. Finally, the comparison between ALTER and the "TwoStage" optimization variant shows the critical advantage our single-stage joint optimization strategy, which facilitates the co-adaptation of UNet's generative ability with the hypernetwork, leading to superior generative quality across all evaluated metries on both datasets. These ablation experiments collectively validate the effectiveness and contributions of ALTER's key design choices.
Analysis of Temporal Experts To understand the characteristics of the temporal experts learned by ALTER, we analyze their pruning patterns, routing dynamics, and the impact of number of experts, shown in Figure 4 The visualization of pruning patterns for the N. 10 experts (Fig 4a)) demonstrates structural diversity. Different experts adopt distinct pruning patterns across the UNet blocks and layers. For instance, some experts being heavily pruned (e.g., experts 3,8,9) while some retain greater capacity (e.g., experts 7). This learned structural differentiation allows for temporal specialization. The router dynamics Fig. (b) illustrates how these specialized experts are allocated across different denoising timesteps. We observe that specific experts are predominantly active during certain phases of the diffusion process. Furthermore, the ablation study on the number of experts highlights the importance of an adequate expert number for achieving optimal performance,

---

as shown in Fig. C). Our chosen configuration of N = 10 experts consistently yields the best or near-best FID and CLIP scores on both CC3M and MS-COCO datasets. Performance tends to degrade with fewer experts, likely due to insufficient capacity for fine-grained temporal specialization across the entire denoising process. Conversely, increasing to N₂ = 15 experts does not yield further improvements and can sometimes slightly degrade performance, potentially due to challenges in effectively training a larger set of specialized experts or increased complexity in the routing decisions.
Generality to DiT. To assess generality beyond U-Nets, we apply ALTER DiT-X1/2 back- bone 7 and evaluate on ImageNet [74) at 256 x56 using 250-step DDPM sampler [2). As shown Table 4 our method achieves a 1.92 speedup while maintaining an FID score (2.36) comparable to the full model (2.27). With approximately half the layers active on average, our method signifi- cantly outperforms static pruning (TinyFusion (73). especially on key quality metries like IS (254.29
vs. 234.50) and FID (2.36 vs. 2.86). When comparing to dynamic methods like DyDiT (54] ALTER brings higher speedup. It is important to note that our approaches are orthogonal and complementary. ALTER performs coarse- -raiined layer-level pruning, while DyDiT performs fine- tneeggrined pruning within each block. This empirically confirms that our framework effective beyond U-Nets and is applicable to modern Transformer-based architectures.
5 Conclusion
In this paper, presented acceleration framework ALTER that transforms the diffusion model to a mixture of efficient temporal experts by employing a trainable hypernetork to dynamically generate layer pruning decisions and manage timestep routing specialized, pruned expert sub- networks throughout the ongoing fine-tuning of the UNet. This timestep wise approach overcomes the limitations of static pruning and sample-wise dynamic methods, enabling superior model capacity utilization and denoising efficiency. The experiments under various settings and ablation analysis demonstrate the effectiveness of ALTER, which could achieve 3.64x speedup the original SDv2. with 35% sparsity.
References
[1] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[3] Sungbin Lim, Eunbi Yoon, Tachyun Byun, Taewon Kang, Seungwoo Kim, Kyungjae Lee, and Sungjoon Choi. Score-based generative modeling through stochastic evolution equations in hilbert spaces. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[5] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint a7Xiv:2204.06115,, 1(2):3 2022.
[7] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[8] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 18208-18215 2022.
10

---

[9] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 6007-6017, 2023.
[10] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pages 3836-3847, 2023.
[11] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Křrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-225 10, 2023.
[12] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[13] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[14] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.
[15] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005112 2022.
[16) Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 15762-15772, 2024.
[17] Yu Xu, Fan Tang, Juan Cao, Yuxin Zhang, Xiaoyu Kong, Jintao Li, Oliver Deussen, and Tong-Yes Yee Lee. Headrouter: training-free image editing framework for mm-dits by adaptively routing attention heads. arXiv preprint arXiv:2411.1500 2024.
(18) Alireza Ganjdanesh, Reza Shirkavand, Shangqian Gao, and Heng Huang. Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models. In The Thirteenth International Conference Learning Representations, 2025.
[19] Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, and Haonan Lu. Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models. arXiv preprint arXiv:2404.1 11098, 2024.
[20] Thibault Castells, Hyoung- Kyu Song, Bo- Bo-Kyeong Kim, and Shinkook Choi. Ld-pruner: Effi- cient pruning of latent diffusion models using task-agnostic insights, 2024.
[21] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models, 2022.
[22] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
[23] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator, 2023.
[24] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models, 2023.
[25] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
[26] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation guided diffusion models, 2023.
[27] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023.

---

[28] David Berthelot, Arnau Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation, 2023.
[29] Zhaoyang Lyu, Xudong XU, Ceyuan Yang. Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process, 2022.
[30] Haowei Zhu, Dehua Tang, Ji Liu, Mingjie Lu, Jintu Zheng, Jinzhang Peng, Dong Li, Yu Wang. Fan Jiang, Lu Tian, et al. Dip-go: A diffusion pruner via few-step gradient optimization. Advances in Neural Information Processing Systems, 37:92581 -926444,,, 2024.
[31] Senmao Li, taihang Hu, Joost van de Weijer. Fahad Khan, Tao Liu, Linxuan Li, Shiqi Yang, Yaxing Wang, Ming-Ming Cheng, and jian Yang. Faster diffusion: Rethinking the role of the encoder for diffusion model inference. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[32] Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cacher Accelerating diffusion transformer via layer caching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[33] Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang. Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 6211-6220, 2024.
[34] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1972-1981, 2023.
[35] Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for diffusion models. Advances in neural information processing systems, 36:48686-48688, 2023.
[36] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint urXiv:2410. 10629, 2024.
[37] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint 15000:151,.00149, 2015.
[38] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks. arXiv preprint 1902:01902209574 2019,
[39] Tianyi Chen, Luming Liang, Tianyu Ding, Zhihui Zhu, and Ilya Zharkov. Otov2: Automatic, 2023.
generic, user-friendly. arXiv preprint
[40] Runxue Bao, Bin Gu, and Heng Huang. An accelerated doubly stochastic gradient method with faster explicit model identification. In Proceedings of the 31 ACM International Conference on Information& & Knowledge Management, pages 57-66, 2022.
[41] Runxue Bao, Xidong Wu, Wenhan Xian, and Heng Huang. Doubly sparse asynchronous learning for stochastic composite optimization. In Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), pages 1916-1922, 2022.
[42] Shangqian Gao, Zeyu Zhang, Yanfu Zhang, Feihu Huang, and Heng Huang. Structural align- ment for network pruning through partial regularization. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 17402-17412, October 2023.
[43] Shangqian Gao, Junyi Li, Zeyu Zhang, Yanfu Zhang, Weidong Cai, and Heng Huang. Device- wise federated network pruning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12342-12352 June 2024.
12

---

[44] Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon, 2017.
[45] Sejun Park, Jacho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: far-sighted alternative of magnitude-based pruning, 2020.
[46] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 00323-10337. PMLR. 2023.
[47] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305. 18403, 2023.
[48] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
[49] Gongfan Fang, Xinyin Ma, and Xinchao Wang- Structural pruning for diffusion models. In seirth- Conference on Neural Information Processing Systems, 2023.
[50] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys. Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model mobile devices within two seconds. Advances in Neural Information Processing Systems, 16:20662--0678, 2023.
[51] Xingyi Yang. Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim, 2022.
[52] Xidong Wu, Shangqian Gao, Zeyu Zhang, Zhenzhen Li, Runxue Bao, Yanfu Zhang, Xiaoqian Wang. and Heng Huang. Auto-train-once, Controller network guided automatic network pruning from scratch. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern June 2024.
Recognition (CVPR), pages 16163-16173,
[53] Bo-Kyeong Kim, Hyoung-Kyy Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion. arXiv preprint arXiv:2305.15799 2023.
[54] Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. In The Thirteenth International Conference Learning Representations, 2025.
[55] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
[56) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 240.000800 2024.
[57] Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts, 2024.
[58] Shangqian Gao, Ting Hua, Reza Shirkavand, Chi-Heng Lin, Zhen Tang, Zhengao Li, Longge Yuan, Fangyi Li, Zeyu Zhang, Alireza Ganjdanesh, Lou Qian, Xu Jie, and Yen-Chang Hsu. To- moe: Converting dense large language models to mixture-of-experts through dynamic structural pruning, 2025.
[59] Alireza Ganjdanesh, Yan Kang, Yuchen Liu, Richard Zhang. Zhe Lin, and Heng Huang. Mixture of efficient diffusion experts through automatic interval and sub-network selection. In European Conference Computer Vision, pages 54 Springer, 2024.
[60] Haotian Sun, Tao Lei, Bowen Zhang, Yanghao Li, Haoshuo Huang, Ruoming Pang, Bo Dai, and Nan Du. EC-DIT: Scaling diffusion transformers with adaptive expert-choice routing. In The Thirteenth International Conference on Learning Representations, 2025.
13

---

[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[62] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[63] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[64] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 16..:6/1111111 144, 2016.
[65] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS Jhard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations, 2021.
[66] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[67] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models, Advances in neural information processing systems, 35:25278 -222944. 2022.
[68] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemnymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long 2018.
Papers), pages
[69] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[70] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[71] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9307-9315, 2024.
[72] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08711, 2021.
[73] Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. arXiv preprint arXiv:2412.01199 2024.
[74] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[75] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308. 3432, 2013.
(76] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. 2019.
14

---

[77] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 12555,, 2022.
arXiv preprint
[78] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation.
Advances Neural Information Processing Systems, 36: -6:15503-15935, 2023.
[79] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint 22ti:::300.9934444 2023.
[80] Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4599-4603, 2023.
15

---

A Limitations
While ALTER employs joint optimization for the co-adaptation of the U-Net and hypernetwork, final fine-tuning stage for the dynamically pruned U-Net is still found to be a necessary step after Tend is reached. This final tuning phase is crucial for fully recovering any potential performance degradation incurred during the temporal-wise structure pruning and for allowing the U-Net to optimally adapt to its newly defined sparse, dynamic structure, thereby maximizing its generative quality.
B
More Details of ALTER
This appendix provides further details on the architectural components of our hypernetwork Hą. the calculation of sparsity metrics, the implementation Gumbel-Sigmod and Humbel-Softmaa with straight-through estimation, and the handling of dynamic masks.
B.1 Detailed Hypernetwork Architectures
The hypernetwork H. consists of main modules: the Expert Structure Generator and the Temporal Router. Their architectures are detailed in Table 5 and Table 6 We use Dispur for the dimensionality of the learnable embeddings for the Expert Structure Generator, Despert for hidden layer, D for the UNet's s timestep embedding dimension, and Drouter for the Temporal Router's hidden layer. Specific values used in our experiments are provided in Appendix
Table 5: Architecture of the Expert Structure Generator module in Hp. Processes Ne x N2 input embeddings to produce N₂ x N₁, logits.
Component
Details
Input
Frozen Embeddings
IR No x xN x Dapu (Initialized orthogonally)
Hidden Layer
Linear( (Drar Input + D expent + LayerNorm(Dexpoe - ReLU (neeat + no bias)
Output Layer
Output
Logits Lexpens E N. (Squeeze last dim)
Table 6: Architecture the Temporal Router module (Brouter) in Hę. Processes each timestep embedding ez to produce Ne routing logits.
Component
Details
Input
Timestep Embedding e E IR. Demb (from Eximesteps E IR Tiuw Demb)
Hidden Layer
Linear( Demb Drouter) + LayerNorm( Drowter) > ReLU Linear( Drouter + Ne. no bias)
Output Layer
Output
Routing Logits L, routing E R. (for a single ex) (Total output L rouling 1uTi XNe)
B.2 Calculation of Current Sparsity
The current effective sparsity S(m,) used the sparsity regularization term Craiio (Equation 5p. quantifies the active computational cost relative to the total potential cost. It is calculated based on the soft (differentiable) pruning masks m generated by H4 during its training phase. Let (mç)) be the soft mask value for the (-th prunable layer and et be the pre-caleulated computational cost (e.g., FLOPs) of that layer. The current sparsity S(m)) is the weighted average of these mask values, normalized by the total cost of all prunable layers:
Ne (m!)
S(m()
(8)
ENr a
16

---

This formulation ensures that S(m,) represents the fraction of the total computational cost that remains active under the current soft mask my. The per-layer costs сj are typically profiled once from the pretrained full UNet architecture.
B.3 ST-GS & ST-GSmax Implementation Details
To enable differentiable sampling of discrete architectural decisions, we employ ST-GS and ST- GSMax techniques.
ST-GS Sampling. For layer pruning decisions within the Expert Structure Generator, logits ( expoosss) are converted to binary masks. We sample Gumbel noise ~ Gumbel 1). Given logit L, a sampling temperature Tg 0.4, and an offset bg- 4, the soft decision yg computed:
Ug L + 9 + bg
Tg
where (.)) is the sigmoid function. To obtain hard binary decision (Mexpens while maintaining differentiability for training, we use STE [75]: (Mespens)e. = round (gg) During backpropagation, the gradient is passed through Vy if the rounding operation were the identity function for values not exactly at 0.5, effectively using V V(round(yg)) R Vyg. This results in a tensor of '0's and '1's for the forward pass, forming the pruning masks.
ST-GSmax Sampling. For temporal routing decisions, given vector of routing logits I., T.yuting R&,, temperature T, = 0.4, an offset 6, O, and vector of Gumbel noise G E IR (components gj sampled independently). the soft Gumbel-Softtax probabilities yr- E [0, 1]% are:
exp(((I,ouiing + 9j + br)/Tr)
(ur) Σ exp(((I., (T..routig) ) + + br)/T,)
To obtain hard one-hot selection vector s, for the forward pass while enabling gradient flow for training. STE is applied. Determine the index of the maximally activated expert: j argmax, (1n)) Create hard one-hot vector (s! 1==ド". The output used in computation, which enables STE, is 8: (shand yト dde..cttch() + Mr- This makes 8; a one-hot vector in the forward pass, while its gradient in the backward pass taken with respect to the soft probabilities yr-
B.4 Dynamic Mask Handling
The dynamic pruning masks, central to ALTER's i efficiency, are generated and applied as follows. For each timestep f, an effective mask m, E [0, N is composed from the Ne expert structure masks Mespens and the per-timestep routing selection deterministic binary masks (m..ndd forming the set MuNe) are derived by setting the hypernetwork Ha evaluation mode. For training H& itself, differentiable soft versions of these masks (m)) are utilized.
When training the UNet or during inference,
The UNet architecture has specific block structure. The generated flat mask vector (whether mard or m²) of length N₂ is mapped to the corresponding prunable layers within the UNet using a utility function. This ensures that the correct mask component (m²) is applied the 1-th prunable layer fi- The application of this component follows Equation from the main text: Tour = (1 - (m²))) Tin (m,)) (zin). This general mechanism applies all designated prunable layers. It is noteworthy that in UNet architectures, particularly in the decoder pathway, layers fi often process inputs that are result of concatenating features from a previous layer in the same pathway with features from an earlier encoder layer via a skip connection. Our layer-wise wise pruning applies directly to such layers ft: fi is skipped due (m²): its entire operation, including its specific handling of any skip-connected features, is bypassed.
C More Implementation Details
Here we provide additional details regarding our experimental setup, training details and hyperparam- eter configurations for ALTER. All models are trained or fine-tuned starting from official pretrained
17

---

050, Peats 4.0
4.5
p Aua 5.0
p- Acau
5.
Less
2.5
0.5
24.
5K
Trerationss
Itabons
(a) Loss Lu
(b) Loss L
0.5
Dattnos
Loos Cretts
12.5
0.4
122
03
0.1
26
6c
3k
3k
Theraooon
(c) Loss Aalance
(d) Loss Fatio
Figure 5: The training dynamics given different ratios p and Aratio weights.
SD-2.1 4 checkpoints. The training conducted at resolution of 512 512 pixels using a randomly
sampled subset of 0.3M images from the LAtON-Aesthetics V2 dataset (rated 6.5+) [67]. The total optimization process spans 7 32, 000 steps with global batch size of 64, distributed across 2 A A100 GPUs (32 samples per GPU). The hypernetwork H& is actively trained until Tend which is set to 2 epochs. We employ N. 10 temporal experts and sparsity ratio p 0.65 for our main experiments. We use the AdamW optimizer [76] for both the U-Net parameters 0 and the hypernetwork parameters 4. A linear learning rate warm-up applied for the first 250 iterations for both optimizers. After the warm-up up phase, constant learning rates are used: x 10-5 for the U-Net and 7 x 10-5 for the hypernetwork. For the GS function used in the Expert Generator, the temperature Tg set to 0.4 and the offset by to 4.0, which initializes expert masks to all-ones, promoting layer activity at the start of training and preventing an overly detrimental impact on the U-Net from aggressive initial pruning. For the GSmax function in the Temporal Router, the temperature is 0.4 and the offset b, is 0.0. Since the denoising loss L'demoise tends to result unstable optimization and slow convergence. The loss weights for the objective functions are set to Adenose Le 4, AoutKD AfeatD 1balance=1.0 and Aratio = 5.0. For hypernetorr''s ' implementation, we set D imput = 64, Dexpent = 256, Dromer = 64, and the Demb is depedent the pre-trained official SDv2. model. For generating samples for evaluation, we use classifier-free guidance (CFG) (77) with the default scale of the respective SDv2.1 checkpoint (e.g.. 7.5). Image generation is performed using the PNDM sampler (22), following practices in prior works for consistency.
D More Experimental Results
D.1 Training Dynamics
In Fig. 5 we visualize the training dynamics for differentt pruning ratios p and their associated Avatio- highlighting the co-adaptation between the shared UNet backbone and the hypernetwork. As shown in Fig. (5)a), the UNet loss Cy generally follows two-phase pattern: it first increases, especially under high sparsity targets (e.g., p = 0.50), and then gradually decreases. This early rise reflects transient mismatch, as the hypemetwork begins sample diverse pruning structures across timesteps while the shared UNet parameters have not yet adapted to the corresponding architectural variations. Over time, however, Lu steadily declines, indicating that the shared UNet progressively learns to meet the performance demands imposed by wide range of expert sub-structure configurations at different timesteps. The final converged LV remains slightly higher under more aggressive pruning configurations, reflecting the expected trade-off between model sparsity and generative fidelity.
18

---

Table Comparison of Image Reward and HPSv2 metrics. The images are sampled with 25-step inference.
Method
MACs(G)()
Image (eward(1)
HPSv2(1)
SDv2.1
1384.2
0.1139
24.47
SK-SDM-V (2-Base
876.5
-0,0365
23.97
APTP (0.66) (COCO)
916.3
-0,8706
19.08
APTP (0.66) (CC3M)
916.3
-1,2226
18.19
ALTER (0.65) (Ours)
899.7
0.1494
24.54
ALTER (0.60) (Ours)
830.5
0.1487
24.36
Table 8: Comparison of SSIM and PSNR metrics.
Method
Steps
MACs (G)(L)
Latency (s)()
CC3M
MS-COCO
SSIM (1)
PSNR (1)
SSIM (1)
PSNR (1)
SDv2.1
25
1384.2
4.0
SDv2.1
20
1384.2
3.2
0.3529
13.71
0.1582
10.24
BK-SDM-v2 (Base)
25
876.5
2.5
0.1208
9.42
0.1551
9.47
APTP (0.66)
25
916.3
2,6
0,0782
8.99
0.0730
8,83
ALTER (0.65) (Ours)
25
899.7
2.6
0.1233
9.90
0.3261
12.81
ALTER (0.65) (Ours)
20
899.7
2.1
0.1247
9.95
0.2484
11.61
ALTER (0.60) (Ours)
830.5
1.9
0.1224
9.94
0.1898
10.73
In contrast, the hypernetwork's s loss Lu (Fig- exhibits steady and consistent decline across all configurations, indicating that H is effectively optimizing its composite objective, which includes the UNet performance term Lprt and the associated regularization components. This consistent improvement suggests that the hypernetwork progressively discovers better expert structures and routing strategies alongside the UNet's parameter update, while gradually approaching the desired computational reduction target.
The regularization losses further elucidate this process. The sparsity loss Cratoo (Fig. 5 d)) decreases smoothly over time, confirming that the hypernetoork successfully aligns the average computational cost of selected experts with the target sparsity level The convergence rate is influenced by both the sparsity target and the loss weight Araioo Similarly, the router balance loss Loalance (Fig. 5ic) shows downward trend, indicating that the temporal router gradually learns to distribute computation evenly across experts. Notably, under the most aggressive pruning setting (p 0.50), Chatance initially spikes, suggesting greater challenge in achieving balanced routing when sparsity constraints are tighter.
This intricate anterplay-weee Lи steadily improves while Ly undergoes transient perturba- hion-highlighhs the bi-level nature of ALTER's optimization process, which enables the emergence of efficient, dynamically structured expert configurations.
This intricate interplay, where LH steadily improves while Ly overcomes initial perturbation, highlights the bi-level nature of the optimization problem that ALTER navigates to achieve its efficient, dynamically structured experts with high performance preservation.
D.2 More Evaluation Metries
Reward and HPSv2 We benchmark models using advanced preference metrics, Im- ageReward (78) and Human Preference Score v2 (HPSv2) (79 in order to provide a more compre- hensive evaluation of generation quality. These metrics
perceptual quality and human aesthetic preference. As shown om Table 7 our method ranks highest on both metrics, even outperforming the original 1Dv2.1. confirming that efficiency gains do come at the cost of visual quality.
SSIM and PSNR We provide the SSIM and PSNR to show the deviation of different methods from the original SDv2.1 25-step inference. shown in Table g our method ALTER not only matches the MACs of prior work but also achieves significantly higher SSIM and PSNR scores,
19

---

Table Scalability with token reduction method ToMeSD on the MS-COCO 2017 validation set.
Method
Steps
ToMe r%
Speedup(1)
CLIP (1)
FID-5K (1)
SDv2.1
50
0
1.00x
31,55
27.29
SDv2.1
20
0
2.49x
31.53
27.83
ALTTE (0.65) (Ours)
20
0
3.64x
31.62
25.25
20
40
4.93x
31.48
25,93
To
50
5.75x
31.34
26.89
20
60
6,54x
31.24
27.96
Table 10: Ablation study on the sparsity ratio The images are sampled with 20-step inference.
CC3M
MS-COCO
Method
MACs (G)()
Latency (s)())
FID
CLIP
CMMD
MACs
Latency (s)
FID
CLIP
CMMD
(L)
(t)
(4)
(GXL)
(U)
(1)
(L)
SDv2.1
1384.2
3.2
17.37
30.89
0.374
1384.2
3.2
14.46
32.08
0.532
ALTER (0.65)
899.7
2.1
17.14
30.92
899.7
2.1
13.89
32.18
0,533
ALTER (0.60)
830.5
1.9
17.87
30.88
0.479
830.5
1.9
14.24
32.17
0.596
ALTER (0.55)
761.3
1.8
18.27
30.87
0.563
761.3
1.8
14.59
32.16
0.679
ALTER (0.50)
692.1
1.6
18.99
30.74
0.612
692.1
1.6
14.70
32.05
0.727
which demonstrates that our approach better preserves the generative characteristics of the original model.
D.3 Generality to DiT
To assess generality beyond U-Nets, apply ALTER DiT-XL/2 backbone and evaluate ImageNet [74] at 256x256 using a 250-step DDPM sampler (2).
As shown in Table 4 our method achieves 1.92x speedup while maintaining an FID score (2.36) comparable to the full model (2.27). With approximately half the layers active on average, our method significantly outperforms static pruming (TinyFusion (73)), especially on key quality metrics like IS (254.29 vs. 234.50) and FID (2.36 vs. 2.86). When comparing to dynamic methods like DyDiT (44, ALTER brings higher speedup. is important to note that our approaches are orthogonal and complementary. ALTER performs coarse- sranseegriined layer-level pruning, while DyDiT performs fine-grained pruning within each block. This empirically confirms that our framework is effective beyond U-Nets and is applicable to modern Transformer-base architectures.
D.4 Scalability with Other Accelerators
ALTER's 's design makes it orthogonal to and compatible with other diffusion acceleration methods such as diffusion cache and token reduction. We conduct an experiment combining ALTER with a token reduction method ToMeSD 80 to demonstrate ALTER's scalability. For this combination, we follow the default setting of ToMeSD. As shown in Table 9 we boost the total speedup from 3.64x impressive 6.54x with only minor, controllable trade-off in performance scores. This empirically validates the excellent scalability of our method.
D.5 Ablation Study of Sparsity Ratio
We investigate the impact of varying the target sparsity ratio p on ALTER's performance. Results reported in Table under the unified 20-step inference setting, showing clear trade-off between sparsity and generation quality. p decreases from 0.65 to 0.50, performance metrics degrade gradually. Notably. ALTER variants with p = 0.65 and = 0.60 match the generative quality of the original SDv2.1 baseline while significantly reducing computational cost. Even the most aggressive setting, ALTER (0.50), maintains competitive performance (e.g., FID of 14.70 and CLIP score of 32.05 on MS-COCO) with the highest efficiency gains. These results highlight ALTER's flexibility and its ability to balance quality and efficiency under different sparsity budgets.
20

---

Table 11: Ablation study on different components of Ly for the "Static" Variant.
CC3M
MS-COCO
Lu
FID (L)
CLIP (1)
CMMD (4)
FID (
CLIP (T)
CMMD (L)
Laenaise
23.76
29.93
0.531
21.57
30.91
0.709
Liennsee + LOuKD
20.15
30.54
0.424
15.98
31.79
0.549
Lienniee LouKn LLeaRD
19.03
30.75
0.397
15.35
32.01
0.544
Table 12: Ablation of post- Tnd fine-tuning.
Method
Steps
CL)P(())
FID-SK(L)
SDv2.1
20
31.53
27.83
ALTER (0.65) (Tena)
20
31.54
25.98
ALTER (0.65) (Tual)
20
31.62
25.25
ALTER (0.65) (Tend)
15
31.05
27.16
ALTER (0.65) (Twal
15
31.19
26.58
D.6 Ablation Loss Components of Lu
We investigate the impact of incorporating knowledge distillation losses into the UNet's training objective Lo for the "Static" variant. As shown in Table [1] introducing the distillation of the output and block features are able to enhance the performance of the statically pruned model, which guides choice of loss components.
D.7 Ablation Final Finetuning
As the results Table I2 demonstrate, the model's performance after Tend already very strong, achieving the majority of the final quality. The final fine-tuning provides consistent and important boost across both metrics and step counts, further improving both FID and CLIP scores.
D.8 More Qualitative Results
We provide more qualitative results compared with the 25-step original SDv2.1 and BK-SDM-V2- Small. As shown in Fig. 6 ALTER exhibits quality comparable to the unpruned SD-v2.1 model and displays fewer artifacts than those produced by BK-SDM-v2.
21

---

SD v2.1 (4,0s)
BK-SDMMy2-Small (2.56)
Ours (2.18)
A bird is perched top branch over a
portrait horse drinking water
mountaii lake is located close
A microwave sitting thelf
A herd zebra slanding next each other while drinking
the docks of the fishing town tourist destination
sondwich sitting in a bowd
a beautiful coral reef thrives
Figure 6: More qualitative results compared with original SDv2.1 and BK-SDM-Small SDv2.1 and BK-SDM-SE SDM-Small adopt the 25-step PNDM while method adopts the 20-step inference.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly reflect the main contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work. Please refer to Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
23

---

Justification: Our propositions are mainly based on experiments and empirical results which do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided all the implementation details of model design, network training and inference in Section 4. and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code will be released upon acceptance. We believe that the results easy to reproduce with the details of hypernetwork provided Appendix B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
aablic/guudees/Coeeetbii.ss
casssiionoolyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtp:/cccpblcgggguuueeeeee
Stteee///ooSSSeeissSioi.a www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the details of experimental setup model training and testing in Section 4. Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experiments conducted in our paper do not involve the use of error bars or statistical significance analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the detailed utilization the GPU resources, training batch size and iterations in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eeurpplt http://www.....pp.aaa//cccaaaaaa
Answer: [Yes]
Justification: Ethics.
Our research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: We do not believe that enhancing and accelerating diffusion models will inherently result in direct negative social impact, nor we anticipate any significant social consequences from this optimization.
26

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets used in the paper, are properly credited. Additionally, the license and terms of use associated with these assets are explicitly mentioned.
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
27

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.paperssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips /att::///earrrpsssc//oooeereeee/ /ct/////oeeeeeccc//22////.. for what should or should not be described.
29


Reviews:
[{"review_id": "Du75NtcfUj", "reviewer": "Reviewer_jkuL", "summary": "The paper proposes a novel method, **ALTER**, which enhances the efficiency of diffusion models by introducing an innovative combination of layer pruning and temporal expert routing. The method aims to reduce computational overhead during inference without sacrificing generative quality. Key contributions include the unification of layer pruning, expert routing, and model fine-tuning into a single-stage optimization framework. This results in a significant reduction in the MACs and inference time while maintaining the visual fidelity of generated images. The experiments show that ALTER outperforms existing methods, such as static pruning and sample-wise dynamic pruning, on standard datasets.", "questions": "- In Tables 1 and 2, the authors use FID, CLIP, and CMMD to evaluate the generation quality. While these are commonly used metrics, considering that this paper focuses on the acceleration of diffusion tasks, the information deviation before and after acceleration should also be considered as a metric. Therefore, adding SSIM and PSNR to measure the generation deviation from the pre-trained model could further enhance the evaluation of generation quality.\n    \n- In principle, the ALTER method has good scalability. It is orthogonal to the current Diffusion Cache and token reduction methods. In the future, further validation on its scalability could be conducted, which would make it even more comprehensive.\n    \n- Additionally, I have a small suggestion. I recommend changing \"Temporal Expert Routing\" to \"Step Expert Routing,\" as it might be easier to understand. The term \"Step\" could have a higher acceptance rate among those familiar with diffusion models, although \"Temporal\" is also fine.\n    \n- Some symbols in the paper are used redundantly. For example, the symbol t represents both the timestep of the diffusion model and the training step in Algorithm 1. It would be helpful to standardize the notation throughout the paper.\n    \n- Overall, the idea presented in this paper is very interesting. If the concerns mentioned above can be addressed, I would consider raising my score.", "limitations": "The authors have adequately addressed the limitations of the model.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The integration of layer pruning with temporal expert routing is a novel approach, offering flexibility and better utilization of model capacity across different time steps in the diffusion process.\n    \n- The method achieves a substantial speedup (e.g., 3.64x faster than the original Stable Diffusion v2.1) while maintaining competitive generative quality, making it a strong candidate for deployment in resource-constrained environments.\n    \n- Extensive experiments across various datasets (e.g., CC3M, MS-COCO) demonstrate the effectiveness of the proposed method. The paper also includes ablation studies, showing the contribution of key components like the temporal router and joint optimization.\n    \n- The paper is well-organized and easy to understand.\n\n**Weaknesses**\n\nPlease refer to the Questions part.", "was_revised": true, "final_justification": "The authors have addressed all previously raised concerns thoroughly. The evaluation has been enhanced by incorporating SSIM and PSNR, offering a more complete assessment of the generation deviation introduced by acceleration. This addition strengthens the empirical support for the proposed method.\n\nThe scalability of the ALTER framework has been further validated, confirming its compatibility with existing techniques such as Diffusion Cache and token reduction methods. These updates underline the method’s generality and practical relevance.\n\nOverall, the paper now presents a clear, well-structured, and comprehensive contribution to efficient diffusion model inference. It demonstrates both technical soundness and practical value. I recommend acceptance.", "rebuttal": "We sincerely thank you for your constructive feedback and for recognizing the novelty and solid empirical results of ALTER. Below we address each question in turn.\n\n---\n\n## Q1 - Add SSIM and PSNR to measure the generation deviation from the pre-trained model\n\nWe provide the SSIM and PSNR to show the deviation of different methods from the original SDv2.1 25 step inference.  \n\n| Model               | Steps | MACs (G) ($\\downarrow$) | Latency (s) ($\\downarrow$) | CC3M SSIM ($\\uparrow$) | CC3M PSNR ($\\uparrow$) | COCO SSIM ($\\uparrow$) | COCO PSNR ($\\uparrow$) |\n| ----------| ----- | ----------------------- | -------------------------- | ---------------------- | ---------------------- | ------------------------- | ------------------------- |\n| SDv2.1              | 25    | 1384.2                  | 4.0                        | -                      | -                      | -                         | -                         |\n| SDv2.1              | 20    | 1384.2                  | 3.2                        | 0.3529                 | 13.71                  | 0.1582                    | 10.24                     |\n| BK-SDM-v2 (Base)    | 25    | 876.5                   | 2.5                        | 0.1208                 | 9.42                   | 0.1551                    | 9.47                      |\n| APTP (0.66)         | 25    | 916.3                   | 2.6                        | 0.0782                 | 8.99                   | 0.0730                    | 8.83                      |\n| ALTER (0.65) (Ours) | 25    | 899.7                   | 2.6                        | 0.1233                 | 9.90                   | 0.3261                    | 12.81                     |\n| ALTER (0.65) (Ours) | 20    | 899.7                   | 2.1                        | 0.1247                 | 9.95                   | 0.2484                    | 11.61                     |\n| ALTER (0.60) (Ours) | 20    | 830.5                   | 1.9                        | 0.1224                 | 9.94                   | 0.1898                    | 10.73                     |\n\nAs shown in the above table, our method **ALTER not only matches the MACs of prior work but also achieves significantly higher SSIM and PSNR scores**, which demonstrates that our approach better preserves the generative characteristics of the original model.\n\nTo provide an even more comprehensive evaluation of generation quality, we also benchmarked our models using advanced preference models, **ImageReward** [1] and Human Preference Score v2 (**HPSv2**) [2] (all at 25 steps). These metrics assess perceptual quality and human aesthetic preference. As shown below, **our method ranks highest on both metrics**, even outperforming the original SDv2.1, confirming that our efficiency gains do not come at the cost of visual quality.\n\n| Model               | MACs (G) | Image Reward ($\\uparrow$) | HPSv2 ($\\uparrow$) |\n| ------------------- | -------- | ------------------------- | ------------------ |\n| sd2.1               | 1384.2   | 0.1139                    | 24.47              |\n| BK-SDM-v2-Base      | 876.5    | -0.0365                   | 23.97              |\n| APTP-Small (COCO)   | 916.3    | -0.8706                   | 19.08              |\n| APTP-Small (CC3M)   | 916.3    | -1.2226                   | 18.19              |\n| ALTER (0.65) (Ours) | 899.7    | 0.1494              | 24.54          |\n| ALTER (0.60) (Ours) | 830.5    | 0.1487                    | 24.36              |\n\n---\n\n## Q2 - Scalability / Orthogonality to other accelerators\n\nWe appreciate the reviewer for highlighting this key advantage of our work. We strongly agree that ALTER’s design makes it orthogonal to and compatible with other diffusion acceleration methods such as Diffusion Cache and token reduction. \n\nHere we conduct a new experiment combining ALTER with a token reduction method ToMeSD [3] to demonstrate ALTER's scalability. For this combination, we follow the default setting of ToMeSD. As shown in the following table, we **boost the total speedup from 3.64x to an impressive 6.54x** with only a minor, controllable trade-off in performance scores. This empirically validates the excellent scalability of our method.\n\n| Model               | Steps | ToMe r% | Speedup      | CLIP ($\\uparrow$) | FID-5K ($\\downarrow$) |\n| ------------------- | ----- | ------- | ------------ | ----------------- | --------------------- |\n| SDv2.1              | 50    | 0       | 1.00$\\times$ | 31.55             | 27.29                 |\n| SDv2.1              | 20    | 0       | 2.49$\\times$ | 31.53             | 27.83                 |\n| ALTER (0.65) (Ours) | 20    | 0       | 3.64$\\times$ | 31.62             | 25.25                 |\n| w/ ToMeSD           | 20    | 40      | 4.93$\\times$ | 31.48             | 25.93                 |\n|                     | 20    | 50      | 5.75$\\times$ | 31.34             | 26.89                 |\n|                     | 20    | 60      | 6.54$\\times$ | 31.24             | 27.96                 |\n\nRegarding caching methods like DeepCache [4], we agree this is another exciting direction. Since ALTER dynamically prunes layers, a compatible cache would only need to store features and reuse for current active layers, potentially enabling a more memory-efficient caching scheme. We will leave this as future work.\n\n---\n\n## Q3 - Rename \"Temporal Expert Routing\" to \"Step Expert Routing\"\n\nThank you for this suggestion. We appreciate you pointing out that \"Step Expert Routing\" might be more immediately familiar to some readers in the diffusion community. We have carefully considered this change. However, we have a deliberate reason for our choice of \"Temporal\" and would prefer to retain it. Our motivation is that the routing mechanism is conditioned on the timestep variable $t$ of the diffusion process, which is represented by the _time embedding_. While it is discretized into steps during diffusion sampling, our framing is rooted in the underlying temporal nature of the process.\n\n---\n\n## Q4 - Standardize symbol usage\n\nThank you for your careful reading and pointing out this redundancy. We have clearly checked the notation definition and standardized all notation in the revised manuscript. We now use $t$ exclusively for the diffusion timestep and have introduced $k$ to denote the training iterations.\n\n---\n\n> References:\n\n[1] Xu _et al._ ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.\n\n[2] Wu _et al._ Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.\n\n[3] Bolya _et al._ Token Merging for Fast Stable Diffusion.\n\n[4] Ma _et al._ DeepCache: Accelerating Diffusion Models for Free."}, {"review_id": "m3u4XVLUAv", "reviewer": "Reviewer_F2cH", "summary": "The paper proposes learning a hypernetwork to predict subnetworks of a pre-trained diffusion U-Net, with subnetworks mapped to certain denoising time steps. The hypernetwork is optimized in alternating steps while the U-Net is fine-tuned, and the result is a faster diffusion inference process that uses pruning but still takes advantage of the full model, pruning layers specific to each time step. Experiments are reported compared to both prior pruning strategies and caching, on SD2.1, for CC3M and COCO.", "questions": "Does the performance hold for other diffusion backbones? What about DiT-based architectures, which have largely replaced UNet models in modern generative pipelines?\n\nDo more modern metrics (Image Reward) validate the results? What about human preference? It is challenging to evaluate the work with only 14 qualitative examples provided, and no human study.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n[S1] The method yields clear improvements over the prior pruning state-of-the-art, and beats caching works (which are somewhat comparable) as well.\n\n[S2] The motivation follows naturally from deficiencies in prior works and is specific to the diffusion process itself.\n\n[S3] The methodology with hypernetworks to predict the network structure is novel in this application.\n\nWeaknesses:\n\n[W1] The empirical evaluation is somewhat limited. More modern measurements of quality (Image Reward, Vision Reward, CLIP-IQA) are not considered, only 2 datasets are used, and, most concerningly, only 1 model is considered. \n\n[W2] The behavior of the experts is slightly unsatisfying. Prior works in caching suggest that these earlier and middle layers can be cached, so the fact that they seem to not be able to be skipped suggests that perhaps, considering this work already computes results for every layer at some time step, perhaps implementing caching in the framework could allow for greater diversity among experts. It also seems important to investigate this behavior for additional diffusion models.", "was_revised": true, "final_justification": "The rebuttal addresses my concerns. I intend to keep my original rating, since I still view the submission as a solid paper and good contribution to the conference, and specifically this sub-area of diffusion acceleration. I think the motivation is sound, the method is interesting/novel, the results are good, and the methods chosen for comparison are relevant/broad.", "rebuttal": "We sincerely thank you for your detailed review and positive assessment of our work's novelty and motivation. We are pleased to present new results that address your concerns.\n\n---\n\n## W1 & Q2 - More Comprehensive Empirical Evaluation\n\nThank you for suggesting a more comprehensive evaluation with modern metrics. We agree this is crucial for assessing generative quality. In response, we have benchmarked our method using **ImageReward** [1] and Human Preference Score v2 (**HPSv2**) [2], two state-of-the-art models designed to measure human aesthetic and semantic preferences. We calculate these 2 metrics on their own benchmark datasets. Since APTP is prompt-based pruning, we use the prompt split method they provided to divide the benchmark's prompts to different experts finetuned on MS-COCO and CC3M.\n\nAs shown in the following two tables, the results are highly encouraging. On ImageReward, **our ALTER models rank #1 (0.65 MACs) and #2 (0.60 MACs)**, outperforming prior methods and even the original SDv2.1 baseline, which indicates a higher alignment with human preference. Similarly, on the fine-grained HPSv2 benchmark, our method consistently achieves the highest average score, demonstrating robust performance across diverse visual styles from animation to photo-realism.\n\n| Model               | MACs (G) | Image Reward ($\\uparrow$) | Rank |\n| ------------------- | -------- | ------------------------- | ---- |\n| SDv2.1              | 1384.2   | 0.1139                    | 3    |\n| BK-SDM-v2 (Base)    | 876.5    | -0.0365                   | 4    |\n| APTP (0.66) (COCO)  | 916.3    | -0.8706                   | 5    |\n| APTP (0.66) (CC3M)  | 916.3    | -1.2226                   | 6    |\n| ALTER (0.65) (Ours) | 899.7    | 0.1494                    | 1    |\n| ALTER (0.60) (Ours) | 830.5    | 0.1487                    | 2    |\n\n| Model               | MACs (G) | HPSv2 ($\\uparrow$) | Rank |\n| ------------------- | -------- | ------------------ | ---- |\n| SDv2.1              | 1384.2   | 24.47              | 2    |\n| BK-SDM-v2 (Base)    | 876.5    | 23.97              | 4    |\n| APTP (0.66) (COCO)  | 916.3    | 19.08              | 5    |\n| APTP (0.66) (CC3M)  | 916.3    | 18.19              | 6    |\n| ALTER (0.65) (Ours) | 899.7    | 24.54              | 1    |\n| ALTER (0.60) (Ours) | 830.5    | 24.36              | 3    |\n\n---\n\n## W2 - Behavior of the Experts\n\nThis is an excellent question that makes us clarify the relationship between our dynamic pruning method and caching. The core difference is that caching methods reuse a layer's **output features** from a previous step, whereas our method, ALTER, learns to skip the **entire computation** of a layer.\n\nCaching methods like DeepCache [3] show that the _outputs_ of early-to-mid layers can be reused across steps, as they capture low-frequency information that evolves slowly.  However, our results show that foundational early and middle layers are rarely pruned, suggest that while their outputs might be stable enough to be cached, the computations performed within them are still **essential at every step** to progressively refine the feature maps for later layers. Skipping these layers entirely is too detrimental to the final image quality. This observation is also consistent with prior work like APTP, which noted that deeper layers were more amenable to pruning than the foundational ones. \n\nWe agree that encouraging greater routing diversity is a valuable direction for future work, potentially using regularization techniques. We also strongly agree that caching methods could **complement** ALTER. This highlights a powerful scheme: ALTER could first prune the model's architecture, and a caching scheme could then be applied to the remaining active layers for further speedups.\n\n---\n\n## Q1 - Does the performance hold for other diffusion backbones?\n\nTo empirically validate that our framework generalizes beyond U-Nets, we have **conducted a new experiment applying ALTER to a DiT-XL/2 [4] backbone** on ImageNet (256 $\\times$ 256 resolution generation, 250 steps). For the DiT architecture, each Transformer block was treated as a prunable unit, and the ALTER framework was applied without fundamental architectural changes. \n\n| Model                        | Speedup      | IS ($\\uparrow$) | FID ($\\downarrow$) | sFID $(\\downarrow)$ | Precision ($\\uparrow$) | Recall ($\\uparrow$) |\n| ---------------------------- | ------------ | --------------- | ------------------ | ------------------- | ---------------------- | ------------------- |\n| DiT-XL/2 [4] (28-layer)      | 1.00$\\times$ | 277.00          | 2.27               | 4.60                | 0.83                   | 0.57                |\n| DyDiT-XL [5] ($\\lambda=0.5$) | 1.72$\\times$ | 248.03          | 2.07               | 4.56                | 0.80                   | 0.61                |\n| TinyFusion [6] (14-layer)    | 1.96$\\times$ | 234.50          | 2.86               | 4.75                | 0.82                   | 0.55                |\n| ALTER (Ours) (avg. 14-layer) | 1.92$\\times$ | 254.29          | 2.36               | 4.63                | 0.82                   | 0.58                |\n\nWith approximately half the layers active on average, **our method significantly outperforms static pruning (TinyFusion [6]), especially on key quality metrics like IS (254.29 vs. 234.50) and FID (2.36 vs. 2.86). It is nearly comparable with the full DiT-XL model,** demonstrating a good quality-efficiency trade-off. When comparing to dynamic methods like DyDiT [5], ALTER brings higher speedup. It's important to note that our approaches are **orthogonal and complementary**. ALTER performs coarse-grained **layer-level pruning**, while DyDiT performs fine-grained pruning _within_ each block. \n\nThis successful application of ALTER to a DiT backbone and its orthogonality to fine-grained methods, validates that ALTER is a general and effective framework for accelerating various diffusion architectures.\n\n---\n\n> References:\n\n[1] Xu _et al._ ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.\n\n[2] Wu _et al._ Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.\n\n[3] Ma _et al._ DeepCache: Accelerating Diffusion Models for Free.\n\n[4] Peebles _et al._ DiT: Diffusion Models with Transformer Backbones.\n\n[5] Zhao et al. Dynamic Diffusion Transformer.\n\n[6] Li _et al._ TinyFusion: Layer Fusion for Diffusion Speed-up."}, {"review_id": "E25MeCab2F", "reviewer": "Reviewer_epHt", "summary": "The paper proposes a novel idea for pruning T2I diffusion models, where instead of using static printing (same pruned model for all prompts as done in BKSDM) or prompt-based uniform pruning (routing prompts to specific pruned model as done in APTP), the authors propose to route each timestamp to a different pruned model based on the prompt. They use a Hypernetwork, which dynamically generates layer pruning decisions and manages timestep routing to specialized, pruned expert sub-networks. This has enabled them to make each expert cater to different groups of timestamps. Though somewhat limited, the quantitative results demonstrate the effectiveness of this method.", "questions": "Apart from the points listed above in \"weaknesses\", below are some additional questions:\n- Pruning primarily happens at the final layers. Why is this the case?\n- Is batch-parallelism possible with this approach? My understanding is that since each timestep can be routed to a different model, it would not be. Do the speed-up numbers take account of this?\n- How much of the performance is accounted for by the final Fine-tuning after T-end? Could you include a table showing performance before and after this step?", "limitations": "Yes, but only technical limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ SOTA results on the SD2.1 architecture. The performance is even better than unpruned SD2.1 (at 25/20 steps). APTP [18], the most similar related work, was unable to achieve this. At standard 50 steps, same performance can be achieved at lower MACs.\n+ Clever implementation of an idea from the Language community to Vision. ToMoE [55] uses MoEs and routes each token to a different pruned subnetwork. This work does a similar thing, where each timestamp is routed to a different pruned subnetwork.\n+ Comprehensive ablations have been done, and they show how different contributions impact the final results.\n+ The exposition is clear and convincing.\n\nWeaknesses:\n- The overall idea is fairly close to APTP in terms of novelty. Almost similar setup is utilized, though instead of prompt-based static routing to experts, they have done prompt-based temporal routing. More insights and discussions around the similarities and differences would be helpful.\n- Quantitative results are limited to a single diffusion architecture. Therefore, generalizability to other diffusion architectures is unclear.\n- The choice of 10 experts as the optimal number of experts is debatable and should not be claimed as such. The experiments are only presented for SD2.1, on only 2 datasets (CLIP score is anyway not the optimal at 10 experts for MS-COCO), and just 2 metrics (FID and CLIP score). Either toning down this claim or more rigorous analysis would help.", "was_revised": true, "final_justification": "The rebuttal has clarified most of my concerns and I will increase my rating to reflect this.", "rebuttal": "We thank you for the insightful review and for recognizing the strengths of our work. Below we  clarify our novelty compared to prior art and provide new results to address your questions.\n\n---\n\n## W1 - Novelty and Comparison to APTP\n\nWe thank the reviewer for this point, and we have revised the paper to better articulate our novelty. While both methods use experts, ALTER introduces two fundamental differences from APTP: a *dynamic, temporal routing mechanism* and a **unified, single-stage training framework**.\n\n1. **Routing Mechanism & Model Utilization:** APTP uses *static, prompt-based routing*. Each prompt is assigned to _one_ expert sub-network for all timesteps. This means an image is generated using only a static portion of the model's total capacity. In contrast, ALTER uses **dynamic, temporal routing**. Every prompt is processed by a _sequence_ of different experts, one for each timestep. This allows every image generation to leverage the **full capacity** of the model, assigning the most suitable specialized expert (e.g., \"structure expert\" vs. \"texture expert\") to each stage of the diffusion process as illustrated in Figure 1 of our paper.\n2. **Training & Deployment Framework:** APTP requires a complex, multi-stage pipeline involving dataset clustering, experts determining and separate fine-tuning for each expert, resulting in multiple model checkpoints. ALTER is an **end-to-end framework** that learns the routing and experts jointly in a single stage, producing a single, unified model that is far simpler to manage and deploy.\n\n| Method       | Routing      | Model Usage                | Training              | Deployment               | Scalability                                       |\n| ------------ | ------------ | -------------------------- | --------------------- | ------------------------ | ------------------------------------------------- |\n| APTP         | Per-prompt   | Partial capacity per image | Multi-stage, Separate | Multiple separate models | Need re-partition & re-train for every new domain |\n| ALTER (Ours) | Per-timestep | Full capacity per image    | End-to-end, Unified   | A single unified model   | Works across domains                              |\n\nThese distinctions are critical to our method's ability to improve upon the baseline's generation quality, a result not achieved by prior pruning work.\n\n---\n\n## W2 - Generalizability to Other Diffusion Architectures\n\nWe agree that demonstrating generality is crucial. To address this, we have **conducted a new experiment applying ALTER to a DiT-XL/2 [1] backbone**. The results show that our method achieves a ~1.92x speedup while maintaining an FID score (2.36) comparable to the full model (2.27). \n\nWith approximately half the layers active on average, **our method significantly outperforms static pruning (TinyFusion [2]), especially on key quality metrics like IS (254.29 vs. 234.50) and FID (2.36 vs. 2.86).** When comparing to dynamic methods like DyDiT [3], ALTER brings higher speedup. It's important to note that our approaches are **orthogonal and complementary**. ALTER performs coarse-grained **layer-level pruning**, while DyDiT performs fine-grained pruning _within_ each block. \n\nThis empirically confirms that our framework is effective beyond U-Nets and is applicable to modern Transformer-based architectures.\n\n| Model                        | Speedup      | IS ($\\uparrow$) | FID ($\\downarrow$) | sFID $(\\downarrow)$ | Precision ($\\uparrow$) | Recall ($\\uparrow$) |\n| ---------------------------- | ------------ | --------------- | ------------------ | ------------------- | ---------------------- | ------------------- |\n| DiT-XL/2 [1] (28-layer)      | 1.00$\\times$ | 277.00          | 2.27               | 4.60                | 0.83                   | 0.57                |\n| DyDiT-XL [2] ($\\lambda=0.5$) | 1.72$\\times$ | 248.03          | 2.07               | 4.56                | 0.80                   | 0.61                |\n| TinyFusion [3] (14-layer)    | 1.96$\\times$ | 234.50          | 2.86               | 4.75                | 0.82                   | 0.55                |\n| ALTER (Ours) (avg. 14-layer) | 1.92$\\times$ | 254.29          | 2.36               | 4.63                | 0.82                   | 0.58                |\n\n---\n\n## W3 - The Choice of Experts Number\n\nThank you for this advice. We agree with the reviewer that claiming 10 experts as universally \"optimal\" is an overstatement. Our intention was to identify a practical and effective configuration for the SDv2.1 model. \n\nWe have **revised the manuscript** to tone down this claim, and refer to 10 experts as a \"well-performing configuration\" that demonstrates a strong efficiency-quality trade-off. Furthermore, our new experiments on the DiT architecture also successfully utilized 10 experts, achieving the strong results reported (responses to W2). This suggests that 10 is a reasonable choice across different architectures, though the ideal number may vary with the specific model and dataset.\n\n---\n\n## Q1 - Why does pruning primarily happen at the final layers?\n\nThis is an excellent question. Our hypothesis is that it reflects the different roles of layers in the U-Net. Early/middle layers capture foundational spatial structures and low-level features that are essential throughout the entire diffusion process. Pruning them would be too destructive. In contrast, the final decoder layers are responsible for high-level semantic refinement and adding fine-grained details, tasks that are more stage-specific. Our time-aware router learns that these specialized refinement tasks can be effectively handled by smaller experts or even pruned at certain timesteps without compromising the core structure. **This observation is also consistent with prior work like APTP**, which noted that deeper layers were more amenable to pruning than the foundational ones.\n\n---\n\n## Q2 - Is batch-parallelism possible with this approach?\n\nThank you for this critical question. **Yes, batch-parallelism is possible for our framework.** Our routing mechanism is deterministic, which means that for a given prompt batch and a specific timestep $t$, the routing decision is **identical for all samples in a batch**. The entire batch can therefore be processed efficiently through the same selected expert sub-network. This provides a significant practical advantage over prompt-routing methods like APTP. In APTP, different prompts within a single batch may be routed to different experts, breaking efficient batching and requiring serialization or complex batch management that can harm real-world throughput. \n\nHowever, to ensure fair and reproducible comparisons, our latency benchmarks in the paper follow prior work and use a batch size of 1.\n\n---\n\n## Q3 - How much of the performance is accounted for by the final Fine-tuning after T-end?\n\nThanks for your suggestion. To show the contribution of the final fine-tuning stage, we have conducted the requested ablation. The results below are for our ALTER (0.65) model on MS-COCO 2017 dataset.\n\n| Model                      | Steps | CLIP ($\\uparrow$) | FID-5K ($\\downarrow$) |\n| -------------------------- | ----- | ----------------- | --------------------- |\n| SDv2.1                     | 20    | 31.53             | 27.83                 |\n| ALTER (0.65) (T$_{end}$)   | 20    | 31.54             | 25.98                 |\n| ALTER (0.65) (T$_{total}$) | 20    | 31.62             | 25.25                 |\n| ALTER (0.65) (T$_{end}$)   | 15    | 31.05             | 27.16                 |\n| ALTER (0.65) (T$_{total}$) | 15    | 31.19             | 26.58                 |\n\nAs the results demonstrate, the model's performance after T$_{end}$ is already very strong, achieving the majority of the final quality. The final fine-tuning provides a consistent and important boost across both metrics and step counts, further improving both FID and CLIP scores.\n\n---\n\n> References:\n\n[1] Peebles _et al._ DiT: Diffusion Models with Transformer Backbones.\n\n[2] Li _et al._ TinyFusion: Layer Fusion for Diffusion Speed-up.\n\n[3] Zhao _et al._ Dynamic Diffusion Transformer."}, {"review_id": "pps9ZnmDR6", "reviewer": "Reviewer_C2Gm", "summary": "This paper proposes to view layer-pruned sub-networks as experts, and learns to dynamically select different sub-networks for different diffusion timesteps. A 2D binary mask is used to represent the layer selection results for each expert, and a router selects an expert based on the timestep. The method effectively reduce the average computation of SD2.1 with comparable generation quality.", "questions": "1. It can be observed from Fig. 4 that each expert's sparsity level is not high (most layers are still preserved). I'm curious why the speedup with 25 steps is still significant in Tab. 1. \n\n2. In T2I settings, GenEval [3] might be helpful to evaluate the generation ability more comprehensively. For SD3 or FLUX-based models, human evaluation is also recommended.\n\n[3] GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The motivation is clear and reasonable;\n\n2. The method is novel and sound;\n\n3. The results on SD2.1 are solid, and the realistic speedup is significant.\n\n### Weaknesses\n\nOverall, I think it's an interesting work. My major concern is the insufficient experimental validation.\n\n1. The validated model is limited on SD2.1. As we all know, SD2.1 is a UNet structure, and the currently popular structures like SD3, Flux, Hunyuan and Wanx are based on DiT architectures. It would be more convincing if the authors validate the method on more DiT architectures, including the original DiT and the T2X models (FLUX and Wanx).\n\n2. The compared baselines are insufficient.\n    - Static layer-pruning method: FLUX-lite (https://huggingface.co/Freepik/flux.1-lite-8B).\n    - Dynamic methods such as DyDiT [1] and TPDM [2].\n    - Few-step methods such as SD-turbo (based on SD2.1, https://huggingface.co/stabilityai/sd-turbo) and FLUX-turbo (https://huggingface.co/alimama-creative/FLUX.1-Turbo-Alpha). \n    \n\n[1] DyDiT++: Dynamic Diffusion Transformers for Efficient Visual Generation\n\n[2] Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation", "was_revised": true, "final_justification": "After reading the rebuttal and other reviewers' comments, I tend to maintain my score", "rebuttal": "Thank you for your review and for recognizing the novelty and solid performance of our work. We have performed several new experiments to address your primary concerns.\n\n---\n\n## W1 - Generalization to DiT Architectures\n\nWe agree that demonstrating generality is crucial. To address this, we have **conducted a new experiment applying ALTER to a DiT-XL/2 backbone**. The results show that our method achieves a ~1.92x speedup while maintaining an FID score (2.36) comparable to the full model (2.27). \n\nWith approximately half the layers active on average, **our method significantly outperforms static pruning (TinyFusion), especially on key quality metrics like IS (254.29 vs. 234.50) and FID (2.36 vs. 2.86).** When comparing to dynamic methods like DyDiT, ALTER brings higher speedup. It's important to note that our approaches are **orthogonal and complementary**. ALTER performs coarse-grained **layer-level pruning**, while DyDiT performs fine-grained pruning _within_ each block. These new results demonstrate that ALTER is a general and effective framework for accelerating various diffusion architectures. \n\n| Model                        | Speedup      | IS ($\\uparrow$) | FID ($\\downarrow$) | sFID $(\\downarrow)$ | Precision ($\\uparrow$) | Recall ($\\uparrow$) |\n| ---------------------------- | ------------ | --------------- | ------------------ | ------------------- | ---------------------- | ------------------- |\n| DiT-XL/2 [4] (28-layer)      | 1.00$\\times$ | 277.00          | 2.27               | 4.60                | 0.83                   | 0.57                |\n| DyDiT-XL [5] ($\\lambda=0.5$) | 1.72$\\times$ | 248.03          | 2.07               | 4.56                | 0.80                   | 0.61                |\n| TinyFusion [6] (14-layer)    | 1.96$\\times$ | 234.50          | 2.86               | 4.75                | 0.82                   | 0.55                |\n| ALTER (Ours) (avg. 14-layer) | 1.92$\\times$ | 254.29          | 2.36               | 4.63                | 0.82                   | 0.58                |\n\nWe have not implemented ALTER on SD3 and FLUX because of the limited computational resources and rebuttal time, but our successful application to DiT provides strong evidence that our approach is a general framework for accelerating various diffusion backbones.\n\n---\n\n## W2 - Comparison with Other Baselines\n\nThank you for providing this comprehensive list. Here is how we position our work with respect to these methods:\n- **DyDiT++ [4] & TPDM [5] :** As shown above, our DiT experiment now includes a comparison with DyDiT. But it's important to note that our approaches are **orthogonal and complementary**. ALTER performs coarse-grained **layer-level pruning**, while DyDiT performs fine-grained pruning _within_ each block. TPDM is an orthogonal approach that accelerates inference by predicting and skipping entire timesteps. Our method, which prunes layers _within_ a step, could be combined with TPDM to accelerate the remaining, non-skipped steps.\n\n- **Few-step & Distilled Models (SD-turbo, FLUX-turbo):** These methods represent a different family of acceleration that uses distillation to train a specialized model for extremely few-step (e.g., 4 step) inference. Our work focuses on accelerating the original, pre-trained model for general-purpose use across various step counts (e.g., 20-50 steps) without requiring a full distillation process. Besides, step distillation methods are computationally intensive, often demanding thousands of GPU hours, whileour ALTER method is lightweight, requiring **only about 48 A100 hours** to fine-tune the SDv2.1 model.\n\n- **FLUX-lite:** This is a statically pruned version of the FLUX architecture. Our primary experiments focus on accelerating the public SD2.1 and DiT backbones. Where we do include a statically pruned version of the SDv2.1 (BK-SDM-v2) and the DiT-XL/2 (TinyFusion).\n\nWe have **updated our related work section** to include a detailed discussion of these methods, clarifying the different axes of acceleration and contextualizing our contribution.\n\n---\n\n## Q1 - Why the speedup with 25 steps is still significant in Tab. 1.\n\nAs shown in Fig.4, the pruned layers are almost at the last part, which is at high-resolution considering the UNet's architecture. The significant speedup arises from the **highly non-uniform computational cost of layers in a U-Net**. Although the pruned layer's number is not big, prunning the high-resolution could benefit more for acceleration compared with the middle layers.\n\n---\n\n## Q2 - GenEval [6] and Human Evaluation\n\nThank you for the valuable suggestion to use GenEval. We made a significant effort to incorporate it but encountered unresolved technical issues with the official GenEval repository's environment setup, which we were unable to fix within the limited rebuttal period. And these technical problems are very common as shown in the github repository's opening issues. Thanks for your understanding.\n\nHowever, to follow the spirit of your suggestion for a more comprehensive evaluation we have benchmarked our method using **ImageReward** [7] and Human Preference Score v2 (**HPSv2**) [8], two state-of-the-art models designed to measure human aesthetic and semantic preferences. We calculate these two metrics on their own benchmark datasets. Since APTP is prompt-based pruning, we use the prompt split method they provided to divide the benchmark's prompts to different experts finetuned on MS-COCO and CC3M.\n\nAs shown in the following two tables, the results are highly encouraging. On ImageReward, **our ALTER models rank #1 (0.65 MACs) and #2 (0.60 MACs)**, outperforming prior methods and even the original SDv2.1 baseline, which indicates a higher alignment with human preference. Similarly, on the fine-grained HPSv2 benchmark, our method consistently achieves the highest average score, demonstrating robust performance across diverse visual styles from animation to photo-realism.\n\n| Model               | MACs (G) | Image Reward ($\\uparrow$) | Rank |\n| ------------------- | -------- | ------------------------- | ---- |\n| SDv2.1              | 1384.2   | 0.1139                    | 3    |\n| BK-SDM-v2 (Base)    | 876.5    | -0.0365                   | 4    |\n| APTP (0.66) (COCO)  | 916.3    | -0.8706                   | 5    |\n| APTP (0.66) (CC3M)  | 916.3    | -1.2226                   | 6    |\n| ALTER (0.65) (Ours) | 899.7    | 0.1494                    | 1    |\n| ALTER (0.60) (Ours) | 830.5    | 0.1487                    | 2    |\n\n| Model               | MACs (G) | HPSv2 ($\\uparrow$) | Rank |\n| ------------------- | -------- | ------------------ | ---- |\n| SDv2.1              | 1384.2   | 24.47              | 2    |\n| BK-SDM-v2 (Base)    | 876.5    | 23.97              | 4    |\n| APTP (0.66) (COCO)  | 916.3    | 19.08              | 5    |\n| APTP (0.66) (CC3M)  | 916.3    | 18.19              | 6    |\n| ALTER (0.65) (Ours) | 899.7    | 24.54              | 1    |\n| ALTER (0.60) (Ours) | 830.5    | 24.36              | 3    |\n\n---\n\n> References:\n\n[1] Peebles _et al._ DiT: Diffusion Models with Transformer Backbones.\n\n[2] Li _et al._ TinyFusion: Layer Fusion for Diffusion Speed-up.\n\n[3] Zhao _et al._ Dynamic Diffusion Transformer.\n\n[4] Lao _et al._ DyDiT++: Dynamic Diffusion Transformers for Efficient Visual Generation. \n\n[5] Ye _et al._ Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation.\n\n[6] Ghosh _et al._ GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.\n\n[7]  Xu _et al._ ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.\n\n[8] Wu _et al._ Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis."}]

Author rebuttal:

Meta-review: This paper introduces ALTER, a unified framework for improving the efficiency of diffusion models by jointly optimizing layer pruning, expert routing, and model fine-tuning in a single stage. Unlike prior uniform or sequential pruning strategies, ALTER employs a trainable hypernetwork to dynamically generate pruning decisions and route timesteps to specialized pruned experts during fine-tuning. This co-optimization achieves substantial efficiency gains—delivering a 3.64× speedup with 35% sparsity and only 25.9% of the original MACs—while maintaining the visual fidelity of the original diffusion model.

The reviewers agreed on the importance of the problem, the novelty of the approach, and the strength of the empirical results. They also highlighted the significant speedups achieved across benchmarks and the clarity of the presentation. For these reasons, we recommend acceptance.
Decision: Accept (poster)