Sparse-to-Sparse Training of Diffusion Models
Authors: Inês Cardoso Oliveira
Decebal Constantin Mocanu
Luis A. Leiva
Venue: Accepted by TMLR ()

Abstract: Diffusion models (DMs) are a powerful type of generative models that have achieved state-of-the-art results in various image synthesis tasks and have shown potential in other domains, such as natural language processing and temporal data modeling. Despite their stable training dynamics and ability to produce diverse high-quality samples, DMs are notorious for requiring significant computational resources, both in the training and inference stages. Previous work has focused mostly on increasing the efficiency of model inference. This paper introduces, for the first time, the paradigm of sparse-to-sparse training to DMs, with the aim of improving both training and inference efficiency. We focus on unconditional generation and train sparse DMs from scratch (Latent Diffusion and ChiroDiff) on six datasets using three different methods (Static-DM, RigL-DM, and MagRan-DM) to study the effect of sparsity in model performance. Our experiments show that sparse DMs are able to match and often outperform their Dense counterparts, while substantially reducing the number of trainable parameters and FLOPs. We also identify safe and effective values to perform sparse-to-sparse training of DMs.

Published in Transactions on Machine Learning Research (09/2025)
Sparse-to-Sparse Training of Diffusion Models
Inés Cardoso Olivcira
i. ooliveera Durii bu
University of Luzzebborr
Decebal Constantin Mocanu University of Luzembourg
Luis A. Leiva University of Luzembourg
luis. Ieiuua Buree. bu
Reviewed on OpenReview: https: openreview. net/ forun? id= sRupdoPL.Ja
Abstract
Diffusion models (DMs) are a powerful type of generative models that have achieved state-of- the-art results in various image synthesis tasks and have shown potential in other domains, such natural language processing and temporal data modcling. Despite their stable training dynamics and ability to produce diverse high-quality samples, DMs are notorious for requiring significant computational resources, both in the training and inference stages. Previous work has focused mostly on increasing the efficiency of model inference. This paper introduces, for the first time, the paradigm of sparse-to-sparse training to DMs, with the aim of improving both training and inference efficiency. We focus on unconditional generation and train sparse DMs from scratch (Latent Diffusion and ChiroDiff) on six datasots using three different methods (Static-DM, RigI.-M,, and MagRan-DM) to study the effect of sparsity in model performance. Our experiments show that sparse DMs are able to match and often outperform their Dense counterparts, while substantially reducing the number of trainable parameters and FLOPs. We also identify safe and effective values to perform sparse-to-sparse training of DMs.
1 Introduction
Diffusion models (DMs) are a class of deep generative models that exhibit extraordinary performance to produce diverse and high-quality data. DMs currently dominate the generative field in computer vision, having been applied
wide range of tasks such as (un)conditional image generation (Ho et al., 2020b; Rombach al., 2021; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021; Nichol et al., 2022; Blattmann et al., 2022; Das et al., 2023), image super-resolution (Saharia al., 2021: Chung et al., 2022), and image inpainting (Nichol et al., 2022; Chung et al., 2022; Saharia et al., 2022), among others. DMs have also shown incredible potential in other domains, including speech generation (Liu et al., 2023a), text generation (Li et al., 2022; Gong et al., 2023), and time-series prediction and imputation (Rasul et al., 2021; Tashiro et al., 2021).
Despite these advantages, DMs are notorious for their slow training, demanding significant computational resources and resulting considerable carbon footprint (Strubell et al., 2020). Due the extensive number of diffusion timesteps required to produce single sample (e.g., Rombach et al. (2021) mentioned up to 500 steps), DMs also suffer from slow sampling speed (Song et al., 2021). Even though progress has been made in improving inference speed, DMs are still considerably slower than other generative approaches such as GANs and VA (Rombach et al., 2021). This inefficiency impacts not only end users, but also the research community, by hindering further developments due to the lengthy process of model training and evaluation.
Reducing the computational costs and memory requirements of DMs critical challenge for the broad implementation and adoption of these models, and active field of research. Much of the existent literature

---

Published in Transactions on Machine Learning Research (09/2025)
has addressed this challenge through improvements to the inference stage (Song et al., 2021; Nichol & Dhariwal, 2021; Fang et al., 2023; Shang et al., 2023; Li et al., 2023; Salimans & Ho, 2022; Meng et al., 2023). Efforts have also been made in the direction of training efficiency, exploring different architectures and training strategics (Wang et al., 2023; Ding et al., 2023; Rombach et al., 2021; Phung et al., 2022), but training DMs is still an exlensive and costly process.
In the last few years, sparse-to-spars, training has emerged as a promising approach to significantly reduce the computational cost of deep learning models, by training sparse networks from scratch (Mocamu et al., 2018; Bellec et al., 2018; Dettmers & Zettlemoyer, 2019; Evci et al., 2020; Zhang et al., 2024b). Interestingly, sparse neural networks have been shown to match, or even outperform, their Dense counterparts in classification tasks (Mocanu et al., 2018; Liu et al., 2021a), generative modeling using GANs (Liu et al., 2023b), and Reinforcement Learning (Sokar et al., 2022), all while requiring less memory and reducing the mumber of floating-point operations (FLOPs). We should note that, currently, most sparse neural networks require roughly the same amount of time to train their dense counterparts, since today's hardware is optimized for dense matrix operations. However, growing interest in sparse models is reshaping the landscape; see Appendix A for discussion in this regard.
We propose to lower the computational cost of DMs by incorporating, for the first time, the paradigm of sparse-to-sparse training for unconditional generation. As such, we introduce three different methods, Static-DM (static strategy), RigL-DM, and MagRan-DM (both dynamic strategies), that can be easily integrated with existing DMs. Since our goal to study the effect these techniques on the performance of DMs, we experiment using two state-of-the-art DMs in two domains: Latent Diffusion (Rombach et al., 2021) for image generation (continuous, pixel-level data) and ChiroDiff (Das et al., 2023) for sketch generation (discrete, spatiotemporal sequence data). In sum, make the following contributions:
We introduce pparse-to-spars: training to unconditional DMs, with both static and dynamic strategies. We consider various sparsity levels, two state-of-the-ar models (Latent Diffusion and ChiroDiff), and six datasets in total. We also perform some experiments using conditional DMs.
Our experiments show great promise of sparse-to-sparss training for DMs, we were able to train a sparse DM for each model/ model/dataset case with comparable performance to their respective Dense counterpart, while significantly reducing the parameters count and FLOPs. In most cases, at least one sparse DM outperformed its Dense version.
We identify safe and effective values to perform sparse-to-sparee training of DMs. Higher performance is achieved using dynamic sparse training with 25 50% sparsity levels. For models with higher sparsity ratio, a conservative prune and regrowth ratio of 0.05 provides better results.
2 Background and Related Work
2.1 Diffusion Models
DMs (Sohl-Dickstein et al., 2015; Ho et al., 2020a; Song al., 2020) are probabilistic models designed learn a data distribution ()x) through two processes: forward noising process and reverse denoising process. The forward process is defined each timestep f, producing a sequence of increasingly noisier samples:
Markov Chain of length T in which Gaussian noise is added at
q( = -N(x /I B,Ir-1,8111
(1)
T
( = II ((1,111-11)
(2)
where Zo is the original data point, s, is the data point at timestep 1, and Be is the pre-defined amount of noise added at timestep t.
The reverse denoising process q(x)-1/Z)), attempts to recover the original data, but it is intractable as it depends on the entire data distribution q(x). As such, we need to parameterize neural network po
2

---

Published in Transactions on Machine Learning Research (09/2025)
to approximate it. This network po can be optimized by training with the simplified objective, L = [1 Vo ea (2x, 1)l?, where ør is a noisy version of input x at the final timestep T, and co the prediction of the neural network pa.
2.2 Efficiency in Diffusion Models
Increasing the efficiency of DMs has been primarily addressed through accelerating the sampling process, by reducing the number of diffusion steps through faster sampling (Song et al., 2021; Karras et al., 2022) and model distillation (Salimans & Ho, 2022; Meng et al., 2023; Yin et al., 2024). As for training acceleration, some works have proposed shifting the diffusion process to the latent space (Rombach et al., 2021; Vahdat et al., 2021). Interestingly, Phung et al. (2022) used discrete wavelet transforms to decompose images into sub-bands, employing these sub-bands to perform the diffusion more efficiently.
Previous studies have also presented refinmment to the training process of DMs. For example, Wang et al. (2023) introduced a plug-and-play training strategy that utilizes patches instead of the full images, to improve training speed. Hang et al. (2024) proposed treating DMs as a multitask learning problem and introduced weighting strategy to balance the different timesteps, achieving significant improvement in training convergence speed.
From the perspective of network compression, prior works have explored techniques such as structural pruning (Fang et al., 2023), post-training quantization (Shang et al., 2023; Li et al., 2023), knowledge distillation (Yang et al., 2023), and the lottery ticket hypothesis (Frankle & Carbin, 2019; Jiang et al., 2023). Very recently, Wang et al. (2024) proposed the incorporation of sparse masks into pro-trained DMs before fine-tuning, and achieved a 50% reduction in mulliply-accummullte operations (MACs) with only a slight average decrease of image quality (as measured by the FID score). Although these techniques work in increasing efficiency, they still require pre-training of full DMs. Our work proposes training sparse DMs from scratch, which has the potential to both accelcrate training and inference, and reduce the memory footprint.
2.3 Soarse-to-Sparse Training
Nowadays most computational models are what is referred to as Dense networks, comprising a stack of layers containing multiple neurons, each connected to all neurons in the following layer. Sparse-to-sparse training techniques aim to train sparse neural networks from scratch, thus reducing the number of parameters and computations. If we define the connectivity graph of a Dense neural network G(V,E), where V represents the set of neurons (vertices), and & the set of connections between them (edges), sparse version of that neural network would be defined as G(V", E'), with V and E being a subset of the neurons and connections of the Dense network. Sparse networks can be obtained using structured methods, where V * V" and unstructured methods, where V= V". Overall, sparse- so-sparse training techniques can be divided into static sparse training (SST) and dynamic sparse training (DST).
Static Sparse Training. In SST methods, the connectivity pattern between neurons is set at initialization, and remains fixed during training. This concept was first introduced by Mocanu et al. (2016), who proposed a non-uniform scale-free topology for Restricted Boltzmann machines, with the sparse models achieving better results than their Dense counterparts. Later, Liu et al. (2022) investigated the efficacy of random pruning at initialization, and found that, using appropriate layer-wise sparsity ratios, randomly pruned subnetwork of WedeResNet-500 can outperform dense Nede on ImageNet. Many other criteria have been proposed to set layer-wise sparsity ratios before training, by trying to identify important connections using information such as connection scnsitivity, as in SNIP (Lee et al., 2019), gradient flow (Wang et al., 2020), as in GraSP. Very recently, two new initialization criteria have been proposed that utilize concepts from network science theory: Bipartite Scale-Free and Bipartite Small-World (Zhang et al., 2024a;b).
Dynamic Sparse Training. In DST methods, the network is initialized with a connectivity pattern and dynamically explores different connections throughout training (Mocanu et al., 2018; Belloe et al., 2018). This was first proposed by Mocanu et al. (2018) through Sparse Evolutionary Training (SET), an algorithm that adjusts the connections using a prune-and-grow scheme every N training steps. In SET, weights are
3

---

Published in Transactions on Machine Learning Research (09/2025)
dropped based on their magnitude (ensuring equal amount of positive and negative weights) and regrown randomly. Rigl. (Evci et al., 2020) proposes an alternative method that prunes the weights based on the absolute magnitude, and regrows them based on the gradients by calculating the dense gradients only at the update step. Although further pruning methods have been proposed (Lee et al., 2019; Yuan et al., 2021), a study by Nowak et. al. (2023) found only minor differences between the tested criteria. The contrast was higher lower density patterns, with magnitude pruning giving the best performance. Other growing criteria have been proposed based on randomness (Mostafa & Wang, 2019) and momentum (Dettmers & Zettlemoyer, 2019).
Recently, Zhang et al. (2024b) proposed Epitopological Sparse Meta-deep Learning (ESML), brain-inspired, gradient-free method, which aims to shift the focus from the weights to the network topology, and uses concepts from network theory. By leveraging ESML, the authors train a sparse network that using just 1% of the connections, able to surpass dense networks, as well as other DST methods, several image classification tasks.
DST has also been applied to the field of generative modelling: Liu et al. (2023b) proposed STU-GAN, comprised of a generator with high sparsity and a denser discriminator. STU-GAN was able to outperform dense BigGAN on CIFAR-10 with a 80% sparse generator and 70% sparse discriminator.
3 Methodology
Our study aims to understand the effect of sparse-to-sparse training techniques DMs. We focus unstructured sparsity due to its ability to maintain high performance even at very high levels of sparsity (Evei et al., 2020). Thus, our experiments cannot rely on current hardware to accelerate sparse computations; for example, NVIDIA A100 and Ampere cards only support 2:4 structured sparsity, which requires to enforce a fixed sparsity level of 50%. In the following sections, we present three methods of introducing sparsity DMs: one SST technique, Static-DM, and DST techniques, MagRan-DM. and RigL-DM.
3.1 Static Sparse Training: Static-DM
Static-DM is a sparse DM trained from scratch, with fixed connectivity between neurons. The pseudocode for Static-DM is shown in Algorithm The training process closely resembles that of dense DM, with the addition of sparse initialization step. In this step, the graph underlying the neural network is sparsified by setting fraction of the neuron connections to zero.
Algorithm Static-DM
1: Input: Dataset D, Network fe, Number of Epochs N, Diffusion steps Ta, Sparsity ratio S 2: 0 + sparse initialization using S
for = 1 to N do
To D
t~ 1011( 2,,,,,,.,aaay. Ta))
6: N~ (0. I)
7: = AdamW (Ve, CDIE (foo(f)( t), e))
8: end for
Following the findings of Liu et al. (2022), we randomly prune the connections at initialization using the Rénčs-Rényi (ER) (Mocanu et al., 2018) strategy to allocate the non-zero weights to non-convolutional layers. With this strategy, larger layers get assigned higher sparsity than smaller layers. The sparsity of each nitni- layer scales with st x x1- n where n' and i-1 represent the number of neurons in layer 1 and / - respectively.
For convolutional layers, we use a modification of ER, ERK (Evei el al., 2020), which takes into account the n'tn'=' +b' where ne and n-11 represent the number of neurons in layer and size of the kernels, s x 1 - " T. respectively, and no and /e the width and height of the corresponding convolutional kernel.
4

---

Published in Transactions on Machine Learning Research (09/2025)
3.2 Dynamic Sparse Training: MagRan-DM and RigL-DM
The key aspect of DST algorithms lies with the process of pruning and regrowing weights. We opted to test the two most common regrowth methods, random growth and gradient growth, combined with the magnitude pruming criteria. Magnitude pruning is a simple criteria, that has been shown to perform well in high sparsity regimes for supervised classification, as well as in other generative models (Nowak et al., 2023; Liu et al., 2023b)
RigL, proposed by Evei et al. (2020), combines gradient growth and magnitude pruning, thus the name of our model RigL-DM. The combination of random growth and magnitude pruning closely resembles the SET algorithm (Mocanu et al., 2018), and has been studied before for other types of models (Nowak et al., 2023), although it has never been named. For simplicity, we refer to this method as MagRan-DM.
Algorithm RigL-DM and MagRan-DM.
1: Input: Dataset D, Network fe, Number of Epochs N, Diffusion steps Ta, Sparsity ratio S, exploration frequency AT,, Pruming rate p, Sparse method METHOD
2: 0 + sparse initialization using S
3: for = 1 to N do
To D
t~ U (1,2,..... Ta))
<~ NOO, I)
7: 8: = AdamW(Vo. CDIr(fo(Oo t),
8: if i mod ATc then
9.
8,, = TopMag(((.), 1-p) p) // Magnitudo pruning
10: if METHOD is RigI-DM then
11:
0 [opGrad([eeDDil)] p) // Gradient growth
12:
elsc if METHOD is MM then
13:
8 Random(p) // Random growth
14: end if
15:
0: + update activated weights using 0i, and 0z,
16: end
17:
end for
The full pseudocode for the training process of MagRan-DM and RigL.-DM can be found in Algorithm 2. At the start of the training process, the network is sparsely initialized using the same strategy described for Static-DM. After every AT, training iterations, a cycle of connection pruning and growth is performed. First, we drop (i.e. set to zero) a fraction of the activated weights with the lowest magnitude from the network, determined using TopMag(10;), 1 1-p), which returns the indices of the top 1- of weights by magnitude. After pruning, we regrow new weights the same proportion in order to maintain the sparsity level. For RigL-DM, the connections to regrow are given by TopGrad( VeLDrrl p). that returns the indices of the top p of weights with highest magnitude gradients. For MagRan-DM the regrowth is determined by Random(p), which outputs the indices of random p of connections.
3.3 Experimental Setup
Note that our goal is not directly compare performance between models or datasets, but to compare the performance of Dense and sparse versions of the same models across different datasets, to gain insights into the impact of sparsity in DM training.
3.3.1 Models and Benchmarks
We test Static-DM, MagRan-DM, and RigL-DM against the Dense baseline, on two different DMs, Latent Diffusion (Rombach et al., 2021) and ChiroDiff (Das et al., 2023), given their popularity among the rescarch literature, on the task of unconditional image generation. Although image generation the most common application and main direction of current. research in DMs, seek to offer more extensive look, and
5

---

Published in Transactions on Machine Learning Research (09/2025)
Denoising process with UNet
Fonward process
3
D
xT
Sparsified
(a) Latent Diffusion: Sparsity applied to the U-Net, leaving the autocncoder parts (E, D) fully dense.
Denaising
with bi-directional GRU encoder
Gru Block
Gru Block
æ2
process
Dense
ft
Sparsified
(b) CiiroDiff: Sparsity applied throughout the whole network.
Figure Sparsification of Latent Diffusion (1a) and ChiroDiff (1b) models.
examined DMs for different modalities, with different backbone architectures. More detailed information about the model architectures and choice of datasets can be found in Appendix B.
Latent Diffusion. Latent Diffusion is a DM that creates high-quality images while reducing computational requirements by training in a compressed lower-dimensional latent space. Although we focus on unconditional generation tasks, Latent Diffusion also allows for conditional generation, by using a general-purposo mechanism based on cross allention (Vaswani al., 2017). Latent Diffusion first employs pre-trained auloencoders obtain latent representation of the input, and then performs the diffuusin process on these representations, using U-Net (Ronneberger al., 2015). Performing the denoising process in the latent space allows to the model to focus relevant semantic-wisg information about the data. We sparsify only the U-Net model, shown in Figure la, and utilize off-the shelf autoencoders provided by Rombach et al. (2021), keeping them dense. We evaluate on the Imagenette (Howard, 2019) datasots.
(Yu et al., 2015), CelebA-HQ (Karras et al., 2018) and
ChiroDiff. ChiroDiff is a DM specifically designed to model continuous-time chirographic data, such as sketches or handwriting, in the form sequence strokes containing both spatial and temporal information. ChiroDiff can handle sequences of variable length and, as a non-antoregressive model, able capture holistic concepts, leading to higher quality samples. This model employs a Bidirectional GRU encoder backbone architecture. The encoder is fed the spatial coordinates, their point-wise velocities, well the entire sequence context, which provides full context of the sequence during the generation process. Sparsity is applied to the entire network, as shown in Figure 1b. We evaluate it on KanjiVG, QuickDraw (Ha & Eck, 2018), and VMNIST (Das et al., 2022). Following the original paper, we use a preprocessed version of KanjiVG. For Qrw we use the following categories: crab, cat, and mosquito; and all results are averaged.
3.3.2 Experimental Details
We train the models set of sparsity rates S E (0.1, 0.5, 0.75, 0.9). For DST methods, set the exploration frequency AT, 1100 for all Latent Diffusion datasets, and AT, 800 for all ChiroDiff datasets.
htttp:///itthub.ccommbaae http:/shhhhh.cccaaaaaaan//ttttte wtt.ennncctttttttteeeetttrraaaaa

---

Published in Transactions on Machine Learning Research (09/2025)
The weight prune and regrowth ratio was set to p 0.5 for all main experiments. These values of AT, and p were based on small random search experiment.
Due to computing limitations, we use 12500/500 training/ training/validation validation images for CelebA-HQ and 10598/2500 images for SSUN-Bedrooms. In Appendix C we conduct experiments using a selection of models with the full CelebA-HQ dataset to demonstrate that using more data does not greatly influence the results. Further, Appendix D we perform experiments using the ImageNet-1k dataset which contains over 1M images.
To be able to compare the performance of different methods and different sparsity levels, train the models for predefined amount of epochs: 150 for Latent Diffusion datasets, and 600 for ChiroDiff datasets. For a complete description of training details please refer to Appendix B.3. Given the extensive number of experiments we conducted, we opted for a shorter training regime. For sampling, we use DDIM sampling (Song et al., 2021) with 100 steps for Latent Diffusion, and 50 steps for ChiroDiff, following the guidance provided in the original papers.
For completeness, we also conducted some experiments on conditional DMs, specifically class-conditional ImageNet and class-condittonal QuickDraw. See Appendix E for details this setup.
For our experiments, we performed approximately 620 training runs of Dense, Static-DM, RigL-DM, and MagRan-DM models, using two high-performance computer (HPC) clusters equipped with NVIDIA Tesla V100 SXM2 and A100 GPUs. Each DM was trained on only one GPU. All experiments consumed around 6, 900 GPU hours.
3.3.3 Evaluation Metrics
We follow common practice and calculate the FID score (Heusel et al., 2017) to assess the performance all models. Refer to Appendix B.4 for more information on FID calculation. For evaluation completeness, we also report the Kernel Inception Distance (KID) (Binkowski et al., 2018), with results presented in Appendix I. To evaluate the computational savings of the sparse methods, we report the network size (number of parameters) proxy for memory requirement, and the FLOPs, to estimate the computational cost of training and inference. We follow the method of FLOPs calculation described by Evci et al. (2020).
4 Experimental Results
We analyze the performance of Static-DM, MagRan-DM, and RigL-DM across various sparsity levels, and compare the results against. the original Dense baseline. Additionally, we perform experiments regarding the training dynamics of Dense vs sparse models. Later on, in Section 4.4 we present experiments comparing a selection of DST Dense models across various diffusion timesteps. Examples of the generated samples be found in Appendix J. Results from the class-conditional experiments provided in Appendix E.
4.1 Latent Diffusion
The results of the studied sparse methods for Latent Diffusion shown in Figure 2. For CelebA-HQ, 50% of the connections can be removed with minimal to no loss in image quality. With higher sparsity level of 75,, the three methods still perform comparably to the Dense model, especially Static-DM. However, when the network is very sparse, S = 0.9, all models fail to generate high-quality data.
a similar overall trend can be observed: performance steadily increases with decrcase sparsity level until 25%. Interestingly, MagRan-DM with 5=0 0.1 shows worse performance than the Dense model, and also significant decrease compared to MagRan-DM with S 0.25. While this goes against the general expectation that more sparsity leads to increasingly worse performance, might be rolated to the balance between regularization and expressivenes of the model. When the sparsity is low, the regularization benefits are not very strong, and the model might suller from a loss of expressiveness due to reduction in parameters, thus obtaining worse results. As such, MagRan-DM with S 0.25 likely striking a better balance between these two factors. This behaviour can be obscrved all three datascts, although less pronounced in CelebA-HQ. However, exploring this topic in depth beyond the scope of this paper.
On
intuition is that this
7

---

Published in Transactions on Machine Learning Research (09/2025)
CelebA-HQ
LSUN Bedrooms
Imagenette
70
170
60
60
160
50
50
150
Stotic DM
140
RigL-DM
MagRann-M
20
120
10
10
110
75
50
25
75
25
Sparsity of Network (9))
Sparsity of Network (50
Sparsity of Network (%)
Figure 2: FID score comparisons between Dense, and Static-DM, MagRan-DM and RigI-DM with various sparsity levels, for Latent Diffusion, with prune and regrowth ratio p = 0.5. Values are averaged over 3 runs.
Imagenette experiments exhibit the same overall tradeoff between sparsity and performance, with the best results being found in 10% and 25% sparse models.
For all datasels, successfully trained at least one sparse DM that oulperforms the original Dense version. Table presents the metrics for the best sparse models for each method. In CelebA-HQ, only RigL-DM at S = 0.25 surpasses Dense performance. In LSUN-Bedrooms, both Static-DM and MagRan-DM were able to outperform it. In Imagenette, all methods were able to achieve superior performance, albeit at different sparsity levels. We note that the variance observed the models is similar when comparing dense and sparse versions in all cases.
Table 1: Performance and cost of training and testing of Dense and best Static-DM, RigI-DM, and MagRan- DM versions for Latent Diffusion. Values are averaged over 3 runs. The FLOPs of sparse DMs are normalized with the FLOPs of their Dense versions. Test FLOPS were calculated for one sample. Sparse models that outperform the Dense version are marked bold. The top-performing sparse model is underlined.
Dataset
Approach
FID SD (1)
Params
Train FLOPs
Test FLOPs
Dense
32.74 + 3.68
274.1M
9.00e16
1.92e13
Static-DM, S = 0.5
33.19 1 2.39
0.50x
0.68x
0.68x
CelebA-HQ
RigI-DDM, S 0.25
32.12 + 3.10
0.75x
0.91
0.91
MagRan-DM, S 0.5
32.83 + 1.68
0.50x
0.67x
0.67x
Dense
31.09 = 12.42
274.1M
7.64e16
1.92e13
Static-DM, 5 = 0.25
28.79 + 12.65
0.75x
0.91
0.91
Bedrooms
RigL-DM, S = 0.10
37.80
13.55
0.90x
0.97x
0.97x
MagRan- DM, S = 0.25
28.20 + 7.64
0.75x
0.91
0.91
Dense
123.42 + 4.25
274.1M
6.83e16
1.92e13
Static-MM, 5 = 0.10
119.92 + 5.94
0.90x
0.97x
0.97x
Imagenette
RigL-DM, S = 0.10
121.59 + 6.91
0.90x
0.97
0.97x
MagRan-DM, 0.25
117.32 + 8.52
0.75x
0.91
0.91
Memory and computational savings. In Table 1, we can observe that the top-performing sparse DM on CelebA-HQ, RigL-DM with $ = 0.25, is able to outperform Dense performance, while reducing by 25% the number parameters and 10% the mumber FLOPs. Although Static-DM = 0.5 and MagRan-DM 5 0.5 achieve slightly inferior performance, they are able reduce FLOPS and number of parameters more significantly, by 30% and 50% respectively. On LSUN-Bedroome and Imagenette, the top-performing sparse DM reduces number of FLOPs by 10%, and number of parameters by 25%.
Prune and regrowth rate experiments. In all datasets, Static-DM has better performance than the dynamic methods in higher sparsity setups, S 0.5. This is interesting, it departs from the usual patterns

---

Published in Transactions on Machine Learning Research (09/2025)
found in sparse-to-sparse training for supervised learning applications and even other generative models such as GANs, where DST usually outperforms SST (Mocanu et al., 2018; Liu et al., 2023b). Liu et al. (2021c) found that, in image classification tasks, DST models consistently achieve better performance over SST with appropriate paramcter exploration, i.c., exploration frequency ATe and prune and regrowth ratio p. To provide insights on the importance of p for DST experiments, we conducted an experiment using a prune and regrowth rate p E (0.05, 0.1, 0.3, 0.53. The results are provided in Figure 8 in Appendix H. The best. results were obtained with p - 0.05.
Following this experiment, repeated all experiments for DST methods presented in Figure 2, using p 0.05, and show the results in Figure 9 and Appendix H. One particularly interesting finding is that, in high sparsity regimes, such as S = 0.9 and S = 0.75, DST methods have consistently better performance when = 0.05, even outperforming Static-DM. However, this performance advantage disappears when using the more aggressive prune and regrowth rate of p = 0.5. Please refer Appendix H for more in-depth analysis.
4.2 ChiroDiff
QuickDraw
KanjiVG
VMNIST
40
40
60
36
36
56
32
32-
52
Dense
2
RigL- DM
28
48
MagRam-DM
90
75
50
25
10
90
75
50
25
10
Networkc sparsity 1%)
Network sparsity
Network sparsity (%5
Figure 3: FID score comparisons between Dense, and Static-DM, MagRan-DM and RigL-DM with various sparsity levels, for ChiroDiff, with prune and regrowth rate p = 0.5. Values averaged over 3 runs.
Figure 3 shows the FID scores of the studied sparse methods for ChiroDiff. For QuickDraw, we observe that both Static-DM and RigT-DM exhibit variations around the performance of the Dense model, with only a subtle tendency to deteriorate sparsity increases. MagRan-DM consistently matches the FID of the Dense model, and is able to outperform it at 90% sparsity. These results suggest that this model is overparameterized, which would explain why it benefits significantly from sparsity, even when removing 90% of the weights. We repeated the same experiments using smaller version of the model, the results of which are shown in Appendix F. Although most sparse models still perform comparably to the dense version, the trend of diminishing performance with increased sparsity apparent, in contrast to the results observed with the larger model.
On KanjiVG, the impact of sparsity is more pronounced, all three methods demonstrate downward trend in performance as sparsity increases. Dynamic methods have consistently better performance than Static-DM, and RigL-DM exhibits top performance in all sparsity levels except for S == 0.75.
In VMNIST experiments, there is, again, a pattern of better performance as sparsity decreases. Similarly Latent Diffusion experiments, SST has better performance higher sparsity settings, S> 0.5. In this dataset, there is slighter larger gap in performance between the sparse and dense models.
We successfully trained at least sparse DM from each method that demonstrates comparable performance to the Dense counterpart, and show the results on Table 2. RigL-DM was the top-performing method on QuickDraw, with S = 0.1, and on KanjiVG, with S = 0.25, while in VMNIST, the top method was MagRan-DM, with 5 = 0.10. For QuickDraw, the lop sparse DM was able to outperform the Dense network.
Memory and computational savings. Table 2 shows that the top-performing sparse DM on KanjiVG achieves a reduction in the number of parameters and FLOPs of about 30%, while achieving a similar FID score. On Quickdraw, MagRan-DM with 90% sparsity achieves considerable reduction of 88%, and even
9

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Performance and cost of training and testing of the Dense and best Static-DM, RigL-DM, and MagRan-DM for ChiroDiff. Values averaged over runs. The FLOPs of sparse DMs are normalized with the FLOPs of the dense versions, and test FLOPS were calculated for one sample. Sparse models that outperform the Dense version are marked in bold. The top-performing sparse model is underlined.
Dataset
Approach
FID + SD (L) Params
Train FLOPs
Test FLOPs
Dense
29.78 + 0.59
736027
5.12 el4
1.29 e10
Static-DM, 0.25
29.39 + 0.24
0.75x
0.75x
0.75x
QuickDraw
RigI-DM, S = 0.10
29.38 + 0.27
0.89x
0.89x
0.89
MagRan-Diie DM, S 0.90
29.45 0.39
0.10x
0.10x
0.10x
Dense
21.10 + 0.25
416859
1.80133
7.35 e9
Static-DM, 5 0.5
22.36 = 0.87
0.50x
0.51x
0.51x
KanjiVG
RigL-DM, S = 0.25
21.14 + 0.71
0.70x
0.70x
0.70x
MagRan-DM, $ 0.25
21.73 + 1.18
0.39x
0.39x
0.39x
Dense
44.21 0.62
65019
1.69e12
7.11 e8
Static-DM, S = 0.25
47.29 = 1.96
0.75x
0.74x
0.74x
VMNIST
RigI-DM, S 0.10
46.81
0.90x
0.90x
0.89x
-Mg, S = 0.10
46.00 11.1
0.90x
0.89x
0.89x
though it is not the top-performing sparse model, it also outperforms the Dense model. The top sparse model VMNIST, provides reduction in FLOPs of about 89%.
Prune and regrowth rate experiments. Similar to Latent Diffusion, we repeated the DST experiments using the more conservative prune and regrowth rate of 0.05. The biggest improvement was seen in the Quickdraw dataset, where DST methods obtained considerably higher performances, as compared with Figure 3. Akin to the Latent Diffusion results, in higher sparsity regimes DST models mostly obtain better performance when using p 0.05. Please refer to Appendix H for more in-depth analysis.
4.3 Training Dynamics
In addition to final FID scores, we analysed training dynamics to better understand the behaviour of sparse models during training. In order to perform this analysis, we selected the sparsity level that achieved the overall best results in previous experiments, S = 0.25, and plotted the FID scores across several epochs.
In general, sparse models appear to follow the trend of the corresponding dense model, which appears to indicate that they retain the stable training behaviour of dense diffusion models.
CelebA-HO
LSUN Bedrooms
Imagenette
150
10C
260
120
120
220
200
80
80
180
60
40
140
of Fporhs
Fpochs
Fpnchs
S=0.25
S=0.25
MagfannDM, Sw0.25
Figure 4: Comparison of training dynamics between Dense, and Static-DM, MagRan- DM and RigL.DDM with
S = 0.25, for Latent Diffusion.
10

---

Published in Transactions on Machine Learning Research (09/2023)
QuickDraw
KanjiVG
VMNIST
38
58-
34
54
30
26
26-
40-
22
22
150
300
450
603
150
330
450
600
150
300
450
600
Epochs
Epochs
Esochs
Dense
Static DK, S-0.25
RigLOM, S-0.25
MagRan DN, S-0.55
Figure 5: Comparison training dynamics between Dense, and Static-DM, MagRan-DA DM and RigL-DM with
S 0.25, for ChiroDiff.
4.4 Impact of Diffusion Steps
The number of timesteps is an important parameter in DMs, as too few can lead to insufficient denoising, and low quality images, while too many might increase computational complexity without improving output quality. We explored the relationship between the number of timesteps and model sparsity, aiming to determine whether very sparse model (S 0.75) with an increased number of sampling steps achieve performance comparable to that of dense model, with less sampling steps. We perform experiments using CelebA-HQ for Latent Diffusion, and KanjiVG for ChiroDiff, the results of which are presented in Figure 6.
CelebA-HO
KanjiVG
50
30
45
28
40
26
35
24
25
20
50
100
150
25
50
75
Diffusion
Diffusion
Dense
StalicDDM, 5=0.75
MagRan-DM, 5m0.75
RigL-DM,, 5=0.75
Figure 6: FID score comparisons between Dense, and Static-DM, MagRan- DM and RigL-DM with S 0.75, using varied diffusion timesteps for Latent Diffusion (CelebA-HQ), and ChiroDiff (KanjiVG). Values averagod over runs.
In general, the mumber of sampling steps does not affect when comparing sparse and dense versions within the same number of timesteps. More experiments are presented in Appendix G. In KanjiVG, sparse model is able to match any version of the dense model, and varying the number of timesteps appears to have little influence on the quality of the output. In CelebA-HQ, when comparing different numbers of limesteps, we observe that MagRan-DM and Static-DM with both 100 and 150 timesteps are able to outperform the Dense model using timesteps. As an example, in Static-DM, with 50 timesteps, Static-DM offers theoretical speedup of 0.29x over the dense model's Training FLOPs, and 0.57x of the Testing FL.OPs, while creating better quality samples.
0.75 with 100 timesteps vs. the dense model
11

---

Published in Transactions on Machine Learning Research (09/2025)
4.5 Limitations and Future Work
Apart from the previously mentioned computational limitations when training on CelebA-HQ and LSUN- Bedrooms, our findings demonstrate systematic trends that prompt for further investigation. Training for longer epochs could provide deeper insights into the capabilities of sparse models. Additionally, there is potential in exploring other pruning strategies and other DST hyperparameters such ATc- Another interesting direction is to adjust DST hyperparameters based on the training phase, in response to changes in training dynamics. Furthermore, employing multiple sparsity masks with varying sparsity levels and dynamically changing them during training, according to the denoising timestep, is a promising line of research.
Table 3: Overview of dense vs sparse Latent Diffusion (LD) and ChiroDiff (CD) models. Experiments where the sparse model outperforms the dense version are marked in grey.
Dense
Best Sparse model
FLOPs
Model
Dataset
FID
Model (S,p)
FID
Train & Test
LD
CelebA-HQ
32.74
RigL-DM (25%, 0.5)
32.12
0.91x
LD
Bedrooms
31.09
MagRan-DM, (10%, 0.05)
25.12
0.97x
LD
Imagenette
123.42
MagRan-DM (25%, 0.5)
117.32
0.91
CD
QuickDraw 29.78
RigL-DM (25%, 0.05)
24.91
0.75
CD
Kanji- VG
21.10
MagRan-DM. DM (50%, 0.05)
20.32
0.51 X
VMNIST
44.21
MagRan-DM (10%, 0.5)
46.00
0.89x
5 Conclusion
We have introduced sparse-to-sparse training of DMs. Our experiments show that both SST and DST methods able to match and often outperform the dense DMs, shown in Table 3, while reducing memory and computational costs. We highlight the importance of choosing the correct method and sparsity level, depending on the model (and even the dataset) that is being used. Taken together, our findings show the great potential of sparse-to-sparse training in improving the efficioncy of both training and sampling from DMs.
Open Science: Our code and models are available https://gtttubb.cm/////l///////t diffusion
sparse
Acknowledgements
Research supported by Fonds National de la Recherche Luxembourg FNR (SCRIPTOR project, grant AFR/22/17117001) and the Europcan Innovation Council Pathfinder program (SYMBIOTIK project, grant 101071147).
The experiments presented in this paper were carried out using the HPC facilities of the University of Luxembourg /attps://ppo http::/hhpc..i.nnn. mni and Luxembourg's national supercomputer McluXina. The authors gratefully acknowledge the ULHPC and LuxProvide teams for their expert support.
References
Abhinav Agarwalla, Abhay Gupta, Alexandre Marques, Shubhra Pandit, Michael Goin, Eldar Kurtic, Kevin Leong, Tuan Nguyen, Mahmoud Salem, Dan Alistarh, Sean Lie, and Mark Kurtz. Enabling high-sparsity foundational llama models with efficient pretraining and deployment. ar Xiv, abs/2405.03354, 2024.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Zahra Atashgahi, Ghada Sokar, Tim van der Lee, Elena Mocanu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pecheniskiy. Quick and robust feature selection: the strength of energy-effiiient sparse training for autoencoders. Machine Learning, 2022.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In Proc. International Conference on Learning Representations (ICLR), 2018.
Mikolaj Binkowski, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. comm-paramctric neural image synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffsse-er faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022.
Selima Curci, Decebal Constantin Mocanu, and Mykola Pochenizkiyi. Truly sparse neural networks at scale. arxiu, abs/2102.01732, 2022.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. SketchODE: Learning neural sketch representation in continuous time. In Proc. International Conference on Learning Representations (ICLR), 2022.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. ChiroDiff: Modelling chirographic data with diffusion models. In Proc. International Conference on Learning Representations (ICLR), 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2009.
Tim Detimers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv, abs/1901 0bs//997.0448/ 2019.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Zheng Ding, Mengqi Zhang, Jiajun Wu, and Zhuowen Tu. Patched denoising diffusion models for high- resolution image synthesis. arv, abs/2308.01316, 2023.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lotter:: making all tickets winners. In Proc. International Conference on Machine Learning (ICML), 2020.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proc. International Conference on Learning Representations (ICLR), 2019.
Songwei Ge, Vedanuj Goswami, C. Lawrence Zitnick, and Devi Parikh. Creative sketch generation. arXie, abs/200 abss/2011.00099 2020.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In Proc. International Conference on Learning Representations (ICLR), 2023.
David Ha and Douglas Eck. A neural representation of sketch drawings. In Proc. International Conference Learning Representations (ICLR), 2018.
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficientt diffusion training via min-snr weighting strategy. arXiv, hos/230330955 2024.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANSs trained by a two time-scale update rule converge to local Nash equilibrium. In Proc. Aduances in Neural Information Processing Systems (NewrIPS), 2017.
Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXis, abs/2006.1111 222/2200611222 2020a.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. In Proc. Advances in Nearal Information Processing Systems (NewrIPS), 2020b.
Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from imagenet, 2019. URL http://gttttttt../ http:://g.ttuu..coo/aasts/a///// hatp://cc///sttttmaaeeeeee immggeette..
Chao Jiang, Bo Hui, Bohan Liu, and Da Yan. Successfully applying lottery ticket hypothesis to diffusion model. arXiv, Abs/2310.18825, 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proc. International Conference on Learning Representations (ICLR), 2018.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space liffusion-based generative models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based on conncction sensitivity. In Proc. International Conference on Learning Representations (ICLR), 2019.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-La LM improves controllable text generation. In Proc. Advances in Neurol Information Processing Systems (NeurIPS), 2022.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proc. IEEE/CW International Conference on Computer Vision (ICCV), 2023.
Sean Lie. Cerebras architecture deep dive: First look inside the hw/sw co-design for deep learning: Cerebras systems. In Proc. IEEE Hot Chips 34 Symposium (HCS), 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley, AudioLDM: Text-to-audio generation with latent diffusion models. Proc. International Conference on Machine Learning (ICML), 2023a.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkly, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Proc. Advances in Neural Information Processing Systems (NeurTPS), 2021a.
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Xiv, abs/1901.0911
2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense nonnpaaraneterzzatio??? in-time wee-paraaetteriaaiion in sparse training. arxin, abs/2102.02887, 2021c.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and
Mykola Pechenizkiy. The unrcasonable offectiveness of random pruning: Return of the most naive bascline for sparse training. In Proc. International Conference on Learning Represenlations (ICLR), 2022.
Shiwei Liu, Yuesong Tian, Tianlong Chen, and Li Shen. Don't be so dense: Sparso-to-sparse gan training without sacrificing performance. International Journal of Computer Vision, 2023b.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Decebal Mocanu, Elena Mocanu, Phuong Nguyen, Madelcine Gibescu, and Antonio Liotta. A topological insight into restricted boltzmann machines. Machine Learning, 2016.
Decebal Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 2018.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization arXiv, abs/1902.05967,, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proc. International Conference on Machine Learning (ICML), 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. International Conference on Machine Learning (ICML), 2022.
Alcksandra Nowak, Bram Grooten, Decebal Constantin Mocanu, and Jacck Tabor. Fantastic weights and how to find them: Where to prune in dynamic sparse training. In Proc. Advances in Neural Information Processing Systems (N)urIPS), 2023.
Hao Phung, Quan Dao, and A. Tran. Wavelet diffusion models fast and scalable image generators. Tn
Pro. TEEE/CT CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2022.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. Proc. International Conference on Machine Learning (ICML), 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. arXiv, abs/2112.10752,, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arxvv, abs/2104.07636, 2021.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffiusion models. In Proc. ACM SIGGRAPH, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Proc. International Conference on Learning Representations (ICLR), 2022.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proc. IEEE/CVE Conference on Computer Vision and Pallern Recognation (CVPR), 2023.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv, abs/1103.000 035585, 2015.
Chada Sokar, Flena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning. In Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, In Proc. International Conference Learning Representations (ICL.), 2021.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011.13456, 1.14466, 2020.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. Proc. AAAI Conference on Artificial Intelligence, 2020.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score- based diffusion models for probabilistic time series imputation. In Proc. Advances in Neural Information Processing Systems (NewrIPS), 2021.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proc. Advances in Neural Information Processing Systems (NewrIPS), 2017.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In Proc. International Conference on Learning Representations (TCLR), 2020-
Kafeng Wang, Jianfei Chen, He Li, Zhenpeng Mi, and Jun Zhu. Sparsedm: Toward sparse efficient diffusion models. ArXin, abs/ 0b8/2404.10445, 2024.
Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proc. IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T Freeman, and Tacsung Park. One-step diffusion with distribution matching distillation. In Proc. IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2024.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. ArXin, abs/1506.03365, 2015.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang Hc, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, and Xue Lin. Mest: Accurate and fast memory-econonic sparse training framework on the edge. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Yinglao Zhang, Jialin Zhao, Ziheng Liao, Wenjing Wu, Umberto Michieli, and Carlo Vittorio Cannistraci. Brain-inspired sparse training in mip and transformers with network science modeling via cannistraci-hebb soft rulc. Preprints, 2024a.
Yingtao Zhang, Jialin Zhao, Wenjing Wu, Alessandro Muscoloni, and Carlo Vittorio Cannistraci. Epitopo- logical learning and cannistraci-hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning. In Proc. International Conference on Learning Representations (ICLR), 2024b.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n:m fine- gra-ne structured sparse neural networks from scratch. ArXiv, abs/2102.044 .40010, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2025)
A Hardware and Software Support
One of the main challenges sparse neural networks research is that most hardware optimized for deep learning designed for dense matrix operations. As result, most of current research attempts to mimic sparsity by using a binary mask over weights, which results in sparse networks offering, in practice, no better training efficiency than dense networks. However, industry is catching up and it is matter of time for hardware to truly leverage sparse operations.
There is a growing trend towards developing hardware that better supports sparse operations. In 2021, NVIDIA released the A100 GPU, which supports accelerating operations in a 2:4 sparsity pattern. Several works have already leveraged this feature (Zhou et al., 2021; Wang et al., 2024). In order to use this capability, the sparse matrices must follow specific structure: among each group of four contiguous values, two values must be zero, thereby fixing the sparsily level at 50%. While this structure enables significant acceleration, it supports only static sparsity level, and makes it impossible to vary the sparsity ratio between layers.
More recently, Cerebras introduced the CS-3 AI accelerator (Lie, 2022), capable of accelerating sparse training and supporting unstructured sparsity. Using Cerebras' CS-3 AI to accelerate Training, and Neural Magic's inference server to accelerate inference, Agarwalla et al. (2024) trained an accurate sparse Llama-2 7B model. Its accelerated training closely matched the thcoretical spcodup, while achieving 91.8% accuracy recovery of Llama Evaluation metrics, with 70% sparsity. This significant finding underscores the potential of sparse training to produce more efficient. neural networks in practice, not just in theory.
In parallel, there have also been advancements in creating software implementations that support truly sparse-to-spars, neural network training, mostly for supervised learning tasks (Liu et al., 2021b; Curci et al., 2022). In addition, a. sparse-to-sparse denoising autoencoder has been developed by Atashgahi et al. (2022), to perform fast and robust feature selection.
These developments in both hardware and software point towards future where sparse-to-spare training may become the de facto approach for developing neural networks, enabling faster, more memory-efficint,, and energy-efficiett deep learning models.
B Experiments setup
B.1 Model Architectures
In Latent Diffusion experiments, the model architecture is the same for L.SUN-Bedrooms, CelebA-HQ and Imagenette datasets. The DM follows the architecture proposed by Rombach et al. (2021). For the autoconder, we utilize pre-trained model released by the Latent Diffussio authors on the project's GitHub," 2 with spatial size 64x64x3, VQ-reg regularization, and downsampling factor ʃ f= 4.
In ChiroDiff experiments, we adopt the architecture proposed by Das al. (2023). The backbone network is a bidircctional GRU encoder with 3 layers, with 96 hidden units for KanjiVG, and 128 hidden units for QuickDraw. For VMNIST, the backbone network is 2-layer bidirectional GRU encoder with 48 hidden units. We also use the code available on the project's GitHub repository.
B.2 Choice of Datasets
We evaluated Latent Diffusion on L.SUN-eerroos and CelebA-HQ, following their use the original paper. Additionally, we included Imagenette, a subset of the popular ImageNet (Deng et al., 2009) dataset. For ChiroDiff, we used the same datasets evaluated the original study: QuickDraw, KanjiVG and VMNIST. While the authors of ChiroDiff analysed seven categories of QuickDraw, namely (cat, crab, mosquito, bus, fish, yoga, flower), we opted to reduce the number of categories to {cat, crab, mosquito) given the large number of experiments involved in our investigation. In Appendix C below we demonstrate that dataset.
httt:://ggthub..cc///mm///////ss http::::/ggtuuuu.cc///aaaaaaaaap
17

---

Published in Transactions on Machine Learning Research (09/2025)
size does not change the main outcomes. Ultimately, our goal is to compare and contrast sparse and dense models, independent of dataset size.
B.3
Training Regime
We follow the configurations provided the GitHub repositories of the original papers and present the main aspects below. The only alterations made were in the batch size and learning rate. The only exception is Imagenette, which was not included in the original paper; for this dataset, we applied the same configuration settings as those used for
Latent Diffusion on LSUN-Bedroonss We use batch size of 12, AdamW optimizer with weight decay le-2 and static learning rate 2.4e-5. We train for 150 epochs. We use 1000 Denoising steps (T), linear noise schedule from 0.0015 to 0.0195, and simusoidal embeddings for the timestep.
Latent Diffusion on CelebA-HQ: We use a batch size of 12, AdamW optimizer with weight decay 1e-2 and static learning rate 2.0e-06. We train for 150 epochs. We use 1000 Denoising steps (T), linear noise schedule from 0.0015 to 0.0195, and simusoidal embeddings for the timestep.
Latent Diffusion on Imagenette: We use a batch size of 12, AdamW optimizer with weight decay le-2 and static learning rate 2.4e-5. We train for 150 epochs. We use 1000 Denoising steps (T), linear noise schedule from 0.0015 to 0.0195, and sinusoidal embeddings for the timestep.
ChiroDiff on QuickDraw: We use batch size of 128, AdamW optimizer with weight docay 1c-2 and static learning rate 1e-3. We train for 600 epochs. We use 1000 Denoising steps (T), linear noise schedule from 1c-4 to 2e-2, and random Fourier features for the timestep embedding.
ChiroDiff on KanjiVG: We use batch size of 128, AdamW optimizer with weight decay 1e-2 and static learning rate 1e-3. We train for 600 epochs. We use 1000 Denoising steps (T), linear noise schedule from le44 to 20-2, and random Fourier features for the timestep embedding.
We use a batch size of 128, Adamn optimizer with weight decay 1e-2 and static learning rate 1e-3. We train for 600 epochs. We use 1000 Denoising steps (T), linear noise schedule from 1c-4 to 2e-2, and random Fourier features for the timestep embedding.
ChiroDiff on VMNIST:
Each setup was trained for 5 sparsity values 00.1, 0.25, 0.5, 0.75, 0.9), and we perform 3 runs for each model/ model/dataset/ /atasat///aaeeeeeees /at/asttspaassstttyyyy combination. For ChiroDiff on QuickDraw, we trained each category {cat, crab, mosquito} for 3 runs, resulting in a total of 9 runs per sparsity level.
B.4 FID calculation
For Latent Diffusion, FID is calculated using the torch-fidelity Python package, and estimated based on 10k samples and the entire training set, as in the original work. For ChiroDiff, following the original paper, we plot and save the chirographic sequences as images, and calculate the FID using the inception model provided by Ge et al. (2020), pre-trained on the QuickDraw dataset, using 10k generated samples and 20k real samples.
C Experiments using the Full CelebA-HQ Dataset
We conducted experiments using Static-DM, MagRan-DM, and RigL-DM with S = 0.5 on the full CelcbA-HQ dataset, for 150 epochs, and compare the results with the previous models trained on 50% of the dalaset. As shown in Table 4, the FID scores are similar across both datasets for each respective method. This supports our docision to focus on a subset of the datasct for our main experiments, to save valuable computational resources. Interestingly, all sparse models are able outperform their dense version when trained on the full dataset.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Table Comparison of FID and KID scores for Latent Diffusion on CelebA-HQ using full dataset vs. reduced datasct. Results based the first run. Sparse models that outperform their Dense version are marked in bold. The lop-performing sparse model is underlined.
Methods
FID (L)
KID (L)
Full dataset
Reduced dataset
Full dataset
Reduced dataset
Dense
32.20
29.68
0.0259
0.0241
Static-DM, S 0.50
29.71
29.91
0.0242
0.0243
Rigl-DM, S = 0.50
30.98
30.82
0.0250
0.0254
MagRan- Mag S = 0.50
26.70
30.71
0.0208
0.0253
Experiments using the Full ImageNet-1k Dataset
In this section, we present the results for the most promising sparse DM trained with the ImageNet 1k dataset, comprising 1000 classes spanning 1, 28, 167 training images, 50, 000 validation images and 100, 000 test images.
Table 5: FID and KID scores for Latent Diffusion on ImageNet.
Methods
FID (L)
KID (L)
Dense MagRan- DM, S = 0.50
63.95
0.0538
77.39
0.0714
E
Experiments using Conditional Models
While the main focus of this paper is investigating how sparso-to-sparse training affects unconditional models, we also performed some initial experiments conditional models to explore its applicability in this setting. Specifically we conducted experiments on coas-cooniiional ImageNet using Latent Diffusion, and class-conditinnal QuickDraw using ChiroDiff.
For these experiments, models were trained for 50 epochs on ImageNet and 150 on Quickdraw. For ImageNet, all classes were utilized in training and sampling was performed using 1=1100 and classifier-free guidance scale of 3.0. For QuickDraw, seven classes were used: cat, crab, mosquito, bus, flower, yoga and fish. Examples of generated samples can be found in Figure 21 and Figure 22 in Appendix J.
On class-conditional ImageNet, the dense model outperforms the sparse version, with moderate gap between them. On Quickdraw, models show extremely similar performance, with the best results achieved by MagRan-DM with 50% sparsity. These results suggest that sparse-to-sparse training may also effective for conditional models.
Table 6: FID and KID scores for Latent Diffusion on class-conditional ImageNet.
Methods
FID (L)
KID (L)
Dense
17.33
0.0071
MagRan-DM, S 0.50
20.33
0.0088
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table 7: FID and KID scores for ChiroDiff on conditoon QuickDraw. Sparse models that outperform
their Dense version marked in bold. The top-performing sparse model is underlined.
Methods
FID (L)
KID (L)
Dense
31.23
0.0327
Static-DM, 0.50
31.57
0.0310
MagRan-DM, 5 0.50
30.20
0.0300
RigL-DM, S 0.50
32.12
0.0315
F Experiments using a smaller model
In order explore the strong results of high sparsity models (S 0.90) on QuickDraw using ChiroDiff, we repeated the experiments in Figure 3, using smaller model. In this smaller model, the backbone network is bidircctional GRU encoder consisting of layers with 96 hidden units cach, compared to 128 in the larger model.
In contrast with the larger model, the trend of diminishing performance as sparsity increases, present in the other datascts, can be observod. Most sparse models perform comparably to the dense, although some performance degradation is apparent in high sparsity models.
QuickDraw
0.038
32
0.036
Dense
2
0.034
RigL-DM
0.032
MagRan-DM
0.030
24
0.028
0.026
20 75
90
50
25
10
90
75
50
25
10
sparsity 196)
sparsity 196)
Figure 7: FID and KID scores comparisons between Dense, and Static-DM. DM, MagRan-DM and RigT-DMM with various sparsity levels, for QuickDraw dataset, with prune and regrowth rate p = 0.5.
G
Experiments using Various Diffusion Timesteps
In Table 8, we report the results of the models listed in Table using 50, 100 and 200 sampling steps. These experiments confirm that the number of sampling steps typically does not affect whether sparse model outperforms dense model. In other words, sparse model that performs better than dense model at 100 timesteps also outperforms it at 50 and 200 timesteps.
For CelebA-HQ, the variation in timesteps does not change the top- nop-performing model, which consistently RigL-DM with 0.25. However, in BeUr--ed,,,,s, the top-performing method varies with different. timesteps.
H Prune and Regrowth Rate Experiments
To provide insight on the importance of the prune and regrowth rate for DST experiments, we conducted experiment using varying values, with the top MagRan-DM and RigL-DM models for the CelebA-HQ
20

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: Comparison of FID scores for models listed in Table 1 using various DDIM sampling steps. Results based the first Sparse models that outperform the Dense version, in the respective sampling steps, are marked in bold. The top-performing sparse model for each sampling step is underlined.
Datasct
FID (4)
50 steps
100 steps
200 steps
Dense
38.14
29.68
26.47
Static-DM, S 0.5
38.16
29.91
26.44
CelebA-HO
RigL-DM, S = 0.25
36.55
28.00
25.25
MagRan- Ro,, S = 0.5
39.31
30.71
28.02
Dense
20.42
20.14
20.58
Static-DM, S 0.25
20.01
18.96
19.26
LSUN-Bedroomd
RigL-DM, S = 0.10
19.01
17.96
20.49
MagRan-DM, S = 0.25
20.69
18.23
17.87
dataset, listed in Table 1, We report the results of the experiments in Figure 8. Although all FID values are extremely similar, the best performing models for both algorithms use prune and regrowth ratio p 0.05, and both outperform the Dense version. This suggests that selecting an optimal ratio can improve model performance, even if only slightly in these lower-sparsity models tested.
35
MegRan-DM, $=0.5
RigI-DM, 5-0.25
Dense
0.05
0.1
0.2
0.3
0.5
Pruning
Figure 8: FID scores comparison between Dense and DST models with varions prune and regrowth ratios, for Latent Diffusion on CelebA-HQ.
Informed by the results of Figure 8, we conducted experiments for DST methods using the same setup as in Figure 2 and Figure 3, but using prune and regrowth rate of p 0.05. As can be observed in Figure 9 and Table 9, the general trend of diminishing performance when sparsity increases still remains, with the exception of QuickDraw, in which all DST models had significant increase in performance.
When looking at very high sparsity regimes, S 0.90, we observe that most models continue to suffer from a significant, performance drop when compared to their Dense version, except Quickdraw, where the new prune and regrowth provides remarkable improvement, and LSUN Bodrooms, where MagRan-DM has an impressively high performance. However, when S 0.90 (and to lower degree S 0.75) DST methods using p 0.05 have consistently better performance than their p 0.5 counterparts. When comparing the DST methods to Static-DM at $ = 0.90, in Table 9, we observe that at least one DST method is able to outperform Static-DM in CelebA-HI and LSUN-Bedrooms, or closely match it in Imagenette, which did not happen with = 0.5. Similarly, for ChiroDiff, in Table 10, almost all DST methods in all three datasels are able to outperform Static-DM.
All in all, these findings suggest that prune and regrowth ratio of 0.5 too aggressive, and that a more conservative choice of 0.05 more appropriate for DMs. Previous work has mentioned that DST methods
21

---

Published in Transactions on Machine Learning Research (09/2025)
CelebA-HQ
LSUN Bedrooms
Imagenette
70
1/0
60
60
160
50
50
150
Stotic DM
140
RigL-DM
MagRannDM
20
120
10
10
110
75
50
25
75
25
Sparsity of Network (9))
Sparsity of Network (%0
Sparsity of Network (%))
Figure 9: FID score comparisons between Dense and Sparse versions (Static-DM, MagRan-DM, RigL-DM) considering various sparsity levels for Latent Diffusion. DST method use a prune and regrowth rate of 0.05. Values averaged over 3 runs.
QuickDraw
KanjiVG
VMNIST
2
Static-DM
DM Magitan-MM
28
28
48
24
24
44
20
20
40
90
75
50
90
75
50
25
10
Network sparsity 1%61
Network sparsity (%)
Network sparsity (%)
Figure 10: FID score comparisons between Dense and Sparse versions (Static-DM, MagRan-DM, RigI-DDM) considering various sparsity levels for ChiroDiff. DST method use prune and regrowth rate of 0.05. Values averaged over 3 runs.
Table 9: Comparison of FID scores for SST (Static-DM) and DST (RigL-DM, MagRan-DM) models, with S 0.9 using two different prune and regrowth rates (p 0.5 and p 0.05) for Latent Diffusion. DST models that outperform SST are marked in bold.
Dataset
Static-DM.
RigL-DM
MagRan-DM p 0.5,p 0.05
p 0.5, p 0.05
CelebA-HQ
52.48 4.88
65.65 + 4.32, 46.07 + 11.08
60.77 + 6.58, 48.39 + 14.05
Bedrooms
46.18 13.42
71.45 18.84, 58.64 + 22.88 168.48 + 15.15, 148.93 + 12.03
46.22 + 10.11, 33.80 3.98 167.19 + 8.20, 159.08 + 14.68
Imagenette
147.47 + 7.74
Table 10: Comparison of FID scores for SST (StaticDMM) and DST (RigL-DM, MagRan-DM) models, with S = 0.9 using two different prune and regrowth rates (p 0.5 and p 0.05) for ChiroDiff. DST models that outperform SST are marked in bold.
Dataset
Static-DM
RigL-DM
MagRan-DM p 0.5, 0.05
0.5.p p = 0.05
QuickDraw
30.25 = 0.43
30.26 + 0.63, 28.84 + 0.37
29.45 + 0.39, 28.60 + 0.37
KanjiVG
30.75 2.16
28.54 + 0.74, 29.12 + 0.57
33.02 3.28, 29.01 + 1.48
VMNIST
52.35 0.84
54.08 + 1.57, 52.25 + 0.20
53.65 0.69, 51.94 + 1.12
22

---

Published in Transactions on Machine Learning Research (09/2025)
are consistently superior to SST as long there is appropriate parameter exploration (Liu et al., 2021c), observations that aligns with our findings.
Additional evaluation metrics
In this section, we present the KTD results corresponding to previously reported experiments, in Figure 2, Figure 3, Figure and Figure 10.
CelebA HQ
LSUN Bedreams
Imagenette
3.07
9.05
0.05
0.09
Static DM
Rigi-OM
0.04
0.04
0.00
MagRan-OM
0.08
0.03
0.07
0.02
0.02
0.06
Scersity Network
Sparsity Network
Spersity of Network
Figure 11: KID comparisons between Dense, and Static-DM, MagRan-DM and RigI.DDM with various sparsity levels, for Latent Diffusiin,, with prune and regrowth rate p = 0.5.
CelebA-HQ
LSUN Redrooms
Imageritte
0.06
0.06
0.11
0.05
0.05
0.10
Deriee
Static-DM
RigL-OM
0.08
Magilan-MM
0.02
0.02
0.07
0.01
0.01
0.06
50
25
10
50
25
10
90
75
50
10
Sparsity Network (%)
Sparsity of Network 196)
Sparsity of Network 195)
Figure 12: KID comparisons between Dense, and Stalic-DM, MagRan-DM and RigL-DM with various sparsity levels, for Latent with prune and regrowth rate = 0.05.
QuickDraw
KanjiVG
VMNIST
0.066
0.042
0.042
0.064
0.040
0.040
0.000
Dense
StaticDDM
0.044
0.034
0.088
RigL-DM
0.032
0.099
0.056
MagRan-3M
0.030
0.030
0.054
0.028
0.050
75
50
25
10
90
75
50
25
90
75
50
10
Network sparsity (9)
Network sparsity (%)
Network sparsity (%6
Figure 13: KID comparisons between Dense, and Static-DM, MagRan-DM and RigL-DM with various sparsity levels, for ChiroDiff, with prune and regrowth rate p = 0.5.
23

---

Published in Transactions on Machine Learning Research (09/2025)
QuickDraw
KanjIVG
VMNIST
0.0255
0.0425
0.0400
0.0400
0.0625
0.0375
0.0375
0.0000
Dense
0.0325
0.0575
RigL DM
MagRann-MM
0.0300
0.0200
0.0525
0.0275
0.0275
0.0500
90
75
50
25
90
75
50
25
10
90
75
50
Network: sparsity (9)
Network: sparsity (%)
Network sparsity 196
Figure 14: KID comparisons between Dense, and Static-DM, MagRan-DM and RigL-DM with various sparsity levels, for ChiroDiff, with prune and regrowth rate p 0.05.
J Examples of Generated Samples
Figures 15 to 20 showcase examples of samples generated by Latent Diffusion and ChiroDiff across the evaluated datasets. Examples are unconditionally sampled from the Dense and the top-performing sparse model in each case. Figures 21 and 22 present examples of class-conditional generated samples on ImageNet and QuickDraw.
24

---

Published in Transactions on Machine Learning Research (09/202)
Dense
(b) Statio-DM, 0.25
Figure 15: Samples from Latent Diffusion trained on L.SUN-eerrooms. The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the top-performing sparse model.
25

---

Published in Transactions on Machine Learning Research (09/2025)
(a) Dense
(b) Rig $ 0.25
Figure 16: Samples from Latent Diffusion trained on CelebA-HQ, The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the top-performing sparse model.
26

---

Published in Transactions on Machine Learning Research (09/2025)
(a) Dense
(b) MagRan-DM, $ 0.25
Figure 17: Samples from Latent Diffusion trained on Imagenette. The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the tor-performing sparse model.
27

---

Published in Transactions on Machine Learning Research (09/2025)
P
X
M3
i
L
(a) Denise
d
(b) RMg.. 5 ===000 0.10
Figure 18: Samples from ChiroDiff trained on Quickdraw. The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the top-performing sparse model.
28

---

Published in Transactions on Machine Learning Research (09/2025)
B
77
1
1/
24
13
x
A
x
1
1
I
2
2
x
1
I
3
-
2
2
R
>
+
h
P.
1
3
P
2
V
K
3
(a) Deniee
N
プラ
i x
7
R
X
Ax
151
-
1
1
ス
23
15
1
232
124
2 2
12
+
A
F
/
12
N=3
-
11
2
p
トサ
(b) RigL-DM, S 0.25
Figure 19: Samples from ChiroDiff trained on Kanji. The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the top-performing sparse model.
29

---

Published in Transactions on Machine Learning Research (09/2025)
2
6
1
2
5
3
3
5
2
3
8
7
9
8
6
5
6
0
9
8
2
8
5
6
S
1
S
3
6
4
7
9
5
0
7
6
7
2
2
2
8
4
7
77
5
57442
1742.33
3
3
G
9998
(a) Denise
5
6
3
4
6
7
9
7
A
7
9
6
2
1
\
1
7
6
7
2
4
3
0
A
3
6
8
8
3
8
2
10
4
q
7
2
6
2
6
3
7
7
5
F
8
5
3
6
3
5
3
(b) MagRan-DM, S 0.10
Figure 20: Samples from ChiroDiff trained on VMNIST. The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the top-performing sparse model.
30

---

Published in Transactions on Machine Learning Research (09/202))
Dense
(b) MagRan-DM, S 0.50
Figure 21: Samples from Latent Diffusion trained class-conditional ImageNet. The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the top-performing sparse model.
31

---

Published in Transactions on Machine Learning Research (09/2025)
x
A
(a) Dense
00
Vol
(b) MagRan-DM, S 0.50
Figure 22: Samples from ChiroDiff trained on class-conditional QuickDraw. The top row presents samples generated by Dense models, whereas the bottom row presents samples generated by the top-performing sparse model.
32


Reviews:
[{"review_id": "9WqqKSXk1z", "reviewer": "Reviewer_JRjh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is concise, well-written and easy to understand\n2. The work demonstrates substantial reduction in FLOPs and parameters.\n3. The experiments are conducted over a wide variety of datasets and span multiple sparsity methods.\n\nWeakenesses:\n1. No runtime benchmarks on sparse-compatible hardware, although there is discussion about A100 hardware. So, the efficiency gains remain theoretical.\n2. Conditional generation tasks are not explored. It would be interesting to see how the sparse training affects given a prompt of any modality.\n3. The method heavily relies on initialization and DST hyperparameters like prune/growth rate. A bit of ablation of these components would be interesting to see.\n4. In the chirodiff experiment on quickdraw dataset, the extremely sparse model also outperforms the dense ones. Is it possible that the dense model is not calibrated properly?\n5. The evaluation is only reported in terms of FID. It would be interesting to see other evaluation metrics like Inception Score or similar generative metrics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the concept of sparse-to-sparse training for diffusion models, enabling models to be trained from scratch with fixed or sparse connectivitons that evolves dynamically. It proposes three methods: Static-DM, MagRan-DM, and RigL-DM and evaluates them on two diffusion architectures across multiple datasets. The study demonstrates that sparse models can match or exceed dense performance while significantly reducing parameters and FLOPs. This work provides insights into efficient training strategies for generative models, addressing both computational cost and scalability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be interesting if the authors include runtime benchmarks on sparse-compatible hardware like A100 to empirically justify the practical efficiency of the method.\n2. It would be great if the authors provide one experimental result on conditional image generation. Perhaps a text-to-image task using LDM\n3. Ablation studies on initialization strategies is appreciated.\n4. The outperformance of the sparse model (magran-dm at 90% sparsity) over dense model raises a red flag. Perhaps scale down the dense model's width or depth so that it has a comparable number of params or FLOPs as the sparse model for fairness.\n5. Results on other evaluation metrics will be appreciated.", "broader_impact_concerns": "There is a broader impact statement provided by the authors which satisfactorily addresses the concerns of misuse of generative content.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n82Ja1OmFb", "reviewer": "Reviewer_fxMz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n+ **S1.**  This paper is the first to verify efficient training using a sparse-to-sparse strategy for diffusion models.\n+ **S2.** Through experiments, this paper confirms that diffusion models with sparse neural networks can achieve performance comparable to dense models.\n\n### Weaknesses\n- **W1.** As acknowledged in the paper, currently, training sparse neural networks does not benefit from hardware optimization, and even when applying the proposed method, it is not possible to reduce the training time compared to dense networks. Existing sparse-to-sparse research is based on the assumption of future hardware optimization, but the fact that no promising hardware optimization has been implemented in major frameworks even after more than five years suggests that this approach is not very promising in terms of reducing training time. To address this issue, it would be more impactful to first research hardware optimization for sparse model training rather than pursuing studies on individual models.\n- **W2.** Most of the proposed methods in this paper are naive applications of existing sparse-to-sparse strategies and lack technical novelty. In particular, dynamic sparsification (RigL-DM and MagRan-DM) is applied naively without considering the time step $t$, making it almost obvious that these methods are inferior to Static-DM because conditioning different $t$ makes models activate different neurons.\n- **W3.** It is already known from previous studies, such as GAN, that sparse-to-sparse can achieve high performance while reducing FLOPS, but the experimental results shown in this paper lack insights specific to diffusion models.\n- **W4.** There is little evidence to support the claim that sparse-to-sparse training in diffusion models is essential for achieving the optimal trade-off between performance and parameter reduction. For example, it is unclear whether the proposed method is more effective than dense models with the same parameter size after pruning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates sparse-to-sparse, a sparsification strategy for neural networks during training, to accelerate the training of diffusion models. The paper applies Static-DM, which prunes parameters before training, RigL-DM, which sparsifies during training, and MagRan-DM, which sparsifies while attempting reconnection during training, to two existing diffusion models, latent diffusion and ChiroDiff, and evaluates the post-training generation quality. The experiments confirm that sparse DM achieves FID scores competitive to dense DM. Additionally, the paper investigates the effectiveness of Static-DM at high sparsity levels and the impact of the number of time steps.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Explain the optimality of the proposed dynamic sparse training. The training dynamics of diffusion models may change significantly in the early, middle, and late training stages. DST may consistently outperform SST by adaptively changing the update rate $p$ and update frequency $\\delta T$ according to the training phase (e.g., start with large changes and gradually stabilize).\n- Analyze the fundamental causes of the correlation between sparsity and generation quality. Experimental results suggest that the optimal sparsity and methods vary depending on the model and dataset. For example, while performance improved with 90% sparsity in QuickDraw, it deteriorated in CelebA-HQ. The underlying mechanisms behind this are not yet fully understood. Could this performance difference be attributed to the inherent redundancy of the dataset, or to which parts of the model architecture (e.g., the attention block in U-Net, skip-connection) are more strongly affected by sparsification? Could analyzing which layers of the model are robust to sparsification and which are vulnerable lead to the design of more effective sparsification strategies?\n- Please consider the possibility of sparse strategies specific to diffusion models. The sparsification method used in this study is a general-purpose method established in existing works. However, diffusion models have a unique input called noise level $t$. Would it be effective to dynamically switch sparse masks according to noise level $t$? For example, in the initial denoising steps where noise is high ($t$ is large), prioritizing dense connections to capture global structure, and in later steps where noise is low ($t$ is small), using different sparse patterns to capture details, could such time-dependent sparsification contribute to performance improvement?\n- Please add experiments on conditional generation. This study does not address conditionally generated tasks such as text-to-image generation, which is important in practical applications. What impact would sparsifying mechanisms such as cross-attention, which injects conditional information into the model, have on generation quality, particularly fidelity to conditions? Could a hybrid strategy that maintains the critical pathway for transmitting conditional information while actively sparsifying other parts be effective for optimizing the efficiency of conditionally generated models?\n- Please provide additional information on the characteristics of sparse-to-sparse training other than generation quality (FID). The paper focuses on the final generation quality (FID), but it does not mention the impact of sparsification on stability or convergence speed during training with a limited number of epochs. Do sparse models tend to require more time to converge or exhibit unstable learning compared to dense models? Under limited computational resources, is there a possibility of different trade-offs, such as the final performance being inferior to dense models but reaching adequate performance more quickly?\n- Please clarify the differences between other previous studies, such as on GANs, and the findings of this study. When applying the sparse-to-sparse strategy, what are the differences in trends between the diffusion model and GANs?", "broader_impact_concerns": "Nothing to report.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rD7uWX8rfM", "reviewer": "Reviewer_HyzT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The idea of introducing sparse-to-sparse training into the diffusion model context is interesting and worth exploring.\n\nS2: The paper has rather clear positioning and structures, as the goal is not to directly compare performance between models or datasets but to gain insights into the impact of sparsity in DM training.\n\nW1: While the paper does not directly emphasize the performance between models or datasets, the current version presents rather limited analysis and insights on the impact of sparsity in DM from the perspective of dynamics. As the stable training dynamics is also one of the main advantages of DMs compared to other generative models, the impact of sparse training on learning dynamics is worth further exploration. The current main results only compare the final generation performance measured in FID, it should not be too much extra work because those models have been trained from scratch anyway.\n\nW2: In Figure 5, the performance of the two dynamic sparse training methods, MagRan-DM and RigL-DM, shows noticeable fluctuations as the pruning ratio changes. This behavior was not evident in the main paper’s reported results. Could the authors elaborate on this discrepancy and provide further analysis?\n\nW3: Regarding the choice of unconditional generative diffusion models, I would assume that using the classic DDPMs, rather than LDMs, would be the more intuitive and straightforward baseline—especially for a study that appears to be in the early exploratory stage. Could the authors clarify the rationale behind choosing LDMs over vanilla DDPMs?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes to improve the efficiency of diffusion models in both training and inference through the sparse-to-sparse training technique. Specifically, the authors focus on the unconditional training from scratch setting and test two diffusion model variants, six datasets, and three sparse training methods. Experiments show that sparse DMs are able to match and often outperform their dense counterparts under the proposed settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses from the previous comment.", "broader_impact_concerns": "This work has properly discussed the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)