OBS-Diff: Accurate Pruning For Diffusion Models in One-Shot

## Abstract

Abstract Large-scale text-to-image diffusion models, while powerful, suffer from prohibitive computational cost. Existing one-shot network pruning methods can hardly be directly applied to them due to the iterative denoising nature of diffusion models. To bridge the gap, this paper presents OBS-Diff , a novel one-shot pruning framework that enables accurate and training-free compression of large-scale text-to-image diffusion models. Specifically, (i) OBS-Diff revitalizes the classic Optimal Brain Surgeon (OBS), adapting it to the complex architectures of modern diffusion models and supporting diverse pruning granularity, including unstructured, N:M semi-structured, and structured (MHA heads and FFN neurons) sparsity; (ii) To align the pruning criteria with the iterative dynamics of the diffusion process, by examining the problem from an error-accumulation perspective, we propose a novel timestep-aware Hessian construction that incorporates a logarithmic-decrease weighting scheme, assigning greater importance to earlier timesteps to mitigate potential error accumulation; (iii) Furthermore, a computationally efficient group-wise sequential pruning strategy is proposed to amortize the expensive calibration process. Extensive experiments show that OBS-Diff achieves state-of-the-art one-shot pruning for diffusion models, delivering inference acceleration with minimal degradation in visual quality.

## 1 Introduction

Recent advances in text-to-image generation have been largely driven by large-scale diffusion models (Rombach et al., 2022; Ramesh et al., 2022; Xie et al., 2024). These models, such as the Stable Diffusion 3 and 3.5 series (Esser et al., 2024), are capable of producing stunning images from textual prompts, revolutionizing fields from digital art to content creation. However, their massive parameter counts—often in the billions (e.g., 8B in Stable Diffusion 3.5-Large)—create prohibitive computational and memory demands, severely limiting their broader accessibility.

To improve the efficiency of the diffusion models, multiple research avenues have been proposed. A significant area of research is dedicated to expediting the sampling process by reducing the required denoising steps (Song et al., 2021; Lu et al., 2022) or knowledge distillation (Salimans & Ho, 2022; Sauer et al., 2024). Orthogonal to these efforts, model compression aims to reduce the intrinsic computational and memory footprint of the model itself. This category includes methods like quantization (He et al., 2023; Li et al., 2023b; Shang et al., 2023; Li et al., 2023a) and pruning, which is the primary focus of our work.

Pruning (Han et al., 2015; 2016; Wang et al., 2021; Fang et al., 2023a; Feng et al., 2024) is an effective way to reduce the computational and memory costs of deep neural networks. However, the rapid evolution of diffusion models underscores the severe limitations of existing pruning techniques. Current methods often lack generality (Fang et al., 2023b; Li et al., 2023c; Kim et al., 2024), as they are typically tailored to specific architectures like the U-Net and are not easily adapted to large-scale, text-to-image diffusion models with diverse structures (e.g., Multimodal Diffusion Transformer). Moreover, the efficiency gains from pruning are frequently undermined by computationally expensive requirements, such as the need for gradient information during pruning (Fang et al., 2023b; Zhang et al., 2024b) or a costly post-pruning fine-tuning stage. Furthermore, unstructured and semi-structured pruning remains largely unexplored for large-scale text-to-image diffusion models. All of these motivate our central research question:
Can we develop a general and training-free pruning framework capable of pruning diffusion models with diverse architectures and supporting multiple pruning granularities in a one-shot manner?

In the domain of Large Language Models (LLMs), one-shot and training-free pruning methods like SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2024) have achieved remarkable success. Subsequent SlimGPT (Ling et al., 2024) and SoBP (Wei et al., 2024) further explored training-free structured pruning. These layer-wise post-training pruning method approaches efficiently compress massive models without requiring costly retraining. However, the existing training-free pruning methods from the LLM field, such as SparseGPT, cannot be directly applied to the diffusion model. This is due to the unique challenges posed by diffusion models: their iterative nature, where parameters are shared across multiple denoising steps. Furthermore, the complex architectures introduce additional difficulties for pruning.

To bridge this gap, we introduce OBS-Diff, a novel one-shot, training-free pruning framework designed specifically for large-scale text-to-image diffusion models. Our approach revitalizes the classic Optimal Brain Surgeon (OBS) (Hassibi et al., 1992) and tailors it to the unique, iterative nature of the diffusion denoising process. By reformulating the pruning objective to account for the temporal dynamics of generation and introducing a computationally efficient calibration strategy, OBS-Diff efficiently removes redundant weights with minimal impact on performance, all without requiring any training during the pruning process or fine-tuning.

Our contributions are summarized as below:

- •
We adapt the OBS framework to handle the complex architectures of modern diffusion models, such as the Multimodal Diffusion Transformer (MMDiT), and demonstrate OBS-Diff versatility across unstructured, semi-structured (e.g., 2:4 sparsity patterns), and structured pruning (e.g., removing entire attention heads or FFN neurons).
- •
Recognizing that errors introduced in the early stages of the iterative denoising process have a compounding effect, we propose a Timestep-Aware Hessian Construction. This novel construction weights the importance of parameters according to their influence across the entire denoising trajectory, prioritizing the more sensitive early steps through a logarithmic weighting scheme.
- •
To overcome the prohibitive cost of sequential calibration in iterative models, we devise a group-wise sequential pruning strategy built upon “Module Packages”. This approach amortizes the expensive data collection process by processing layers in batches, striking an effective balance between computational time and memory requirements for the pruning process.
- •
Extensive experiments demonstrate that OBS-Diff sets a new state-of-the-art for training-free diffusion model pruning. It achieves inference acceleration while maintaining high visual quality, outperforming other layer-wise pruning methods across various sparsity levels and patterns.

## 2 Related Work

#### Pruning for Diffusion Models.

Several methods have explored pruning for diffusion models. An early approach, Diff-pruning (Fang et al., 2023b; a), introduced a gradient-based method for structured pruning. However, its demonstration on small-scale, non-text-to-image models (e.g., DDPMs) and its dependency on expensive retraining limit its applicability to modern, large-scale systems with diverse architectures. A significant line of research has since focused on compressing the UNet-based text-to-image diffusion models, with works like SnapFusion (Li et al., 2023c), MobileDiffusion (Zhao et al., 2024), BK-SDM (Kim et al., 2024), LAPTOP-Diff (Zhang et al., 2024a), and LD-Pruner (Castells et al., 2024) all targeting less salient components of the UNet architecture.

Other works have explored different architectures or techniques; for instance, Tinyfusion (Fang et al., 2025) introduced depth pruning for the DiT architecture. More recently, EcoDiff (Zhang et al., 2024b) introduced a general pruning framework for text-to-image models applicable to diverse architectures; however, it remains dependent on a costly training phase to learn a pruning mask and requires extensive hyperparameter tuning. A common theme among these methods is a dependency on training or fine-tuning and a primary focus on architecture-specific, structured pruning. Furthermore, unstructured and semi-structured pruning for large-scale, text-to-image diffusion models remains a largely unexplored area.

#### Layer-Wise Pruning Methods.

Early post-training compression methods, notably Optimal Brain Damage (OBD) (LeCun et al., 1989) and Optimal Brain Surgeon (OBS) (Hassibi et al., 1992), utilized Hessian-based saliency scores to prune individual weights. However, the prohibitive cost of computing and storing the full Hessian matrix limited their scalability. This challenge spurred the development of layer-wise approaches such as L-OBS (Dong et al., 2017) and Optimal Brain Compression (OBC) (Frantar & Alistarh, 2022), which approximate the Hessian locally to make pruning tractable.

As models scaled to billions of parameters, particularly in Large Language Models (LLMs), new methods emerged. SparseGPT (Frantar & Alistarh, 2023), Wanda (Sun et al., 2024), and DSnoT (Zhang et al., 2024c) focused on efficient unstructured and semi-structured (N:M pattern) pruning. Subsequently, SlimGPT (Ling et al., 2024) and SoBP (Wei et al., 2024) extended the OBS methodology to a structured granularity. The applicability of the OBS framework has also been demonstrated in other architectures, such as the Mamba model via SparseSSM (Tuo & Wang, 2025). Nevertheless, this family of compression methods remains unexplored in the field of diffusion models.

## 3 Preliminaries

### 3.1 Layer-Wise Post-Training Pruning

Post-training pruning often decomposes the global network compression problem into a series of independent, layer-wise subproblems (Hubara et al., 2021; Nagel et al., 2020; Aghasi et al., 2017). For each layer $l$, the objective is to find a pruned weight matrix $\mathbf{\hat{W}}_{l}$ that minimizes the output reconstruction error, given input activations $\mathbf{X}_{l}$ and a target sparsity $S_{l}$. This is formulated as:

$$ $\operatorname{argmin}_{\mathbf{\hat{W}}_{l}}\left\|\mathbf{W}_{l}\mathbf{X}_{l}-\mathbf{\hat{W}}_{l}\mathbf{X}_{l}\right\|_{2}^{2}\quad\text{s.t.}\quad\operatorname{sparsity}(\mathbf{\hat{W}}_{l})=S_{l},$ (1) $$

where $\|\cdot\|_{2}^{2}$ is the squared Euclidean norm. The network is pruned by sequentially solving this optimization problem for each layer.

### 3.2 Optimal Brain Surgeon for Layer-Wise Pruning

The Optimal Brain Surgeon (OBS) framework (Hassibi et al., 1992) offers an efficient solution to the layer-wise problem in Eq. ([1](https://arxiv.org/html/2510.06751v1#S3.E1)). A key insight of OBS is that the $\ell_{2}$-norm objective allows the problem to be decoupled into independent subproblems for each row of the weight matrix $\mathbf{W}_{l}$.

For each row, OBS approximates the objective with a second-order Taylor expansion centered around the current weights. This relies on the Hessian of the reconstruction error, $\mathbf{H}=2\mathbf{X}_{l}\mathbf{X}_{l}^{T}$. This approximation yields a closed-form solution to identify the least salient weight $w_{q}$—the one whose removal minimally increases the error—and to compute the optimal update $\delta\mathbf{w}$ for the remaining weights in its row. The saliency score $\mathcal{L}_{q}$ and the update are defined as:

$$ $\mathcal{L}_{q}=\frac{w_{q}^{2}}{2[\mathbf{H}^{-1}]_{qq}},\quad\delta\mathbf{w}=-\frac{w_{q}}{[\mathbf{H}^{-1}]_{qq}}\mathbf{H}_{:,q}^{-1},$ (2) $$

where $[\mathbf{H}^{-1}]_{qq}$ is the $q$-th diagonal element of the inverse Hessian and $\mathbf{H}_{:,q}^{-1}$ is its $q$-th column.

This process is repeated iteratively until the target sparsity $S_{l}$ is reached. After each weight is removed, the inverse Hessian must be updated. To circumvent the prohibitive cost of full re-inversion and the error accumulation of approximate rank-one updates, SparseGPT (Frantar & Alistarh, 2023) imposes a fixed pruning order. This structural constraint enables efficient and stable updates to the inverse Hessian information using methods like Cholesky decomposition (Frantar et al., 2024) as weights are progressively removed.

## 4 Methodology

Figure: Figure 2: Illustration of the proposed OBS-Diff framework applied to the MMDiT architecture. Target modules are first partitioned into a predefined number of “Module Packages” and processed sequentially. For each package, hooks capture layer activations during a forward pass with a calibration dataset. This data, combined with weights from a dedicated timestep weighting scheme, is used to construct Hessian matrices. These matrices guide the Optimal Brain Surgeon (OBS) algorithm to simultaneously prune all layers within the current package before proceeding to the next.
Refer to caption: x2.png

### 4.1 Timestep-Aware Hessian Construction

The layer-wise pruning objective defined in Eq. ([1](https://arxiv.org/html/2510.06751v1#S3.E1)) is effective for models with a single forward pass, but insufficient for diffusion models, which are iterative and operate over a denoising trajectory parameterized by discrete timesteps $t\in\{1,\cdots,T\}$.(^1^11Here, $t$ denotes the sequential index of the denoising iteration during inference, where $T$ is the total number of inference steps (e.g., for $T=28$, $t$ ranges from $1,2,\cdots,28$).) The impact of pruning-induced errors is not uniform across this trajectory. Errors introduced in early inference steps (small t) are inherently more damaging, as they propagate and compound through all subsequent steps ($t+1,\cdots,T$), leading to larger deviations in the final output.

Therefore, a robust pruning strategy must prioritize preserving network function during these critical early stages. We reformulate the layer-wise optimization problem to minimize a weighted reconstruction error that places greater importance on earlier, higher-impact steps:

$$ $\arg\min_{\hat{W}_{l}}\mathbb{E}_{t\sim[1,T]}\left[\alpha_{t}\left\|W_{l}X_{l,t}-\hat{W}_{l}X_{l,t}\right\|_{2}^{2}\right],$ (3) $$

Here, $X_{l,t}$ is the input to layer $l$ at step $t$, and $\alpha_{t}$ is a step-dependent weight. We define $\alpha_{t}$ using a simple and effective logarithmically decreasing schedule:

$$ $\alpha_{t}=\alpha_{\min}+\frac{\alpha_{\max}-\alpha_{\min}}{\ln(T)}\ln(T-t+1),\quad t\in\{1,2,\cdots,T\}.$ (4) $$

This schedule ensures the weight is highest at the beginning of inference and decays smoothly, such that $\alpha_{1}>\alpha_{2}>\cdots>\alpha_{T}>0$.

By incorporating this weighting, we adapt the Optimal Brain Surgeon framework (Hassibi et al., 1992). The Hessian, which captures the second-order information of this weighted loss, is now computed as a weighted sum over all inference steps:

$$ $H_{l}=2\sum_{t=1}^{T}\alpha_{t}\mathbb{E}[X_{l,t}X_{l,t}^{T}],$ (5) $$

which is termed as Timestep-Aware Hessian. It encapsulates the varying importance of parameters over the generation process. Saliency scores derived from its inverse are thus more sensitive to weights that are critical during the early, formative stages of the denoising process, resulting in a more faithfully pruned model.

### 4.2 Module Packages: A Group-wise Sequential Pruning Strategy

Conventional post-training pruning methods, such as SparseGPT (Frantar & Alistarh, 2023), employ a sequential layer-wise calibration. This paradigm is computationally prohibitive for diffusion models, as calibrating each layer necessitates executing a full, multi-step denoising trajectory. To address this bottleneck, we introduce Module Packages, a group-wise strategy that amortizes calibration costs by processing layers in batches.

Our approach is built upon two concepts. A Basic Unit is a set of layers with mutually independent inputs in a forward pass (e.g., query, key, and value projections), allowing for parallel processing. A Module Package comprises one or more Basic Units, which are pruned and calibrated collectively. Our framework processes these packages sequentially. For each package, we first execute a Group-wise Data Collection phase: we run the complete denoising trajectory once across the calibration dataset, using forward hooks to concurrently gather input statistics for all modules within the package. Subsequently, all modules are pruned simultaneously using their respective Timestep-Aware Hessian matrices.

Crucially, the network state is updated sequentially between packages but remains static within a package during data collection. This preserves the principle of sequential calibration at a coarser, group-wise granularity, rendering the process computationally feasible. This strategy drastically reduces the number of calibration runs, with the primary trade-off being an increased memory footprint to store multiple Hessian matrices concurrently. Notably, our empirical results demonstrate that pruning accuracy has low sensitivity to package granularity, granting practitioners the flexibility to balance computational cost against memory constraints without a significant performance sacrifice.

### 4.3 Extension to Semi-Structured and Structured Pruning

A key advantage of our OBS-Diff framework is its adaptability. While focusing on unstructured pruning, it readily extends to both semi-structured and structured sparsity.

#### Semi-Structured Pruning.

For semi-structured patterns like 2:4 sparsity, the extension is direct. Within each block of four weights, we simply prune the two with the lowest per-weight OBS-Diff saliency scores, efficiently creating hardware-friendly models.

#### Structured Pruning.

For structured pruning of Feed-Forward Network (FFN) layers, we assess a neuron’s importance by aggregating the saliency of its associated weights. The saliency $\mathcal{L}_{q}$ for an entire neuron (column $q$) and the corresponding weight update are:

$$ $\mathcal{L}_{q}=\frac{\sum W_{:,q}^{2}}{2[\mathbf{H}^{-1}]_{qq}},\quad\delta\mathbf{W}=-\frac{W_{:,q}}{[\mathbf{H}^{-1}]_{qq}}\mathbf{H}_{:,q}^{-1},$ (6) $$

where the lowest-scoring neurons are removed.

Similarly, for Multi-Head Attention (MHA), we prune entire heads. Our approach, inspired by SlimGPT (Ling et al., 2024), quantifies the saliency of each head.

The calculation begins with the full Hessian matrix, $\mathbf{H}$, for the output projection layer. For the $j$-th head, we consider its weight matrix $\mathbf{W}_{j}$ and the corresponding Hessian block $\mathbf{H_{j}}$. The total saliency for this head, $\mathcal{L}_{j}$, is found by aggregating the importance of its individual weights. The saliency is calculated as:

$$ $\mathcal{L}_{j}=\sum_{k=1}^{d}\frac{\sum(\mathbf{W}_{j})^{2}_{:,k}}{(\mathbf{H_{j}}^{-1})_{k,k}},$ (7) $$

where $(\mathbf{W}_{j})_{:,k}$ is the $k$-th column of the weight matrix $\mathbf{W}_{j}$, $(\mathbf{H_{j}}^{-1})_{k,k}$ is the $k$-th diagonal element of the inverse Hessian block, and $d$ is the dimension of each head.

However, MMDiT’s joint attention mechanism presents a unique challenge. Shared attention heads process concatenated multi-modal inputs, but are fed into separate, modality-specific output paths. This structure yields two distinct importance rankings for the same set of heads (one for each modality), while OBS-Diff processes the two output projection matrices after separation. To resolve this, we fuse these rankings into a single, decisive list using Reciprocal Rank Fusion (RRF):

$$ $S_{j}^{\text{RRF}}=\frac{1}{k+\text{rank}_{A}(j)}+\frac{1}{k+\text{rank}_{B}(j)},$ (8) $$

where $\text{rank}_{A}(j)$ is the rank of head $j$ for modality A, and $k$ is a stabilizing hyperparameter (e.g., 60). This fused score provides a unified ranking to guide the pruning of shared attention heads.

Subsequently, the weights of the entire output projection layer are updated using the full Hessian matrix, $\mathbf{H}$, following the formulation presented in Eq. ([6](https://arxiv.org/html/2510.06751v1#S4.E6)).

## 5 Experiments

### 5.1 Settings

#### Models.

To demonstrate the generalizability of OBS-Diff, we evaluate it across a diverse range of text-to-image models: Stable Diffusion v2.1-base (866M) (Rombach et al., 2022), Stable Diffusion 3-Medium (2B) (Esser et al., 2024), Stable Diffusion 3.5-Large (8B), and Flux.1-dev (12B) (Black Forest Labs, 2024). For comparison with prior work, we also evaluate our method on DDPM (35.7M) (Ho et al., 2020) trained on the CIFAR-10 (32 $\times$ 32) dataset (Krizhevsky et al., 2009).

#### Baselines.

For text-to-image models, we compare against methods adapted from the Large Language Model (LLM) domain for unstructured/semi-structured sparsity, namely Wanda (Sun et al., 2024) and DSnoT (Zhang et al., 2024c), as well as standard magnitude pruning. For structured pruning, we employ an L1-norm based baseline (Li et al., 2017). On the CIFAR-10 DDPM, our method is directly compared with Diff-Pruning (Fang et al., 2023b). The sparsity refers to the pruning ratio of all the linear layers within MHA and FFN for each MMDiT block. For calibration, we utilize text prompts from the GCC3M dataset (Sharma et al., 2018).
To ensure a fair comparison, all methods and baselines utilize identical configurations (computational resources provided in Appendix [B](https://arxiv.org/html/2510.06751v1#A2)).

The Wanda (Sun et al., 2024) and DSnoT (Zhang et al., 2024c) baselines are originally designed for unstructured and semi-structured pruning of LLMs. Their direct application to diffusion models is non-trivial due to the iterative nature of the diffusion model. Specifically, we extended their pruning logic by incorporating the concept of module packages, enabling them to perform unstructured and semi-structured pruning targeted at the key components of the diffusion architecture. Critically, to ensure an equitable comparison with our Hessian-based method, the adapted DSnoT baseline is configured to use its Hessian-based importance score calculation mode.

#### Evaluation Metrics.

We evaluate the performance of the text-to-image models on a subset of 5K prompts from the MS-COCO 2014 validation set (Lin et al., 2014). The evaluation is based on three metrics: Fréchet Inception Distance (FID) (Heusel et al., 2017), CLIP Score (ViT-B/16) (Hessel et al., 2021), and ImageReward (Xu et al., 2023). For the DDPM on CIFAR-10, we report the FID score. We measure efficiency gains in terms of wall-clock time reduction and the decrease in FLOPs.

### 5.2 Results Of Unstructured Pruning

**Table 1: Quantitative comparison of unstructured pruning methods on text-to-image diffusion models. The best result per metric is highlighted in bold.**
| Base Model | Sparsity (%) | Method | FID ↓ | CLIP ↑ | ImageReward ↑ |
| --- | --- | --- | --- | --- | --- |
| SD v2.1-base | Dense Model | 31.25 | 0.3142 | 0.3627 |  |
| 40 | Magnitude | 27.86 | 0.3111 | 0.1864 |  |
| DSnoT | 31.63 | 0.3099 | -0.0422 |  |  |
| Wanda | 27.96 | 0.3122 | 0.1367 |  |  |
| OBS-Diff | 28.19 | 0.3131 | 0.2061 |  |  |
| 50 | Magnitude | 49.38 | 0.2959 | -0.5580 |  |
| DSnoT | 69.05 | 0.2829 | -1.1395 |  |  |
| Wanda | 41.84 | 0.2988 | -0.4704 |  |  |
| OBS-Diff | 27.41 | 0.3102 | -0.0356 |  |  |
| SD 3-medium | Dense Model | 36.14 | 0.3162 | 0.9029 |  |
| 50 | Magnitude | 221.24 | 0.1864 | -2.2719 |  |
| DSnoT | 63.37 | 0.2908 | -0.5941 |  |  |
| Wanda | 43.98 | 0.3000 | -0.1076 |  |  |
| OBS-Diff | 27.20 | 0.3167 | 0.6468 |  |  |
| 60 | Magnitude | 349.53 | 0.1864 | -2.2807 |  |
| DSnoT | 211.58 | 0.2222 | -2.2271 |  |  |
| Wanda | 170.33 | 0.2352 | -2.0641 |  |  |
| OBS-Diff | 28.49 | 0.3099 | 0.1213 |  |  |

**Table 2: Performance of semi-structured (2:4 sparsity pattern) pruning on the Stable Diffusion 3.5-Large model. Pruning is applied to the 3rd through 25th MMDiT blocks. The best result is shown in bold.**
| Base Model | Method | FID ↓ | CLIP ↑ | ImageReward ↑ |
| --- | --- | --- | --- | --- |
| SD 3.5-Large | Dense Model | 31.59 | 0.3156 | 0.7549 |
| Magnitude | 45.39 | 0.2945 | -0.4705 |  |
| DSnoT | 32.40 | 0.3069 | 0.0307 |  |
| Wanda | 32.08 | 0.3036 | -0.1363 |  |
|  | OBS-Diff | 32.13 | 0.3129 | 0.4493 |

The results in Table [1](https://arxiv.org/html/2510.06751v1#S5.T1) show the superiority of our OBS-Diff in terms of CLIP score and ImageReward. An interesting phenomenon is observed with the FID metric – the pruned model can occasionally outperform the original dense model. E.g., at 40% sparsity on SD v2.1-base, the Magnitude method beats the dense model in FID, while our results suggest Magnitude does not produce visually better results. It is thereby conceived that FID may not be a very reliable metric here to evaluate different pruning methods.

Regarding the CLIP score, OBS-Diff is the best-performing method in the vast majority of test cases, exhibiting only a slight decrease compared to the dense models. Most notably, OBS-Diff consistently leads in the ImageReward metric across all benchmarks, indicating superior alignment with human aesthetic preferences.

The superiority of our approach becomes most pronounced at high sparsity levels. For example, at 60% sparsity on SD 3.5-Large or 70% on Flux 1.dev, the performance of all baseline methods collapses, resulting in metrics that are significantly worse than ours. This quantitative degradation corresponds to a qualitative failure; as illustrated in Figure [1](https://arxiv.org/html/2510.06751v1#S0.F1), the images generated by baseline methods at high sparsity are often totally destroyed and suffer from severe artifacts, whereas OBS-Diff continues to produce high-quality and coherent results. Beyond its performance in generation quality, OBS-Diff is also highly efficient. For instance, the entire pruning process for the 2B-parameter SD 3-medium model completes in under 15 minutes on a single NVIDIA RTX 4090, highlighting its excellent cost-effectiveness. Detailed analyses of pruning time and the impact of sparsity on ImageReward are provided in Appendix [C.1](https://arxiv.org/html/2510.06751v1#A3.SS1).

### 5.3 Results On Semi-Structured Pruning

The results for 2:4 semi-structured pruning are presented in Table [2](https://arxiv.org/html/2510.06751v1#S5.T2). Although Wanda obtains a slightly better FID of 32.08 compared to our 32.13, OBS-Diff shows substantial advantages in semantic-level metrics. Notably, it surpasses the strongest baseline by a large margin in both CLIP score (0.3129) and ImageReward (0.4493). This highlights our method’s effectiveness in maintaining high-level semantic consistency and visual fidelity under hardware-friendly sparsity constraints.

### 5.4 Results On Structured Pruning

**Table 3: Performance of structured pruning on the Stable Diffusion 3.5-Large model across various sparsity levels. The first and last transformer blocks were excluded from the pruning process. The TFLOPs metric represents the theoretical computational cost for a single forward pass of the entire transformer. For each sparsity group, the best result per metric is highlighted in bold.**
| Base Model | Sparsity (%) | Method | #Params | TFLOPs ↓ | FID ↓ | CLIP ↑ | ImageReward ↑ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SD 3.5-Large | Dense Model | 8.06 B | 11.26 | 31.59 | 0.3156 | 0.7549 |  |
| 15% | $L_{1}$-norm | 7.28 B | 9.63 ($\downarrow$14.5%) | 158.89 | 0.2376 | -2.0502 |  |
| OBS-Diff | 32.64 | 0.3157 | 0.6446 |  |  |  |  |
| 20% | $L_{1}$-norm | 7.02 B | 9.09 ($\downarrow$19.3%) | 189.50 | 0.2124 | -2.2385 |  |
| OBS-Diff | 32.46 | 0.3149 | 0.5475 |  |  |  |  |
| 25% | $L_{1}$-norm | 6.76 B | 8.55 ($\downarrow$24.1%) | 228.82 | 0.2040 | -2.2651 |  |
| OBS-Diff | 33.73 | 0.3128 | 0.3741 |  |  |  |  |
| 30% | $L_{1}$-norm | 6.54 B | 8.10 ($\downarrow$28.1%) | 327.48 | 0.2093 | -2.2663 |  |
| OBS-Diff | 34.51 | 0.3107 | 0.2221 |  |  |  |  |

The results are presented in Table [3](https://arxiv.org/html/2510.06751v1#S5.T3). The baseline $L_{1}$-norm pruning suffers from catastrophic performance degradation even at a modest 15% sparsity, with its FID score deteriorating from 31.59 to 158.89.
In stark contrast, our method, OBS-Diff, demonstrates remarkable resilience.
At the same 15% sparsity, OBS-Diff maintains an FID of 32.64, nearly identical to the dense model’s performance.
This robustness persists up to 30% sparsity, where OBS-Diff sustains a strong FID of 34.51 while the baseline model fails completely (FID of 327.48).
These findings highlight OBS-Diff’s superior ability to preserve critical model structures under aggressive structured pruning.

To benchmark our method against established techniques, we compare OBS-Diff with Diff-Pruning (Fang et al., 2023b), a well-recognized method that leverages gradient information for structured pruning on small class-conditional DDPMs. For brevity, the detailed results and analyses for the comparison are deferred to the  [C.3](https://arxiv.org/html/2510.06751v1#A3.SS3), where our method outperforms consistently.

### 5.5 Wall-Clock Time Comparison

**Table 4: Wall-clock inference time (ms) and speedup for a single MMDiT block under various sparsity schemes.**
| Sparsity Type | Time (ms) | Speedup |
| --- | --- | --- |
| Dense | 14.36 | / |
| Semi-structured (2:4) | 11.71 | 1.23$\times$ |
| Structured (15%) | 13.96 | 1.03$\times$ |
| Structured (20%) | 11.95 | 1.20$\times$ |
| Structured (25%) | 11.17 | 1.29$\times$ |
| Structured (30%) | 10.99 | 1.31$\times$ |

To quantify the practical efficiency gains, we measure the wall-clock time for a single forward pass through an MMDiT block of the SD3.5-Large model, on a single NVIDIA 4090 GPU with batch size 4, resolution 1024$\times$1024.

Table [4](https://arxiv.org/html/2510.06751v1#S5.T4) shows that both methods effectively reduce inference latency. The 2:4 semi-structured approach achieves a 1.23$\times$ speedup, while our structured pruning method attains 1.31$\times$ speedup at 30% sparsity. These results validate the tangible practical acceleration benefits of applying these pruning techniques.

### 5.6 Ablation Study

We perform an ablation study to analyze the impact of three key components: (1) the timestep-aware Hessian construction, (2) the number of module packages, and (3) the number of prompts in the calibration dataset. For this study, all variants are evaluated using the ImageReward metric on 1,000 prompts from the MS-COCO 2014 validation set.

Figure: Table 5: Ablation study of timestep weighting strategies, conducted on the SD3-Medium model at 50% unstructured sparsity. (For reference, the ImageReward of uniform strategy is 0.6355.)

#### Timestep-Aware Hessian Matrix Establishment.

To incorporate temporal information from the diffusion process, we introduce timestep-aware weighting during the Hessian matrix construction. This method assigns a distinct weight to the hooked activations at each timestep. Empirical results demonstrate that assigning greater importance to earlier inference steps yields superior performance. As shown in Table [5](https://arxiv.org/html/2510.06751v1#S5.T5), a logarithmic decrease strategy significantly outperforms other weight distribution methods.

Figure: Figure 3: Effect of the number of prompts in calibration dataset on the ImageReward.
Refer to caption: x3.png

#### Module-Package.

The concept of module packages partitions the model’s layers for layer-wise compression. This approach introduces a critical trade-off between computational resources and time. Processing the model in more packages reduces peak GPU memory, as the Hessian matrix for each pruning step is smaller. However, it proportionally increases the total runtime because the entire calibration dataset must be forwarded for each package. As shown in our ablation study (Table [6](https://arxiv.org/html/2510.06751v1#S5.T6)), while the resource trade-off is evident, the number of packages does not show a clear, predictable relationship with the final pruned model’s performance. Consequently, practitioners can select a configuration that best fits their hardware constraints without sacrificing final model quality.

#### The Number of the Prompts in the Calibration Dataset.

The size of the calibration dataset is a critical hyperparameter that directly influences the quality of the approximated Hessian matrix. To find an optimal size, we evaluated post-pruning performance against the number of text prompts in the calibration dataset, as shown in Figure [3](https://arxiv.org/html/2510.06751v1#S5.F3). The pruned model’s ImageReward score improves sharply up to 100 prompts and then plateaus, indicating a point of diminishing returns where additional data offers no significant benefit to the Hessian approximation. Therefore, to balance performance gains with computational efficiency, we selected 100 prompts for our calibration dataset in all main experiments.

## 6 Conclusion

This work introduces OBS-Diff, a novel one-shot, training-free pruning framework tailored for large-scale text-to-image diffusion models. By revitalizing the classic Optimal Brain Surgeon method, we address the unique challenges of iterative denoising through our proposed timestep-aware Hessian construction, which prioritizes critical early-stage generation steps. To overcome prohibitive calibration costs, we devise a group-wise sequential pruning strategy that effectively balances memory overhead and computational efficiency. The versatility of our framework extends across unstructured, semi-structured, and structured pruning, demonstrating its broad applicability. Extensive empirical results show that OBS-Diff establishes a new state-of-the-art in training-free diffusion model pruning, consistently outperforming existing methods by maintaining high generative quality, especially at high sparsity regimes.

## References

- Aghasi et al. (2017)
Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg.
Net-trim: Convex pruning of deep neural networks with performance guarantee.
In *NeurIPS*, 2017.
- Black Forest Labs (2024)
Black Forest Labs.
Flux.
[https://blackforestlabs.ai/](https://blackforestlabs.ai/), 2024.
Accessed: 2025-09-25.
- Castells et al. (2024)
Thibault Castells, Hyoung-Kyu Song, Bo-Kyeong Kim, and Shinkook Choi.
Ld-pruner: Efficient pruning of latent diffusion models using task-agnostic insights.
In *CVPR*, 2024.
- Dong et al. (2017)
Xin Dong, Shangyu Chen, and Sinno Pan.
Learning to prune deep neural networks via layer-wise optimal brain surgeon.
In *NeurIPS*, 2017.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *ICML*, 2024.
- Fang et al. (2023a)
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang.
Depgraph: Towards any structural pruning.
In *CVPR*, pp. 16091–16101, 2023a.
- Fang et al. (2023b)
Gongfan Fang, Xinyin Ma, and Xinchao Wang.
Structural pruning for diffusion models.
In *NeurIPS*, 2023b.
- Fang et al. (2025)
Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang.
Tinyfusion: Diffusion transformers learned shallow.
In *CVPR*, 2025.
- Feng et al. (2024)
Sicheng Feng, Keda Tao, and Huan Wang.
Is oracle pruning the true oracle?
*arXiv preprint arXiv:2412.00143*, 2024.
- Frantar & Alistarh (2022)
Elias Frantar and Dan Alistarh.
Optimal brain compression: A framework for accurate post-training quantization and pruning.
In *NeurIPS*, 2022.
- Frantar & Alistarh (2023)
Elias Frantar and Dan Alistarh.
Sparsegpt: Massive language models can be accurately pruned in one-shot.
In *ICML*, 2023.
- Frantar et al. (2024)
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.
Optq: Accurate quantization for generative pre-trained transformers.
In *ICLR*, 2024.
- Han et al. (2015)
Song Han, Jeff Pool, John Tran, and William Dally.
Learning both weights and connections for efficient neural network.
28, 2015.
- Han et al. (2016)
Song Han, Huizi Mao, and William J Dally.
Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.
In *ICLR*, 2016.
- Hassibi et al. (1992)
Babak Hassibi, David G Stork, and Gregory J Wolff.
Optimal brain surgeon and general network pruning.
In *NeurIPS*, 1992.
- He et al. (2023)
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang.
Ptqd: Accurate post-training quantization for diffusion models.
In *NeurIPS*, 2023.
- Hessel et al. (2021)
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
Clipscore: A reference-free evaluation metric for image captioning.
*arXiv preprint arXiv:2104.08718*, 2021.
- Heusel et al. (2017)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
In *NeurIPS*, 2017.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In *NeurIPS*, 2020.
- Hubara et al. (2021)
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry.
Accurate post training quantization with small calibration sets.
In *ICML*, 2021.
- Kim et al. (2024)
Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi.
Bk-sdm: A lightweight, fast, and cheap version of stable diffusion.
In *ECCV*, 2024.
- Krizhevsky et al. (2009)
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.
- LeCun et al. (1989)
Yann LeCun, John Denker, and Sara Solla.
Optimal brain damage.
In *NeurIPS*, 1989.
- Li et al. (2017)
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.
Pruning filters for efficient convnets.
In *ICLR*, 2017.
- Li et al. (2023a)
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, and Song Han.
Svdquant: Absorbing outliers by low-rank component for 4-bit diffusion models.
In *ICLR*, 2023a.
- Li et al. (2023b)
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer.
Q-diffusion: Quantizing diffusion models.
In *ICCV*, 2023b.
- Li et al. (2023c)
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren.
Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.
In *NeurIPS*, 2023c.
- Lin et al. (2014)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In *ECCV*, 2014.
- Ling et al. (2024)
Gui Ling, Ziyang Wang, and Qingwen Liu.
Slimgpt: Layer-wise structured pruning for large language models.
In *NeurIPS*, 2024.
- Lu et al. (2022)
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.
Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.
In *NeurIPS*, 2022.
- Nagel et al. (2020)
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.
Up or down? adaptive rounding for post-training quantization.
In *ICML*, 2020.
- Ramesh et al. (2022)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
*arXiv preprint arXiv:2204.06125*, 2022.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *CVPR*, 2022.
- Salimans & Ho (2022)
Tim Salimans and Jonathan Ho.
Progressive distillation for fast sampling of diffusion models.
In *ICLR*, 2022.
- Sauer et al. (2024)
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.
Adversarial diffusion distillation.
In *ECCV*, 2024.
- Shang et al. (2023)
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan.
Post-training quantization on diffusion models.
In *CVPR*, 2023.
- Sharma et al. (2018)
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.
In *ACL*, 2018.
- Song et al. (2021)
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
In *ICLR*, 2021.
- Sun et al. (2024)
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter.
A simple and effective pruning approach for large language models.
In *ICLR*, 2024.
- Tuo & Wang (2025)
Kaiwen Tuo and Huan Wang.
Sparsessm: Efficient selective structured state space models can be pruned in one-shot.
*arXiv preprint arXiv:2506.09613*, 2025.
- Wang et al. (2021)
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu.
Neural pruning via growing regularization.
In *ICLR*, 2021.
- Wei et al. (2024)
Jiateng Wei, Quan Lu, Ning Jiang, Siqi Li, Jingyang Xiang, Jun Chen, and Yong Liu.
Structured optimal brain pruning for large language models.
In *EMNLP*, 2024.
- Xie et al. (2024)
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al.
Sana: Efficient high-resolution image synthesis with linear diffusion transformers.
*arXiv preprint arXiv:2410.10629*, 2024.
- Xu et al. (2023)
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.
Imagereward: Learning and evaluating human preferences for text-to-image generation.
In *NeurIPS*, 2023.
- Zhang et al. (2024a)
Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, and Haonan Lu.
Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models.
*arXiv preprint arXiv:2404.11098*, 2024a.
- Zhang et al. (2024b)
Yang Zhang, Er Jin, Yanfei Dong, Ashkan Khakzar, Philip Torr, Johannes Stegmaier, and Kenji Kawaguchi.
Effortless efficiency: Low-cost pruning of diffusion models.
*arXiv preprint arXiv:2412.02852*, 2024b.
- Zhang et al. (2024c)
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji.
Dynamic sparse no training: Training-free fine-tuning for sparse llms.
In *ICLR*, 2024c.
- Zhao et al. (2024)
Yang Zhao, Yanwu Xu, Zhisheng Xiao, Haolin Jia, and Tingbo Hou.
Mobilediffusion: Instant text-to-image generation on mobile devices.
In *ECCV*, 2024.

## Appendix A Declaration Of LLM Usage

The use of Large Language Models (LLMs) in this work served two purposes: (1) to aid and polish the paper writing, and (2) to generate some of the text prompts used by the diffusion model to create figures that are shown in the paper.

## Appendix B Implementation Details

This section provides further details on the experimental setup, including common configurations, baseline adaptations, and the computational hardware used for our evaluations.

#### Common Configurations.

To ensure a controlled and fair comparison, all experiments, unless otherwise specified, adhere to a common set of configurations. For all text-to-image generation tasks, we set the output resolution to $512\times 512$ pixels to facilitate rapid experimentation across the diverse and large-scale models. For our method and baselines such as Wanda (Sun et al., 2024) and DSnoT (Zhang et al., 2024c), we consistently group model parameters into 4 module packages. Furthermore, a logarithmic decreasing timestep weighting scheme (log decrease) was uniformly applied across all diffusion models and pruning methods to schedule the pruning process over the diffusion timesteps.

#### Computational Resources.

The training of the DDPM on the CIFAR-10 dataset was conducted on NVIDIA A100 GPUs. For the large text-to-image models, all pruning methods are training-free. The pruning and evaluation for Stable Diffusion v2.1-base, Stable Diffusion 3-Medium, and Stable Diffusion 3.5-Large were performed on a single NVIDIA RTX 4090 GPU, each equipped with 48GB of VRAM. Due to its substantial memory footprint, all experiments involving the FLUX.1-dev model, including its pruning and evaluation, was conducted on a single NVIDIA A100 GPU with 80GB of VRAM.

## Appendix C More Experimental Results

### C.1 More Analysis for unstructuredly pruned SD3-medium

As illustrated in Figure [5](https://arxiv.org/html/2510.06751v1#A3.F5), our proposed OBS-Diff method consistently outperforms all baseline approaches in terms of the ImageReward metric across all evaluated sparsity levels. The superiority of our method is particularly pronounced at higher sparsity ratios. For instance, at 60% sparsity, the performance of competing methods collapses, yielding negative ImageReward scores. In stark contrast, OBS-Diff maintains a positive score, demonstrating its exceptional robustness in high-compression scenarios.

In terms of computational efficiency, Table [C.1](https://arxiv.org/html/2510.06751v1#A3.SS1) indicates that OBS-Diff has the longest pruning time on a single NVIDIA RTX 4090. However, the additional overhead is marginal, requiring only slightly more time than DSnoT (14.95 vs. 14.25 minutes). Considering the substantial gains in generation quality and model robustness, we conclude that OBS-Diff offers a superior trade-off between performance and computational cost, establishing it as a highly cost-effective pruning solution.

Figure: Figure 4: Pruning time of different unstructured pruning methods on SD3-Medium (2B) at 50% sparsity.
Refer to caption: x4.png

### C.2 Structured Pruning For SD3-medium

We evaluate our structured pruning method, OBS-diff, on the Stable Diffusion 3-medium model and compare it against the widely-used $L_{1}$-norm magnitude pruning baseline. As summarized in Table [7](https://arxiv.org/html/2510.06751v1#A3.T7), the baseline method suffers from severe performance degradation as sparsity increases. In contrast, our approach maintains performance remarkably close to the original dense model across all tested sparsity levels, demonstrating its effectiveness and robustness.

**Table 7: Performance comparison of structured pruning methods at 10%, 15%, 20%, and 25% sparsity on the Stable Diffusion 3-medium model (2B). The first and last transformer blocks were excluded from the pruning process. The TFLOPs metric represents the theoretical computational cost for a single forward pass of the entire transformer. For each sparsity group, the best result per metric is highlighted in bold.**
| Base Model | Sparsity (%) | Method | #Params | TFLOPs ↓ | FID ↓ | CLIP ↑ | ImageReward ↑ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SD 3-medium | Dense Model | 2.03 B | 2.84 | 31.59 | 0.3156 | 0.7549 |  |
| 10% | $L_{1}-norm$ | 1.91 B | 2.59 ($\downarrow$8.8%) | 267.32 | 0.2035 | -2.2611 |  |
| Ours (OBS-diff) | 35.65 | 0.3166 | 0.8118 |  |  |  |  |
| 15% | $L_{1}-norm$ | 1.83 B | 2.43 ($\downarrow$14.4%) | 326.92 | 0.1942 | -2.2768 |  |
| Ours (OBS-diff) | 34.33 | 0.3168 | 0.6717 |  |  |  |  |
| 20% | $L_{1}-norm$ | 1.78 B | 2.31 ($\downarrow$18.7%) | 348.77 | 0.1926 | -2.2768 |  |
| Ours (OBS-diff) | 33.15 | 0.3163 | 0.4997 |  |  |  |  |
| 25% | $L_{1}-norm$ | 1.72 B | 2.19 ($\downarrow$22.9%) | 365.24 | 0.1906 | -2.2786 |  |
| Ours (OBS-diff) | 32.96 | 0.3143 | 0.2782 |  |  |  |  |

### C.3 Comparison with Diff-Pruning on DDPM

To evaluate the generalizability of our method beyond large-scale text-to-image models, we adapt it to the task of structured pruning for a Denoising Diffusion Probabilistic Model (DDPM) on the CIFAR-10 dataset. Our adaptation leverages the column masks identified by Diff-Pruning, which are then integrated with our OBS weight update mechanism as detailed in Eq. ([6](https://arxiv.org/html/2510.06751v1#S4.E6)).

As presented in Table [8](https://arxiv.org/html/2510.06751v1#A3.T8), our method surpasses the current state-of-the-art baseline, Diff-Pruning, by achieving a superior FID score under an identical fine-tuning budget (100K steps). This result demonstrates not only the versatility of our approach but also suggests that the model pruned by OBS-Diff serves as a more effective checkpoint for subsequent fine-tuning.

**Table 8: Performance of pruned DDPMs on CIFAR-10 ($32\times 32$). All pruned models are fine-tuned for 100K steps. Evaluations are conducted on samples generated via 100 DDIM steps. The best FID score is highlighted in bold.**
| Method | #Params $\downarrow$ | MACs $\downarrow$ | FID $\downarrow$ | Train Steps $\downarrow$ |
| --- | --- | --- | --- | --- |
| Pretrained | 35.7M | 6.1G | 4.50 | 800K |
| Random Pruning | 13.95M | 2.1G | 7.85 | 100K |
| Magnitude Pruning | 13.95M | 2.1G | 7.91 | 100K |
| Diff-Pruning | 13.95M | 2.1G | 7.72 | 100K |
| OBS-Diff | 13.95M | 2.1G | 7.55 | 100K |

## Appendix D Additional Qualitative Results

Figure: Figure 6: More qualitative comparison of unstructured pruning methods on the SD3-Medium model. We evaluate Magnitude, DSnoT, Wanda, and our method (OBS-Diff) at various sparsity levels (20%, 30%, 40%, and 50%) using the same prompt and negative prompt. All images are generated at a resolution of $512\times 512$.
Refer to caption: x5.png

Figure: Figure 7: More qualitative comparison of unstructured pruning methods on the SD3-Medium model. We evaluate Magnitude, DSnoT, Wanda, and our method (OBS-Diff) at various sparsity levels (20%, 30%, 40%, and 50%) using the same prompt and negative prompt. All images are generated at a resolution of $512\times 512$.
Refer to caption: x6.png

Figure: Figure 8: More qualitative comparison of unstructured pruning methods on the SD3-Medium model. We evaluate Magnitude, DSnoT, Wanda, and our method (OBS-Diff) at various sparsity levels (20%, 30%, 40%, and 50%) using the same prompt and negative prompt. All images are generated at a resolution of $512\times 512$.
Refer to caption: x7.png

Figure: Figure 9: Qualitative comparison of unstructured pruning methods on Flux 1.dev at 70% sparsity. Results from Magnitude, DSnoT, Wanda, and our proposed OBS-Diff are shown.
Refer to caption: x8.png

Figure: Figure 10: Qualitative comparison of structured pruning methods on the SD3.5-Large model at various sparsity levels (15%, 20%, 25%, and 30%). Results from the L1-norm baseline and our proposed OBS-Diff are shown.
Refer to caption: x9.png

Figure: Figure 11: Qualitative comparison of structured pruning methods on the SD3.5-Large model at various sparsity levels (15%, 20%, 25%, and 30%). Results from the L1-norm baseline and our proposed OBS-Diff are shown.
Refer to caption: x10.png

Figure: Figure 12: Qualitative comparison of structured pruning methods on the SD3.5-Large model at various sparsity levels (15%, 20%, 25%, and 30%). Results from the L1-norm baseline and our proposed OBS-Diff are shown.
Refer to caption: x11.png

Figure: Figure 13: Qualitative comparison of structured pruning methods on the SD3.5-Large model at various sparsity levels (15%, 20%, 25%, and 30%). Results from the L1-norm baseline and our proposed OBS-Diff are shown.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Same as above.
soundness: 3
strengths: 1. The proposed timestep-aware Hessian is an elegant and intuitive solution. Weighting the pruning criteria based on the temporal dynamics of the diffusion process is a clever way to mitigate performance degradation.

2. The framework is explicitly designed to handle modern, complex architectures (like MMDiT) and supports a wide range of pruning granularities (unstructured, semi-structured, and structured). This makes it a potentially universal tool for diffusion model compression.
summary: This paper introduces OBS-Diff, a one-shot, training-free pruning framework designed to compress large-scale text-to-image diffusion models. The authors identify that existing methods fail because they don't account for the iterative nature of the denoising process, where errors can accumulate. OBS-Diff addresses this by adapting the classic Optimal Brain Surgeon (OBS) method. Its core innovations are a timestep-aware Hessian construction that gives more weight to earlier, more critical denoising steps, and a group-wise pruning strategy to make the process computationally efficient.
weaknesses: 1. the main result (Table 1) presents the unstructured pruning results. However, unstructured pruning have limited application and cannot provide memory reduction and speedup on conventional hardware. From the table, models can be pruned up to 50%. Similar results have been presented by baseline works (SparseGPT, Wanda), which shows the redundancy of transformer weights. Therefore, showing diffusion models can be pruned by 50% in an unstructured way has limited contribution.
2. Comparison results are mainly for unstructured pruning. Authors should show comparison results in structured pruning and semi-structured pruning settings to support the advantage of OBS-Diff in a practical scenario. From Table 2, Wanda is better than OBS-Diff under semi-structured setting. Comparison under structured setting is missing.
3. While many models are used under unstructured setting. Only SD3.5 is used under semi-structured and structured setting.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See above
soundness: 3
strengths: - Unified support for unstructured, N:M (e.g., 2:4), and structured sparsity in diffusion series models. 

- Efficient group-wise sequential pruning and effective weighted Hessian construction. 

- Qualitative and quantitative results across SD3-Medium/SD3.5-Large and Flux; Strong experimental metrics.
summary: OBS-Diff is a training-free, one-shot pruning framework for text-to-image diffusion models. OBS-Diff employs optimized OBS for diffusion architectures and supports multiple granularities (unstructured, N:M semi-structured, and structured pruning of MHA heads / FFN neurons). A timestep-aware Hessian with decreasing weights prioritizes early denoising steps to optimize error accumulation, and a group-wise sequential procedure optimizes calibration cost. Experiments on SD3/3.5/Flux models report minimal visual quality loss with notable speedups on a wide range of sparsity types.
weaknesses: - Figure 2 needs to be cited and introduced at the beginning of the methods section. 

- Wall-clock numbers are shown for a single MMDiT block; whole-model, sampler-inclusive speedups are not fully established. Efficiency gain under unstructured sparsity also needed to be reported.  

- The comparison set is narrow, mainly Magnitude, Wanda, DSnoT for un/semi-structured pruning and an L1-norm baseline for structured pruning (with Diff-Pruning only on CIFAR-10). Please broaden or justify the baseline choice by including (or at least discussing) recent training-based / non-one-shot diffusion compression methods 

- What would be the result of implementing SparseGPT like wanda? This experiment might demonstrate the advantages of the proposed approach over standard OBS. 

- The log-decrease schedule is intuitive but needs stronger ablation/theory vs. alternatives (e.g., cosine, exponential, other weights).  

- For CIFAR-10 DDPMs, the pruned models are fine-tuned (100k steps, C.3); clarify when fine-tuning is required vs. truly training-free regimes. Also, the enhancement of the proposed method seems to be very limited compared to other methods (even random pruning), please clarify. 

- In Table 8, the higher the sparsity, the better the FID metrics, explain why. 

- How sensitive is OBS-Diff to the timestep-weighting schedule and to the number/selection of timesteps used for Hessian estimation, and do the gains persist across samplers (DDIM/DPM++/EDM) and extend to video diffusion or cascaded/staged T2I? A qualitative explanation is sufficient; empirical results are optional if running new experiments would be time-consuming.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: My major concern lies in the experimental comparison. This paper only compares with LLM-based method.
soundness: 3
strengths: - OBS-Diff adapts OBS with a timestep-aware Hessian that gives earlier steps more weight, which matches error build-up in denoising. It also supports many sparsity types (unstructured, N:M, and structured like MHA heads and FFN neurons), making the pruning precise and flexible without extra training.
- It is a one-shot, training-free pipeline with a group-wise sequential strategy that cuts calibration cost. Experiments show clear speedups with little quality drop across common text-to-image backbones (e.g., SD/SDXL/DiT-style models), suggesting good practicality.
summary: The paper introduces OBS-Diff, a training-free, one-shot pruning framework tailored for large text-to-image diffusion models, whose iterative denoising makes standard LLM-based pruning ineffective. Experiments show OBS-Diff achieves state-of-the-art one-shot pruning for diffusion models, delivering notable inference speedups with minimal drop in visual quality.
weaknesses: - Most comparisons are against LLM-oriented, training-free heuristics (e.g., Wanda/DSnoT/magnitude) adapted to diffusion, plus simple L1 for structured cases. This risks overstating, because these baselines were not designed around DiT denoising dynamics. The paper should add results against diffusion-native pruning on Stable Diffusion/SDXL/DiT—e.g., LD-Pruner, EcoDiff, and TinyFusion.
- The proposed timestep weighting and the group-wise pruning are tuned on a small prompt set and tested in limited settings. It’s not clear they hold for other samplers, step counts, CFG scales, or out-of-domain prompts/styles. Wider tests across these factors would better show robustness.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why doesn’t the paper include a **comparison of inference-time acceleration** between different pruning methods? Only pruning-time cost is reported.
2. Is the **generalization ability limited by the calibration dataset**? Since pruning depends on calibration prompts, will prompts outside that distribution generate lower-quality results?
3. If the **calibration dataset size and prompt diversity** increase, could neuron activations become more uniform, reducing contrast in neuron importance and potentially leading to over-pruning of useful neurons?
soundness: 3
strengths: 1. The method is **highly generalizable**, applicable to different diffusion architectures such as U-Net and MMDiT.
2. It **extends OBS pruning** to diffusion models, introducing timestep-aware weighting that better preserves early-stage generation accuracy.
3. Supports multiple pruning granularities (unstructured, semi-structured, structured) in a unified framework.
summary: This paper proposes a **training-free OBS-Diff framework** for pruning multi-step diffusion models to reduce computation cost. It extends the **Optimal Brain Surgeon (OBS)** algorithm to diffusion models, using a **timestep-aware Hessian** to reflect the varying importance of parameters across denoising steps. The authors also design **Module Packages** to group layers for batch pruning, reducing the cost of activation collection. OBS-Diff supports **unstructured**, **semi-structured**, and **structured** pruning. Experiments show that it outperforms existing methods in ImageReward and CLIP score. And the quantitative experiment shows that it perform much better than other methods at 30% and 40% sparsity level.
weaknesses: 1. The pruning ratio is fixed per layer (layer-wise), without dynamic or weighted adjustment based on layer importance. Different layers may contribute unequally to the final output, so uniform sparsity could be suboptimal.


Decision: Accept (Poster)