Title: VETA-DiT: Variance-Equalized and Temporally Adaptive Quantization for Efficient 4-bit Diffusion Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Transformer

Post-training Quantization

Image and Video Generation

Efficient Inference
Abstract: Diffusion Transformers (DiTs) have recently demonstrated remarkable performance in visual generation tasks, surpassing traditional U-Net-based diffusion models by significantly improving image and video generation quality and scalability. However, the large model size and iterative denoising process introduce substantial computational and memory overhead, limiting their deployment in real-world applications. Post-training quantization (PTQ) is a promising solution that compresses models and accelerates inference by converting weights and activations to low-bit representations. Despite its potential, PTQ faces significant challenges when applied to DiTs, often resulting in severe degradation of generative quality. To address these issues, we propose VETA-DiT (**V**ariance-**E**qualized and **T**emporal **A**daptation for **Di**ffusion **T**ransformers), a dedicated quantization framework for DiTs. Our method first analyzes the sources of quantization error from the perspective of inter-channel variance and introduces a Karhunen–Loève Transform enhanced alignment to equalize variance across channels, facilitating effective quantization under low bit-widths. Furthermore, to handle the temporal variation of activation distributions inherent in the iterative denoising steps of DiTs, we design an incoherence-aware adaptive method that identifies and properly calibrates timesteps with high quantization difficulty. We validate VETA-DiT on extensive image and video generation tasks, preserving acceptable visual quality under the more aggressive W4A4 configuration. Specifically, VETA-DiT reduces FID by 33.65 on the DiT-XL/2 model and by 45.76 on the PixArt-$\Sigma$ model compared to the baseline under W4A4, demonstrating its strong quantization capability and generative performance. Code is available at: https://github.com/xululi0223/VETA-DiT.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies two key challenges in quantizing DiT models: (1) large inter-channel variance caused by outliers, and (2) significant shifts in activation distributions across denoising timesteps. To address these issues, the authors propose a KLT-enhanced Hadamard transform for variance alignment and an incoherence-aware adaptive calibration method to manage temporal dynamics. Experimental results demonstrate strong performance compared to existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* **S1**: The paper is clearly written and easy to follow. The presentation is organized and the explanations are generally accessible, which helps the reader understand the main ideas without much difficulty.\n* **S2**: The proposed method is simple and straightforward, making it easy to implement. This practical aspect increases its usability and potential for adoption in real-world applications.\n* **S3**: The experimental results are strong and well-supported. The model is evaluated across multiple datasets, tasks, and metrics, showing its effectiveness and generality. Notably, the clearer performance gap observed when further quantizing from W4A8 to W4A4 provides additional evidence of the proposed method’s effectiveness.\n\n## Weaknesses:\n* **W1**: For a quantization method like this, I would expect an analysis of hardware resource usage. Unfortunately, the paper does not include such an evaluation.\n* **W2**: The paper could be further strengthened by evaluating the proposed method at even lower bit settings, such as W2A8 or W4A2.\n\nQUESTIONS:\nQ1: Can you provide more details on the practical implementation aspects of your method, especially regarding hardware compatibility or deployment on resource-constrained devices?\n\nQ2: Given the clear performance differences observed when quantizing from W4A8 to W4A4, do you anticipate similar trends at lower bit-widths? What challenges do you foresee when pushing the quantization limits further?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VETA-DiT, a post-training quantization framework for Diffusion Transformers (DiTs) under W4A4 setting. The method addresses two major challenges, large inter-channel variance due to outliers and timestep-dependent activation shifts during iterative denoising. To mitigate these, the authors introduce a KLT enhanced Hadamard transform for better variance alignment and an incoherence-aware temporal adaptation strategy. VETA-DiT outperforms existing PTQ baselines across image and video generation tasks, achieving high visual quality under low-bit quantization settings without retraining.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is well-structured, with clear visualizations and an accessible algorithm description, aiding reproducibility and understanding.\n2. This paper integrates the Karhunen–Loève Transform (KLT) with the Hadamard transform in the context of DiT for the first time, and demonstrates outstanding generative quality in low-precision settings.\n3. The incoherence-aware temporal adaptation and synthetic data generation strategy are straightforward and well-validated through diverse empirical experiments.\n4. The proposed scheme is applied to large-scale networks for both image and video generation, demonstrating its generality and robustness.\n\nWeaknesses\n1. The combination of KLT with the Hadamard transform has already been presented in MambaQuant (ICLR 2025), which predates this NeurIPS submission. Thus, the novelty of this paper is somewhat reduced, although its contribution to the DiT domain remains valid and meaningful.\n2. As a result, the main contribution of this work lies in the design of the incoherence-aware adaptation. While the idea is interesting, comparisons with alternative temporal calibration or sampling strategies are insufficient.\n3. Since the method combines KLT with the Hadamard transform, the computational overhead is likely higher than using Hadamard alone. The paper should provide practical validation of inference speedups on real hardware.\n4. While the paper analyzes the time-varying activation distributions for image generation, it gives limited attention to similar temporal characteristics in video synthesis tasks.\n\nQUESTIONS:\n1. The combination of KLT with the Hadamard transform appears similar to the method proposed in MambaQuant (ICLR 2025). Can the authors clearly articulate how their usage is different in either formulation, application, or motivation specific to DiTs?\n\n2. While the paper addresses temporal variance in image generation (across denoising steps), it is unclear whether similar temporal adaptation is used or needed in video synthesis. Do video generation tasks show similar incoherence trends, and is the adaptation equally effective?\n\n3. How does your incoherence-aware adaptation with synthetic data generation compare to previous methods built upon KLT-Hadamard transforms, in terms of effectiveness and novelty?\n\n4. What is the actual runtime improvement or slowdown introduced by your method, including the overhead of the KLT-H computation and calibration?\n\n5. You mention disabling mixed-precision in ViDiT-Q for fair comparison. Could you elaborate on the implications of this choice, and whether similar mixed-precision designs could benefit VETA-DiT as well?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VETA-DiT, a post-training quantization framework tailored for Diffusion Transformers (DiTs) to enable efficient low-bit inference (W4A4 and W4A8) while maintaining high generative quality. 1) To address inter-channel variance caused by outlier activations, the authors propose a Karhunen–Loève Transform (KLT) enhanced Hadamard transformation that equalizes channel variance and improves quantization robustness. 2) To handle the temporal variation in activation distributions during the iterative denoising process, they design an incoherence-aware adaptive calibration method that constructs a synthetic calibration set by prioritizing timesteps with high quantization difficulty. Experiments on image and video generation benchmarks demonstrate that VETA-DiT outperforms prior PTQ methods and remains effective under aggressive 4-bit settings without requiring model retraining.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Proposes a novel KLT-enhanced Hadamard transform that effectively equalizes inter-channel variance and improves quantization performance under low-bit settings.\n\n2. The writing is clear and easy to follow.\n\n3. Demonstrates strong empirical performance across multiple datasets (ImageNet, COCO, VBench) and models (DiT-XL/2, PixArt-Σ, STDiT3), especially under challenging W4A4 configurations.\n\nWeaknesses:\n\n1. Efficiency metrics are missing. The authors only report accuracy results, despite claiming improved runtime and efficiency as core motivations. Without latency and memory comparisons—e.g., against FP16 on TorchScript or TensorRT—the work lacks self-contained evidence of practical gains.\n\n2. Diffusion timesteps can be distilled to a small number. It is unclear whether the proposed method remains effective in such scenarios, such as with DMD [1,2]. How much improvement does it bring in terms of both accuracy and efficiency when applied on top of already distilled, smaller models?\n\n3. Although multiple models are evaluated, the core approach is mainly benchmarked on relatively small-scale models except OpenSora (should also clarify how many parameters in models). The scalability of the method to larger models remains unclear. E.g., how does it perform on models such as Wan2.1-T2V-1.3B or Wan2.1-VACE-14B?\n\n4. The method requires computing eigenvectors of covariance matrices, which may introduce non-negligible overhead during calibration for large models. It would be helpful to quantify this preprocessing cost.\n\n[1] Improved Distribution Matching Distillation for Fast Image Synthesis\n[2] One-step Diffusion with Distribution Matching Distillation\n\nQUESTIONS:\nSee weaknesses. The main concerns are 1) missing efficiency comparison, 2) effectiveness on distilled smaller models or larger-scale foundation models, and 3) overhead analysis.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces VETA-DIT, a 4-bit quantization framework for Diffusion Transformers (DiTs) that tackles activation variance imbalance and temporal dynamics. It uses a KLT-enhanced Hadamard transform to equalize variance and a temporally adaptive strategy to handle shifting statistics across diffusion timesteps. This dual approach allows VETA-DIT to maintain high-quality generation in aggressive W4A4 settings, making powerful DiT models practical for resource-constrained devices.

STRENGTHS AND WEAKNESSES:
Strengths:
1. It introduces an innovative KLT-enhanced Hadamard transform to effectively mitigate the challenging issue of channel-wise variance imbalance in activations.
2. The proposed temporally adaptive strategy intelligently handles the dynamic statistical shifts of activations across different diffusion timesteps.
3. VETA-DIT achieves state-of-the-art performance, preserving high generation quality even in the highly aggressive W4A4 quantization setting.

Weaknesses:
1. Its temporally adaptive strategy relies on the quality of the calibration dataset, which could limit its effectiveness when dealing with extreme or out-of-distribution data.

QUESTIONS:
1. How much computational overhead does the KLT-enhanced Hadamard transform introduce during inference?
2. How sensitive is the final model performance to the specific method used for calculating the timestep importance scores?
3. Since the KLT is data-dependent, how well does the learned transform generalize to datasets with different statistical properties?
4. What are the anticipated challenges in applying this framework to even lower bit-rates, such as 3-bit or 2-bit quantization?

I will consider raising the score if the authors can address my concerns.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-12