DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space
Authors: Mang Ning
Mingxiao Li
Jianlin Su
Jia Haozhe
Lanmiao Liu
Martin Benes
Wenshuo Chen
Albert Ali Salah
Itir Onal Ertugrul
Venue: ICML 2025 poster (2025)
Keywords: generative models
diffusion models
image frequency
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion samplers demonstrate that DCTdiff outperforms pixel-based diffusion models regarding generative quality and training efficiency. Remarkably, DCTdiff can seamlessly scale up to 512$\times$512 resolution without using the latent diffusion paradigm and beats latent diffusion (using SD-VAE) with only 1/4 training cost. Finally, we illustrate several intriguing properties of DCT image modeling. For example, we provide a theoretical proof of why `image diffusion can be seen as spectral autoregression', bridging the gap between diffusion and autoregressive models. The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space. The code is at https://github.com/forever208/DCTdiff.

DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space
Mang Ning' Mingxiao Li
Jianlin Su *3 Jia H Lanmiao Liu Martin Benes Wenshuo Chen
Albert Ali Salah
Itir Onal Ertugrul
Abstract
This paper explores image modeling from the fre- quency space and introduces DCTdiff, an end- to-end diffusion generative paradigm that effi- ciently models images in the discrete cosine trans- form (DCT) space. We investigate the design space of DCTdiff and reveal the key design fac- tors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion sam- plers demonstrate that DCTdiff outperforms pixel- based diffusion models regarding generative qual- ity and training efficiency. Remarkably, DCTd- iff can seamlessly scale up to 512x512 resolu- tion without using the latent diffusion paradigm and beats latent diffusion (using SD-VAE) with only 1/4 training cost. Finally, we illustrate sev- cral intriguing properties of DCT image model- ing. For example, we provide theoretical proof of why 'image diffusion can be seen as spectral autoregression' bridging the gap between diffu- sion and autoregressive models. The effective-- ness of DCTdiff and the introduced properties suggest promising direction for image model- ing in the frequency space. The code is ht tpts //githul com/ forever208 DCTdifff.
Figure I. Image samples generated by DCTdiff trained on the dataset AFHQ 512x512 without using any VAE.
tion (DCT) and PNG applies DEFLATE (Deutsch, 1996) for compression in lossy and lossless manner, respec- tively. In this paper, explore image modeling in the DCT space with a focus on generative tasks, they require complete understanding of the entire image (Goodfellow, 2016). Recently, diffusion models (Song & Ermon, 2019; Ho et al., 2020) have demonstrated remarkable generative performance and been adapted in various tasks, including text-to-image generation (Ramesh et al., 2022; Esser et al., 2024), video generation (Blattmann al., 2023; Polyak et al., 2024), and 3D synthesis (Poole et al., 2022; Lin et al., 2023). However, ceffusion-basse generative modeling in the pixel space is expensive and difficult to scale directly to high-resolution generation. Researchers have explored alternatives such as latent space modeling (Rombach et al., 2022) and neural network-bases upsampling (Dhariwal Nichol, 2021) to address these challenges.
1. Introduction
Image discriminative and generative modeling the RGB space has been the mainstream approach in deep learning for long time due to the success of Convolutional Neural Networks (Krizhevsky et al., 2012; He et al., 2016) and Vision Transformers (Dosovitskiy et al., 2021). ). In contrast, images are often stored in a compressed form. For example, JPEG (Wallace, 1991) uses Discrete Cosine Transforma-
"Equal contribution Utreehht University, the Netherlands 'KU Leuven. Belgium 'Moonshot AI, China 'Shandong University, China Max Planck Institute for Psycholinguisties, the Netherlands University Innsbruck, Austria. Correspondence to: Mang Ning <m.ning@ @u.ning@uuul>...
We argue that image diffusion modeling in the pixel space is unnecessary due to its inherent redundancy. Instead, we advocate using a (near) lossless compression that provides a compact space for efficient diffusion modeling. JPEG achieves significant image compression by converting pixels
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
to the DCT frequency domain and eliminating the highest- frequency signals, as they have low energy and are less perceptible to the human eye. Motivated by JPEG, we pro- pose DCTdiff which models the image data distribution entirely in the DCT frequency space. We explore the design space of DCTdiff and uncover the key factors contributing to diffusion modeling in the frequency domain. Through extensive experiments, we demonstrate that DCTdiff sur- passes exxel-based diffusion in both generative quality and training efficiency. Crucially, DCT allows for significant lossless compression with negligible computation, enabling DCTdiff to seamlessly scale to 512x512 image generation without relying on an auxiliary Variational Autoencoder (VAE) (Rombach et al., 2022) which is typically trained with 9 million images and compound loss.
guidance (Ho & Salimans, 2022), ODE solver (Lu et al., 2022; Zhou et al., 2024) and SDE solver (Xue et al., 2024), exposure bias (Ning et al., 2023; Li et al., 2024), training dy- namies (Karras et al., 2024), model architecture (Peebles & Xie, 2023), noise schedule (Hoogehoom et al., 2023; Hang & Gu, 2024) and sampling schedule (Sabour et al., 2022), sampling variance (Bao et al., 2022), and distillation (Sal- imans & Ho, 2022: Song et al., 2023). Moreover, Poisson Flow (Xu et al., 2022), Flow Matching (Lipman et al., 2023) and Rectified Flow (Liu et al., 2023) are closely related to the ODF- diffusion models. Orthogonal to previous studies, we investigate image diffusion modeling from the DCT space for the first time.
2.2. Frequency Modeling in Neural Networks
We further reveal some unique properties of DCT-based image modeling. In Section 5.3, we present theoretical analysis that frames image diffusion modeling as spectral au- toregression. Particularly, the coarse-to-fine autoregressive generation of VAR (Tian et al., 2024) can be summarized as first generating low-frequency signals and then generat- ing high-frrquency image details. Also, we highlight that DCT image modeling has the flexibility and advantage of prioritizing different image frequencies according to the granularity of the task. Finally, we introduce new theorem for image upsampling within the DCT space, offering supe- rior performance over traditional methods such as bilinear or bicubic interpolation. In summary, our contributions are:
Frequency transformation is often performed as a module of the neural network to speed up the computation (Mathieu et al., 2014; Pratt et al., 2017; Zhang et al., 2018; Tamkin et al., 2020), increase the accuracy of image classification (Fridovich- Keil et al., 2022), or improve the image genera- tive performance (Phung al., 2023). For example, Huang et al. (2023) applied the Fourier transform to latent repre- sentations to adaptively select useful frequencies for target tasks. Yang et al. (2023) incorporate the anvelet-based gat- ing mechanism into the diffusion network to enable dynamic frequency feature extraction. Additionally, DCT was uti- lized by (Kuzina & Tomezak, 2023) as a low-dimensional latent representation and applied to the VampPrior frame- work to obtain the flexible prior distribution.
We propose DCTdiff to perform image diffusion mod- eling in the DCT space for the first time.
In contrast to treating frequency modeling as module or auxiliary component of the whole network, researchers have investigated image modeling within the frequency space by transforming image pixels into frequency signals and feeding them the neural network. For instance, DCTrans- former (Nash et al., 2021) proposes generative modeling in the DCT space in an autoregressive manner. Buchholz & Jug (2022) perform image super-resolution tasks in the Fourier domain using an autoregressive model, where low frequen- cies of an image are conditioned to predict the missing high frequencies. Likewise, Mattar et al. (2024) apply Wavelets Transform to images for autoregressive generation. In addi- tion, Wavelet-Based Image Tokenizer is proposed for image discriminative tasks (Zhu & Soricut, 2024) and generative tasks (Esteves et al., 2024). Recently, JPEG-LM (Han et al., 2024) directly models images and videos as compressed files saved on computers by outputting file bytes in JPEG and AVC formats. In this paper, we adopt DCT space for image generative modeling because DCT concentrates most of the signal's 's energy into a few low-frequency components (Rao & Yip. 2014), making it very effective for compres- sion. Also, DCT operates on real numbers, simplifying the practical implementation.
We elucidate the design space of DCTdiff and show that it outperforms the pixel-based and SD- -D-VAE-based latent diffusion models regarding generation quality and training speed.
We reveal several intriguing properties of image model- ing in the DCT space, suggesting its potential for both discriminative and generative tasks and its advantages over conventional pixel-based image modeling.
2. Related Work
2.1. Diffusion Models
Diffusion models were introduced by Sohl-Dickstein et al. (2015) and improved by Song & Ermon (2019) and Ho al. (2020). Furthermore, Song et al. (2021b) unify score- based models and denoising diffusion models via stochas- tie differential equations (SDE), and EDM (Karras et al., 2022) provides a disentangled design space for diffusion models. Recent advancements in diffusion models have been achieved across various dimensions, including classi- fier guidance (Dhariwal & Nichol, 2021) and classifier-free
2

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
3. Background
generative modeling than RGB space in a near-lossless way. The architecture and pipeline of DCTdiff are illustrated in Figure 2 and we now elaborate on each component.
Diffusion models progressively perturb a random data sam- ple zo. drawn from data distribution Pamm. into pure noise Ør as time& flows. The forward perturbation process is de- scribed by the Stochastic Differential Equation (SDE) (Song et al., 2021b)
4.1. Color Space Transformation and Chroma Subsampling
We follow the JPEG codec and first convert images from the RGB space to the YCbCr color space, containing a brightness component Y (luma) and two color components Cb and Cr (chroma). Formally, given an image E Axxx with height and width w, the color space transformation function can be written as x' Mx + b, where M 3x3x is a fixed transformation matrix and b TR3 is the offset vector, and the output x' E R represents the YCbCr image. Then we perform 2x chroma subsampling for both cb and Cr channels since the human eye is more sensitive to brightness than color details (Gonzalez, 2009). As a result, the Y channel of x' stays the same (denoted as x,y E RAx"", while the Cb and Cr channels become x'bb <Ri* ; x respectively (shown in Figure 2 (b)).
dx; fIz, t)dt + g(t)dwr,
(1)
where t E [0, T > 0 is a constant, -(:-)) and g(::-) are the drift and diffusion coefficients, and w, defines the standard Wiener process. A key property of the forward SDE is that there exists an associated reverse-time SDE
dz,- t)) g² (1) (x + g(t) (2)
where df represents an infinitesimal negative time step, in- dicating that this SDE must be solved from = 7 to = O. Moreover, Pe (a( denotes the probability distribution of Zt., and W. is now a standard Wiener process in the reverse time. This reverse-time SDE results in the same solution {zx}²-m ) as the forward SDE (Eq. (1)) (Anderson, 1982), given that ær is sampled from a prior noise distribution. After training a score model so ( t) x V2, log Pt ( parameterized by 0 via denoising score matching (Vincent, 2011: I; Song et al., 202 20n can plug so (z,, t) into Eq. (2) to get
and X'er E Note that, chroma subsampling in the YCbCr space brings 2x compression, reducing the signal amount from 3hw to 1.5/w. In Appendix B.1, we empirically show that this 2x compression produced by chroma subsampling significantly accelerates the diffusion training but at the cost of generation quality. In contrast, further transforming the subsampled YCbCr channels to the DCT space improves the quality of generative modeling (Appendix B.2).
dz:- = if (zt. t) (f) (zt, t) + g(t)dw,. (3)
Then, we can sample ær from the prior distribution I0, and solve Eq. (3) backwards in time to obtain the predicted solution trajectory (#t ) where t. is viewed as a gener- ated sample from the data distribution Pasta- Importantly, Song et al. (2021b) reveal that the reverse-time SDE shares the same marginal probability densities (p:( as the Probability Flow ODE:
4.2. 2D Block DCT
After the chroma subsampling, Y, cb and Cr channels are split into non-overlapping two-dimensional blocks with block size B, and the results are denoted as three sets: 1x
1 X =xxy Jimm X's = {xxb= fi=1 X = [xer 1-1' where
{xor'),=,
dx: = if(x t)- 2 log
(4)
N is the number of blocks in cb and Cr channels, and " two-dimensional DCT. We use the most common type-ll DCT (Ahmed et al., 1974) which converts zero-centered matrix A E IR x B into DCT block D E IR R B using a series of horizontal and vertical cosine bases:
Xy xes 'xer" i R Rx Each block is then transformed by
Again, by replacing the score function Vz, (og) with the learned score model se sz(z,,)), any numerical ODE solver, such as Fuler (Song al., 202 and Heun solvers (Karras et al., 2022), can be applied to solve this ODE to obtain an estimated data sample Zo.
D(u, v) = a())a())
4. Design Space of DCTdiff
B B-1
Σ T=U
A(r, v) cos
(2x + 1)uT
Cos
(2y + 1 1)eT
2B
2B
The DCTdiff proposed in this paper is inspired by the canon- ical JPEG image codees (Wallace, 1991). The main idea behind JPEG is that we can achieve data compression by discarding information that is less perceptible to the human eye, especially subtle color variations and high-frequency details, while retaining the essential visual quality of the im- age. In this paper, we utilize the DCT of JPEG codecs and show that DCT provides a more compact space for image
(5)
VI/B
if ==0
where alu) =
22BB if u +
The resulting D(u, u) is the DCT coefficient at position (u, v) in the frequency domain, A(z, y) is the YCbCr value at position (a, y) in the spatial domain, au) and (v) are
3

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
image x
b) Image channels: xy X Ner
c) blocks: Dy Dan Der
d) Respace blocks D Der
block DCT transform
VCBCF
Su Subsampling
4Y+1Cb+10
f) Entropy- Consistent Scaling
Rearder & remove coefficients
dim: 6(0 - m²)
Diffusion modeling
low high
Patch size
Diffusion Transformer
SNR scale
1
high freas
add noise
tokens
concat
D2 D D/r
Zigtag flatten
DCT image X
blacks size R
Operations single token
Figure 2. The architecture and pipeline of DCTdiff.
normalization factors. We represent the DCT blocks D, [Dy') AN h Des (Des" " Der = {Drr" i) 1==
blocks Dy Deb Dor E RBxB into one-dimensional vectors Dy Deb' D E TR using the zigzag pattern (shown in Figure 2 (c)), resulting in the coefficients or- dered from low-to-high frequency. In order decide the number of high-frequency coefficients to be chopped off for Dy Des Der', we propose the following criteria for generative tasks:
N
corresponding to the DCT outcomes transformed from the Y, Cb and Cr blocks, respectively. JPEG codec uses fixed block size B 8 due to the trade-off between efficiency and visual quality. However, resolutions require different block sizes for optimal diffu- sion modeling, thus the block size B is set as a variable in DCTdiff. The effect B is discussed in Appendix B.6.
find that different image
m arg max (m: rFID Pact data (m)) < < (6)
4.3. Frequency Tokenization
where we compute the reconstruction Fréchet Inception Distance (rFID) (Heusel al., 2017) between the data dis- tribution Pdata and the distribution Pact.aata which is derived from DCT compression by eliminating high-frequency coefficients. is constant and we empirically found that = 0.5 yields a good trade-off between generation qual- ity and compression rate. After removing m² coefficients, the dimension of Dy DDe Dar reduced from B to B² m Since we concatenate 14Y+ICb+1C; ken, a Transformer token contains 6(B² m²) frequency coefficients and the number of DCT tokens is N. So far, have transformed the RGB image to DCT coefficients, represented
We notice that the non-overlapping block operation in DCT resembles the patchfication Vision Transformer (Doso- vitskiy et al., 2021), therefore it is natural to use ViT-based models for DCT diffusion modeling. We propose using UViT (Bao et al., 2023) and DiT (Peebles & Xie, 2023) to construct DCTdiff due their remarkable performance.
To ensure that the Y, Cb, and Cr blocks within each Trans- former token correspond to the same spatial area, we respace the Cb and Cr blocks to expand to the same space as Y blocks (see Figure 2 (d)). Then, four blocks, one Cb block, and one Cr block are packed into single token. Note that, the patch size P in ViT-based models and the block size B in DCTdiff have the relationship P 2B which will be used throughout the paper. We also tested with other tokenization methods, for instance, each DCT block was considered as a token, but the performance was inferior to the "4Y+ICb+1C' combination.
to-
R x2xx66m=mm) Our goal to model Pasta given all DCT samples X using diffusion models.
4.4. Diffusion Modeling and Coefficients Scaling
Continuous-time (Song et al., 2021b) and discrete-time dif- fusion models (Ho et al., 2020) can all be applied for DCTd- iff to model Piata- For simplicity, we summarize the training process of continuous-tii diffusson models and one can refer to Ho et al. (2020) for the details the discrete-time case. Following Song et al. (2021b), we construct a diffu- sion process { indexed by a continuous time variable
notable property of DCT that many freeueryy coefficients are typically near-zero. These coefficients be eliminated as they contribute little to the visual qual- ity. To this end, we first turn the two-dimensional DCT

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
T] [] such that Xu Pasta- We use (x) to de- note the perturbation kernel from Xn to kg- Specifically; we follow UVIT to employ the Variance Preserving (VP) SDE, so that Poe (&g (8o) is a Gaussian:
L E
destroyed by the noise term during the forward diffusion process (detailed discussion in Section 5.3). Consequently, this results in the phenomenon that perturbing an image in the DCT space is faster than perturbing an image in pixel space, despite using the same forward SDE (see Figure 7). Furthermore, the larger the block size B, the more the en- ergy is concentrated in low frequencies, thus the faster the forward perturbation is. We can also infer this phenomenon from the bound 1) mentioned above. To this end, we in- troduce brief corollary concerning v/: given a dataset, ") doubles if the block size B is doubled.
N Na, Koe $ lg ((s)ds I- le- So ((s)ds)
in which ((.) is the noise scale. Then the score model se (R,. t) is trained by denoising score matching:
(0) = E, |I (R,, t)- - Lo. (x, [00]
(8)
where X(1) is a positive weighting function, is sampled from the uniform distribution U0, T), Ro Pama and R ~ Po (K. After training the diffusion model, we synthesize images by converting the generated DCT coeffi cients back to RGB pixels via inverse DCT.
Sketch of Proof. Since dj is derived from D(0,0) 0). we in- vestigate how D(0, 0) changes under block size 2B and B. Plugging 2B and B into Eq. (5) yields
2
2
A(x,0) 22
ΣΣ A(x y=
2B
A notable pre-processing in diffusion models is that Xo should be rescaled into the interval [-1, before perturba- tion. It is trivial for RGB pixels to be shifted and scaled from [0, 255] to [-1, 1]. However, we notice that the seal- ing method affects the diffusion training speed and sample quality when Xo are the DCT coefficients. Different from RGB, the bound of frequency coefficients varies greatly, depending on its position on the spectrum and the channels (Y/Cb/Cr). For instance, the upper bound of the lowest frequency signal in the Y channel (a.k.a DC component) is higher than that of the Cb channel by two orders of mag- nitude. Therefore, we initially considered Naive Scaling method: compute the bounds for each frequency and each channel (Y, Cb, Cr) individually, then apply each bound to scale the corresponding frequency coefficients into [-1, 1]. respectively. However, we observe that Naive Scaling broad- ens the distribution of high-frequency coefficients, leading to slow training and low generated sample quality (see Ap- pendix A. To maintain the shape of the original probabil- Scaling
==0
if we assume that A(x, y) has the same mean within the 2B x 2B block and B x B block. Thus D(0, 0) doubles when B is doubled, causing 7) to increase twofold. Recall that we scale down all frequency coefficients by before adding noise, a larger would result in more coefficients close to zero and destroyed by the noise in the early stage of forward diffusion. To counteract the effect of block size, we propose to scale the SNR -Signal-Noie--Ran noise schedule (inspired by Hoogeboom et al. (2023)). We leave the derivation and implementation of SNR Scaling to Appendix A.2. Experiments show that SNR Scaling improves the sample quality without affecting the training convergence (Appendix B.5).
of the default
5. Intriguing Properties of Image Modeling in the DCT Space
5.1. Frequency Prioritization
ity density, we thus propose the approach in which all frequency signals are scaled by the bound of the DC component (D(0, 0) of Y blocks) since it yields the largest bound. To avoid the influence of extreme values, we compute the bound 17 E IR within 5 percentile and 100- percentile:
Recall that training the score model is equivalent to predict- ing the isotropic Gaussian noise added on the clear image given the noisy image (Dhariwal & Nichol, 2021). Intu- itively, the task is reconstruct each frequency coefficient in & or reconstruct each pixel in the case of RGB image Xo. Since we cannot say which pixel is more important than another pixel, the training objective (Eq. (8)) treats every pixel equally. However, an intriguing property of DCT coef- ficients is that a low-frequency signal typically contributes more to the image quality than a high-frequency signal. Meanwhile, we observe that the lower the frequency of the signal, the larger the entropy of its distribution. Thereby, we can prioritize the modeling of low-frequency signals of Xn by adding Entropy-Based Frequency Reweighting (EBFR) into eq. (8), leading to LEBFR(O):
17 = max([IP),,1Poom) 11)
(9)
where P, denotes the rth percentile of the DC component distribution. We empirically find that 7 = 98.25 yields the best performance of DCTdiff among all datasets. Hereafter, we assume that &o has been scaled by 17, namely Xo = Xo/n.
4.5. SNR Scaling
An inherent property of DCT is that most of the signal's 's en- ergy is compacted into few www-ureuency components, so that high-freyuency components are near zero and quickly
EtA(t) x, [B sg (R,. t) - Vx, log (2. 1
(10)
5

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
where (B) E RS(B²-m") is the entropy vector of the valid frequency distributions. Given dataset, H (B) only depends on the block size B once m² is fixed. We empirically show that EBFR improves the sample quality without affecting the training speed (see Appendix B.4).
5.3. Image Diffusion Is Spectral Autoregression
Recently, Dieleman (2024) has empirically shown that pixel- based diffusion models perform approximate autoregression in the frequency domain. Intuitively, diffusion models de- stroy an image's ' high-frequency signals and then progres- sively destroy lower-frequency signals as time / flows in the forward diffusion process (Yang et al., 2023; Kingma & Gao, 2023; Rissanen et al., 2023). In this paper, we provide the theoretical proof for this phenomenon.
The frequency prioritization strategy can be easily extended to discriminative tasks using prior knowledge. Specifi- cally, we can allocate more network capacity to model high- frequency inputs on tasks requiring good understanding of fine details, for example, text and handwriting recognition, medical image analysis (Ronneberger et al.. 2015), finger- print recognition, forgery detection (Wu et al., 2019), etc. In contrast, we can explicitly highlight the low- wwwfrequency signals on tasks focusing on general shapes and overall struc- tures, for instance, scene recognition (Zhou et al., 2017), object detection in natural scenes (Redmon, 2016), action recognition (Simonyan & Zisserman, 2014), and so on.
Theorem 5.1. Consider diffusion model described by dx, = f( t)dt + g(t)dw, Let w denote the frequency, to(w) and z (i(w) represent the Fourier transform of the pixel image Du and It, respectively. The averaged power spectral density of the noisy image It satisfies:
E [[E[(W)]]]] = ==((w) = + +ʃ, (o(s) (11)
in which ( is the power spectral density of the image Do and natural images have the power-law: ( K Kw||- (Ruderman, 1997) and a are constants). Mean- while, Je (g(ssg Pds is independent of frequency w and ap- pears as a horizontal line in the spectral density graph.
5.2. Significant Lossless Compression under DCT
Unlike the Fourier transform, DCT operates on real num- bers using cosine functions, which effectively match the even symmetric extension of a signal (Rao & Yip, 2014). This alignment with signal characteristics allows the DCT to represent an image or other signals using fewer frequency coefficients (mostly the low-frequency ones). For generative tasks, we can again use rFID( dara, dat (m)) to measure the information loss when removing m high-frequency co- efficients. Table 1 presents the results of rFID using 50k im- ages from the data distribution Panta- If we consider 7 = 0.5 as a lossless compression for image generation, DCT could achieve 4x compression on 256 225 images and 7.1 7.11x compression on 512x512 images. In Section 6.1, we will show that the significant compression of DCT enables the diffusion model to scale smoothly up high-resolution generation, while pixel diffusion fails due to its high dimen- sionality. We also visually compare the image quality of VAE compression (Rombach et al., 2022) and DCT com- pression. As a training-free, computationally negligible, and domain insensitive compression method, DCT retains more image details than VAE compression (Figure 9).
Sketch of Proof. Taking the integral of the forward diffusion SDE yields x; Iu + + g(s)dw, (assuming fz, t) = 0 for VE-SDE). Since Fourier transform is linear, we have t.(w) = =)(w) + +,(w in the frequency domain, where é, (w) is the Fourier transform of the noise term L ((s)dw, By taking the expectation over the Wiener process Ws. we can obtain [w)) = [x(w)]² + [(x{(w)]]] due to E [[e:(w]]] O. According to Itô isometry (Itō, 1944), we have (w)) = So [=(s) which leads to Eq. (1I) (Please refer to Appendix A.3 for the detailed proof).
In Eq. (11), (2 quickly decreases to near-zero as frequency increases. So, the spectral density of the high- frequency component, IE [w])] is mainly decided by
the noise term 1 [ Since the noise term is mono- tonically increasing as grows from 0 - T, we can see that the noise term in the forward diffusion SDE first mainly de- stroys the high-frequency component of image 2o. and then gradually diminishes the lower- frequency signals. For every frequency w, we can further determine the required time to reach a specific SNR, see Appendix A.3 for the derivation. Interestingly, VAR proposed to generate images from coarse to fine by predicting the next-resolution image (Tian et al., 2024). We believe the success of VAR stems from the 'spec- tral autoregression' property of images. More recently, Yu et al. (2025) and Huang et al. (2025) have shown the efficacy of explicit frequency autoregression for image generative modeling.
Table rFID on the dataset FFHQ 256x256 and FFHQ 512x512. The com- pression ratio is relative to the RGB image having 3h signals.
(FID( Pas (m)())) when removing m coefficients
Dataset
Block size
ma
rFID
Compression ratio
0.19
3.56
FFHQ
0.49
4.00
256x256
9
0.96
4.57
44
0.23
6.40
FFHQ
8
46
0.48
7.11
512x512
48
1.18
8.00
Note that Theorem 5.1 holds if we replace the Fourier trans-
6

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
form with DCT since DCT is also a linear transformation and is simplified, real-valued variant of the Fourier trans- form. Inspired by (Dieleman, 2024), we visualize the aver- aged power spectral density from the DCT space (detailed in Appendix A.4). The resulting curves in Figure 6 resem- ble the case of the Fourier Transform, indicating that pixel diffusion is also spectral autoregression in the DCT space.
since they are near-zero in practice, The resulting D can be converted back to the pixel space to create high-resolution image.
2D
(k.1)
D(k.D
(k.
cos
(K.1)
D B
Similar to pixel diffusion, a frequency- diffusion pro- cess (e.g. DCTdiff) simultaneously adds isotropic noise to the whole spectrum in the forward diffusion process, i.e., equally perturbing high-frequency and low-frequency sig- nals at each time step. As high frequencies have low energy. they are first corrupted by the noise. Thereby, we conclude image diffusion is also spectral autore- gression. However, DCT concentrates the image energy into low frequencies (Rao & Yip, 2014), leaving most high- frequency components close to zero, so that DCT exhibits fast 'noise-adding" forward diffusion process (see Figure 7), which motivated the proposal of SNR Scaling method (discussed in Section 4.5).
fill up with 0
IR 2822x2
Figure 3. Illustration the DCT Upsampling algorithm.
We evaluate DCT Upsampling both qualitatively and quan- titatively. We show example in Figure 10 to illustrate the difference between Pixel Upsampling by bicubic in- terpolation (Gonzale7, 2009) and DCT Upsampling. The latter alleviates the blurry effect and exhibits an improved image quality. Also, we apply FID to evaluate the dis- tance between the ground truth data distribution and the upsampled data distribution. Experimentally. DCT Upsam- pling achieves FID 9.79, outperforming Pixel Upsampling (FID 12.53). One application of DCT Upsampling is super- resolution image generation. Specifically, one can follow the paradigm of ADM (Dhariwal & Nichol, 2021): first train a low-resolution image generation model, then replace the bilinear interpolation with our DCT Upsampling to obtain better draft high-resolution image, and another diffusion model can finally refine this draft image. Our experimental results and discussion are presented in Appendix A.10.
that frequency-
5.4. DCT Upsampling Outperforms Pixel Upsampling
In the aspect of image processing, we find that upsampling in the DCT space produces higher-quality images than up- sampling the pixel space (e.g., using bilinear or bicubic interpolation). Motivated by Dugad & Ahuja (2001), we introduce the following theorem to relate the frequency be- tween low- resolution and high-resolution images in the DCT space given any DCT block size B.
Theorem 5.2. Let A E 22Bx2B be matrix representing an image, and define À E BBxB as the matrix obtained by average pooling of A, where each element is computed as:
6. Experiments
To evaluate the performance of DCTdiff, we construct the models based on UViT and DiT without changing their Transformer architectures and compare DCTdiff with these two base models. The metrics used for the comprehensive comparison are FID (Heusel et al., 2017), training cost, inference speed, and scalability. Other evaluation metrics (c.g., recall and precision) are presented in Appendix A.7. To ensure fairness, we always use the same model size, patch size, and training parameters for DCTdiff and the base model (unless otherwise noted). The complete network parameters and training settings are listed in Appendix A.6, in which we also elaborate the choice of DCT parameters and show that determining these parameters is effortless for a new dataset. We leave all ablation studies to Appendix B.
A(i,j)
1 Σ
Σ
A(2i + m. m,
x=
Suppose D E B2Bx2B represents the DCT of A under block size 2B and D E BBx represents the DCT of Á under block size B. Then, for k, (0,1....,BB B 1}. the elements of D can be approximated by:
1
km
D(k, 1)
COS
cos
D(k
(12)
2
4B
4B
where (k, indexes the elements of the matrices D and D. Appendix A.5 provides the full proof.
6.1. Results on UVIT
Based on Theorem 5.2, we propose the DCT Upsampling algorithm. For each DCT block D converted from a low- resolution image, the algorithm computes D(k,)) () from D(k, I) according to Eq. (12), generating the low-frequency coefficients (purple block in Figure 3) of D. For the remain- ing frequency coefficients in D, we fill them up with zeros
UVIT (Bao et al., 2023) utilizes the continuous-time diffu- sion framework and different solvers for sampling. We train both UVIT and DCTdiff from scratch using the default train- ing parameters suggested by UVIT. The datasets include CIFAR- 10 (Krizhevsky et al., 2009), CelebA 64 (Liu et al.,
7

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Table 2. FID-50k of UVIT and DCTdiff using DDIM sampler and DPM-Solver under different NFEs, We implement class-conditional
generation on ImageNet 64, and unconditional generation on the rest of the datascts.
Euler ODE solver (DDIM sampler)
DPM-Solver
NFE
Model
CTFAR-10
CelehA 64
ImageNet 64
FFHQ 128
CIFAR-10
CelehA 64
ImageNet 64
FFHQ 128
UVIT
6.23
1.99
10.65
13.87
5.80
1.57
10.07
9.18
100
DCTdiff
5.02
1.91
8.69
8.22
5.28
1.71
9.73
6.25
UViT
7.88
3.50
15.05
26.26
5.82
1.58
10.09
9.20
50
DCTdiff
5.21
2.24
8.70
9.99
5.30
1.72
9.78
6.28
UViT
21.48
31.09
$2.10
87.68
6.19
1.73
10.25
9.21
20
DCTdiff
6.81
3.84
21.88
24.88
5.54
1.84
9.85
7.29
UVIT 81.67
224.21
166.63
209.69
26.65
4.37
13.27
14.26
10
DCTdiff 12.45
67.78
129.93
161.05
9.10
5.29
12.38
12.87
2015), ImageNet 64 (Chrabaszez et al., 2017), FFHQ 128, FFHQ 256, FFHQ 512 (Karras et al., 2019) and AFHQ 512 (Choi et al., 2020). We perform class-condittonal genera- tion on ImageNet 64 and unconditional generation for the other datasets. We test the sample quality using FID-50K under different Number of Function Evaluation (NFE) and two ODE solvers: DDIM sampler (Song et al., 2021a) and DPM-Solver (Lu et al., 2022).
Table 3. FID-50k UViT (latent) and DCTdiff on high-resolution image datasets for unconditional generation. DPM-Solver used for sampling.
Dataset
NFE
Model
FFHQ 256
FFHQ 512
AFHQ 512
UVIT (latent)
4.26
10.89
10.86
100
DCTdiff
5.08
7.07
8.76
UVIT (latent)
4.29
10.94
10.86
Results on Table show that DCTdiff consistently and some- times dramatically outperforms UVIT regardless NFEs and solvers (except for the outlier of CelebA 64 using DPM- Solver), demonstrating the effectiveness of image diffusion modeling in the DCT space. We believe the outlier can be attributed to the UViT training parameters being highly suited to CelebA 64, since the results of CelebA 64 based on DiT still demonstrate the superiority of DCTdiff.
50
DCTdiff
5.18
7.09
8.87
UVIT (latent)
4.74
11.31
11.94
20
DCTdiff
6.35
8.04
10.05
UVIT (latent)
13.29
23.61
28.31
10
DCTdiff 12.05
19.67
21.05
Importantly, we find that the parameters (B, T; m .) of DCTdiff only depend on the image resolution, and are in- variant the base models and dataset types (Appendix A.6), making the application DCTdiff convenient practice.
Note that UViT uses SD-VAE to perform latent diffusion (Rombach et al., 2022) when the image resolution reaches 256x256 because pixel diffusion modeling in the high- dimensional space is difficult (we attempted to train the UVIT model directly in the pixel space FFHQ 256, re- sulting in FID=120). However, we show that the diffusion paradigm of DCTdiff can be easily scaled to 512x512 im- age generation without VAE. denote the UVIT using SD-VAE as UVIT (latent) and compare it with our DCTdiff on three datasets. Table 3 indicates that DCTdiff achieves competitive FID to UVIT (latent) on FFHQ 256 and outper- forms UVIT (latent) when the resolution rises to 512.
Table 4. FID-50k DiT and DCTdiff using DDPM sampler under different NFEs for unconditional generation.
Dataset
NFE
Model
CelebA 64
FFHQ 128
DIT
5.11
12.81
100
DCTdiff
3.84
11.16
DiT
8.17
18.44
50
DCTdiff
6.23
15.23
6.2. Results DiT
15.64
33.56
20
DCTdiff
12.96
25.59
DIT (Peebles & Xie, 2023) applies the discrete-ting time diffu- sion framework and the Fuler SDE solver (DDPM sampler) for image sampling. To verify the generalization of DCTdiff. we utilize DiT as the baseline, and train DiT (in the pixel space) and DCTdiff from scratch with the same training set- tings on CelebA 64 and FFHQ 128 datasets. The resulting FIDs in Table 4 show that DCTdiff surpasses DiT under different sampling steps regarding the generation quality.
DiT
24.76
49.64
10
DCTdiff
20.87
43.14
6.3. Training Cost and Inference Speed
In addition to sample quality, we also compare the training costs between pixel-based UVIT and DCTdiff. Results in Table 5 demonstrate that the training of DCTdiff is faster
8

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
than that of UViT with the training acceleration up 2.5x. When comparing UVTT (latent) with DCTdiff, we follow Peebles & Xie (2023) and leverage GFLOPs (Giga Floating Point Operations) to indicate the computational complexity required for a single network-forward-pass We find that the GFLOPs of SD-VAE are huge and increase dramatically when the resolution rises from x56x256 to 512x512. As result, the total training cost of DCTdiff on FFHQ 256 is comparable that of UVTT (latent), but the training cost of DCTdiff on AFHQ 512 is only oue-quarter of that of UVIT (latent). We attribute the efficient training of DCTdiff to the compact space provided by DCT.
6.4. Scalability of DCTdiff
A key advantage of diffusion Transformers is their network scalability (Peebles & Xie, 2023). We empirically demon- strate that DCTdiff inherits this property and achieves im- proved sample quality as network capacity increases. Tables and 8 report the scalability results on the datasets CIFAR- 10 and FFHQ 128, respectively, where DCTdiff consistently outperforms UViT terms of FID. A similar scaling effect of DCTdiff is observed on the dataset ImageNet 64 as well (Appendix A.8).
Table 7. Scalability UVTT and DCTdiff on CIFAR-I0 (patch
4ize=4). FID-50k is reported using DDIM sampler.
Table 5. Training cost of UViT and DCTdiff. We use the same batch size for the two models each dataset. The training cost is indicated by GFLOPs and Training steps (convergence).
NFE
Model
Dpeth
# Para
100
50
20
Dataset
Model
Parameters
GFLOPs
Training stops
UVIT (small)
7.36
8.45
21.18
12
44M
ColebA 64
UVIT DCTdiff
44M
11
400k
DCTdiff (small)
6.51
6.62
7.87
44M
11
250k
UVIT (mid)
6.23
7.88
21.48
FEEQ 128
44M
11
750k
DCTdiff (mid)
16
131M
5.02
5.21
6.81
DCTRi
44M
11
300k
FFHQ 256
DCTdiff
UVIT (latent)
131M
131M
84M
133
200k
300k
DCTdiff (mid, deop)
UVIT (mid, deep)
20
161M
6.05
4.25
4.54
7.33
5.96
20.27
UWIT (latent)
131M
84M
575
225k
512
DCTdif
131M
133
225k
We further evaluate the inference speed by measuring the wall-clock time for both DCTdiff and UVIT. Since dCT and the pixel-based UViT share the same network param- eters and GFLOPs, they exhibit the same inference times. However, differences arise when comparing DCTdiff and latent UVIT at resolutions of 256x256 and 512x512: the SD- VAE component in latent UVTT incurs high computa- tional cost, whereas DCTdiff consumes more Transformer tokens. As shown in Table 6, DCTdiff is faster than latent UVIT in low-NFE settings but is slower at high-NFE condi- tions (without considering the sampling quality). Notably, DCTdiff demonstrates clear advantages over UVIT in terms of inference time when achieving comparable generation quality (refer to Appendix A.9).
Table 8. Scalability of UViT and DCTdiff on FFHQ 128. FID-50k is reported using DoMver.
NFE
Model
Dpeth
# Para
100
50
UVIT (small)
9.18
9.20
9.21
12
44M
DCTdiff (small)
6.25
6.28
7.29
UViT (mid) DCTdiff (mid)
6.96
6.99
7.40
16
131M
5.13
5.20
6.19
UViT (mid, deep) DCTdiff (mid, deep)
6.05
6.06
6.27
18
146M
4.98
5.05
5.94
7. Conclusion
In this paper, we explore image generative modeling in the DCT space and propose DCTdiff which shows superior per- formance over pixel-based and latent diffusion models. In particular, we reveal several interesting properties of im- age modeling from the DCT space, suggesting a promising research direction for image discriminative and generative tasks. However, the frequency-orientee Transformer archi- tecture and image generation in 1024x 1024 resolution are not explored in this paper, which encourages future study, Additionally, considering the inherent high temporal redun- dancy, video compression and modeling in the frequency space also hold potential for future exploration.
Table 6. Wall-clock inference time on AFHQ 512x We gen- erate 10k samples using one A000 GPU. Inference GFLOPs is appended in the brackets.
NFE
Model
100
50
20
10
20.2 min (4640)
13.4 min (2940)
9.3 min (1920)
7.9 min (1580)
UVIT (latent)
47.8 min
23.9 min (6650)
9.6 min (2660)
4.8 min (1330)
DCTdiff
(13300)

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Impact Statement
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
This paper advances the field of image generative model- ing through the DCT frequency space. Similar to other diffusion-based and autoregressive models, our work has applications in text-to-image generation and image editing over various domains (art, education, social media, and so on). There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Dugad, R. and Ahuja, N. A fast scheme for image size change in the compressed domain. IEEE Transactions Circuits and Systems for Video Technology, 11(4):
461-474, 2001.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J.,
Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
References
Ahmed, N., Natarajan, T., and Rao, K. R. Discrete cosine transform. IEEE transactions on Computers, 100(1 100(1))90- 93, 1974.
Esteves, C., Suhail, M., and Makadia, A. Spectral image tokenizer. arXiv preprint arXiv:2412.09607, 2024.
Fridovich- Keil, S., Gontijo Lopes, R., and Roelofs, R. Spec- tral bias in practice: The role of function frequency in generalization. NeurlPS, 35:7368-747 7382, 2022.
Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313- 326, 1982.
Gonzalez, R. C. Digital image processing. Pearson educa- tion india, 2009.
Bao, Li, C., Zhu, J., and Zhang, B. Analytic-dpm: an ana- lytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
Goodfellow, Deep learning, 2016.
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, J. All are worth words: vit backbone for diffusion models. In CVPR, pp. 12669-22679 2023.
Han, X., Ghazvininejad, M., Koh, P. W., and Tsvetkov, Jpeg-lm: Llms as image generators with canonical codec representations. arXiv preprint arXiv:2408. 08459, 2024.
Hang, T. and Gu, S. Improved noise schedule for diffusion training. arXiv preprint arXiv:2407.03297 2024.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15517,, 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Buchholz, T.-O. and Jug, F. Fourier image transformer. In CVPR, pp. 1846-18544 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 30, 2017.
Choi, Y. Uh, Y., Yoo, J., and Ha, J.W. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, pp. 8188-8197, 2020.
Ho, and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
Chrabasze7, P., Loshchilov, I., and Hutter, F. A downsam- pled variant of ImageNet as an alternative to the CIFAR datasets.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. NeurIPS, 33:6840-685) 6851, 2020.
2017.
Deutsch, P. Deflate compressed data format specification version 1.3. Technical report, 1996.
Hoogeboom, E., Heek, J., and Salimans, T. simple diffusion: End-to-end diffusion for high resolution images. In ICML, pp. 13213-13232. PMLR, 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-8794, 2021.
Huang. Z., Zhang, Z., Lan, C., Zha, Z.-.., Lu, Y., and Guo, B. Adaptive frequency filters as efficient global token mixers. In ICCV, pp. 6049-6059, 2023.
Dieleman, S. Diffusion is spectral autoregression, 2024. URL htttps://sander.a///000 spectral aanttraallattoregeessioo...t html.
Huang, Z., Qiu, X., Ma, Y., Zhou, Y., Zhang, C., and Li, X. Nfi: Autoregressive image generation with next- frequency prediction. arXiv preprint arXiv:2503.0707 2025.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
10

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Itó, K. 109. stochastic integral. Proceedings of the Imperial Academy, 20(8):519-524, 1944.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 35:5775-5787, 2022.
Jayasumana, S., Ramalingam, S.. Veit, A., Glasner, D., Chakrabarti, A., and Kumar, S. Rethinking fid: Towards a better evaluation metric for image generation. In CVPR, pp. 9307-9315, 2024.
Mathieu, M., Henaff, M., and LeCun, Y. Fast training of convolutional networks through ffts. ICLR, 2014.
Karras, T., Laine, S., and Aila, T. A style-based genera- tor architecture for generative adversarial networks. In CVPR, pp. 4401-4410, 2019.
Mattar, W., Levy, L., Sharon, N., and Dekel, S. Wavelets are all you need for autoregressive image generation. arXiv preprint arXiv:2406. 19997, 2024.
Karras, T., Aittala, M., Laine, S., and Aila, Elucidating the design space diffusion-basee based generative models. In NeurlPS, pp. (6565-26577) 2022.
Nash, C., Menick, J., Dieleman, S., and Battaglia, P. Gen- erating images with sparse representations. In ICML, pp. 7958-7968. PMLR, 2021.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In CVPR, pp. 24174- 24184, 2024.
Ning, M., Sangineto, E., Porrello, A., Calderara, S., and Cucchiara, R. Input perturbation reduces exposure bias in diffusion models. In ICML. pp. 2023.
PMLR,
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, pp. +195-4205, 2023.
Kingma, and Gao, Understanding diffusion objectives as the elbo with simple data augmentation. NeurIPS, 36: 2023.
Phung, H., Dao, Q., and Tran, A. Wavelet diffusion models are fast and scalable image generators. In CVPR, pp. 10199-10208, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., et al. Movie gen: cast of media foundation models. arXiv preprint arXi::2410.13720 2024.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. NeurlPS, 25, 2012.
Kuzina, A. and Tomczak, J. M. Hierarchical vac with diffusion-based vampprior. TMLR, 2023.
Poole, B., Jain, A., Barron, T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. NeurIPS, 32, 2019.
Pratt, H., Williams, B., Coenen, F., and Zheng, Y. Fenn: Fourier convolutional neural networks. In Machine Learn- ing and Knowledge Discovery in Databases: European Conference, ECMI. PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part I 17, pp. 786- 798. Springer, 2017.
Li, M., Qu, T., Yao, R., Sun, W., and Moens, M.-F. Alleviat- ing exposure bias in diffusion models through sampling with shifted time steps. ICLR, 2024.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., and Lin, T.-Y. Magic3d: High-resolution text-to-3d content creation. In CVPR, pp. 300-309, 2023.
Rao, K. R. and Yip, P. Discrete cosine transform: algo- rithms, advantages, applications. Academic press, 2014.
Lipman, Y., Chen, R. T., Ben- Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. ICLR, 2023.
Redmon, J. You only look once: Unified, real-time object
detection. In CVPR, 2016.
Liu, X., Gong, C., and Liu, Q Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023.
Rissanen, S., Heinonen, M., and Solin, A. Generative mod- elling with inverse heat dissipation. In ICLR, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. ICCV, pp. 3730-3738, 2015.
Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10694 2022.

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted Intervention-MICCN. 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Wallace, G. The jpeg still picture compression standard. Communications of the ACM, 34(4):30 44, 1991.
Wu, Y., AbdAlmageed, W., and Natarajan, P. Mantra-net: Manipulation tracing network for detection and local- ization of image forgeries with anomalous features. In CVPR, pp. 9543-9552, 2019.
Ruderman, D. Origins of scaling in natural images. Vision research, 37(23) -3988, 1997.
Xu, Y., Liu, Z., Tegmark, M., and Jaakkola, T. Poisson flow generative models. In NeurIPS, pp. 16782-16795, 2022.
Sabour, A.. Fidler, S., and Kreis, K. Align your steps: Optimizing sampling schedules in diffusion models. In ICML, 2022.
Xue, S., Yi, M., Luo, W., Zhang, S., Sun, J., Li, Z., and Ma. Z.-M. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. NeurIPS, 36, 2024.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Yang, X., Zhou, D., Feng, J., and Wang. X. Diffusion probabilistic model made slim. In CVPR, pp. 22552- 22562, 2023.
Salimans, T., Goodfellow, I. Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. NeurlPS, 29, 2016.
Yu, H., Luo, H., Yuan, H., Rong, Y., and Zhao, F. Frequency autoregressive image generation with continuous tokens. arXiv preprint arXiv:2503.0553
Simonyan, K. and Zisserman, A. Two-stream convolutional networks for action recognition in videos. NeurIPS, 27, 2014.
2025.
Zhang. J., Lin, Y., Song, Z., and Dhillon, I. Learning long term dependencies via fourier recurrent units. In ICML, pp. 3815-5823. PMLR, 2018.
Sohl-Dickstein, J.. Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequilib- rium thermodynamics. In ICML, pp. 2256-2265. PMLR, 2015.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. TPAMI, 4066):1452-1466 2017.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. ICLR, 2021a.
Zhou, Z., Chen, D., Wang, C., and Chen, C. Fast ode-based sampling for diffusion models around steps. In CVPR, pp. 7777-7786, 2024.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. NeurlPS, 32, 2019.
Zhu, Z. and Soricut, R. Wavelet-based image tokenizer for vision transformers. arXiv preprint arXiv:2405. 18616, 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Con- sistency models. In ICML, pp. 32211-32252. PMLR, 2023.
Tamkin, A., Jurafsky, D., and Goodman, N. Language through prism: A spectral approach for multiscale lan- guage representations. NeurIPS, 33:5492-5504, 2020.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang. L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 2024.
Turiel, A. and Parga, N. The multifractal structure of con- trast changes in natural images: From sharp edges to textures. Neural Computation, 12(4):763-793, 2000. doi:
10. 000116228899966600001553333
Vincent, P. A connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661- 1674, 2011.
12

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
A. Appendix
A.I. Naive Scaling and Entropy-Consistens Scaling
We show the bounds of Naive Scaling and Entropy- Scaling in Algorithm and Algorithm Given a dataset, we randomly draw 50,000 images and convert each image into DCT blocks {Dy J j==1" (De') 1/ i= {Der' "er' i=1' The total Y blocks Dy E IR 1000000NN n3? Cb blocks Dos E 5550000 and Cr blocks Der E R-50000 are used for Monte Carlo estimation of the bounds (f)) of Naive Scaling. We use only Y blocks Dw IR 20000000 XB" estimate the bound (r) of Coospy-Consistent Scaling given block size B and percentile T.
Algorithm Bound of Naive Scaling
1: Given B, T. Dy. Deb. Der
Algorithm 2 Bound of Entropy-Consisen Scaling
2: Initialize 7 list()
3: for := Dy. Das Der do
I: Given B, Dy
4: for ; :- 0, 0,....
do
2: Initialize 1)
5: up = np.perrcentile(al:.i r).
3: x +- Dyl:.
6: low np.percentile(ff:. 100 7)
4: up = mp-perccentile(z.1 T)
7: |low| > [up| then
5: low = np. np..ercentile((,, 100 - T)
8:
j.append([owr)
6: if [low| > Jupl then
9: else
7: " low
10:
(upl)
8: else
11: end if
9: < |upl
12: end for
10: end if
13:
end for
11: return !7
14:
return r7
Consistent Scaling using B = 2 and r = 97 on CelebA 64 dataset. The first row of Figure 4 displays the distributions of DCT coefficients (D(0, 0), D 1), D(1, 0), D(1, 1)) before scaling. It is clear that Naive Scaling increases the entropy of the original Scaling preserves the entropy.
In Figure 4, we illustrate the difference between applying Naive Scaling and
distributions of D(O, 1); D(1,0) 0) and D(1, 1) while
Hideggmm
Hetagan s1
Original
-1.0
Hittogrm
iecopan DXG,
1.2
Naive Scaling
400
Masggu
Hoaggan
Entropy-Consissent Scaling
-33
021
190
12.08
2.22
240
0:1
X70
108
12.0 1030
+339
330
633
030
354
12.00
4.30
3.00
523
075 100
Figure Histograms of DCT coefficients before scaling and after Naive Scaling and Entropy-Consistee Scaling.
A.2. SNR Scaling in Continuous-time and Discrete-time Diffusion Models
1. SNR SCALING IN CONTINUOUS-TIME DIFFUSION MODELS
Following UViT (Bao et al., 2023), we use VP-SDE for the continuous-timm diffusion model which has the forward perturbation kernel:
13

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
(R 20 ((a)da I- le- Je (()))))))
The noise schedule (()) al + and a, are constants (usually 0.1 and b 19.9), so that Jo B(s)ds at + 0.5b6
The SNR at time is denoted as:
SNR(() e- 56 A(s)ds
C (at++0.566")
(13)
E S6 B(s)ds
1 e (ot+00.56))
The goal of SNR Scaling to have the new RN (t) such that
(N) SNR())
(14)
where E R is the introduced factor of SNR Scaling. Given we need to derive the new noise schedule (3( c) for practical
implementation. Let Ju ((s)ds at 0.5bt and y' So' f (s)dss, Eq. (14) becomes
e
(15)
--e-w 1 c
from which, we derive
e - 1 (c 1)e
e
(16)
Taking the logarithm both sides of Eq. (eq: SNR yields
y y lnc In(1 + (c- 1
(17)
at 0.5bt - Inc In[1 + (c 1)e (attt.t0)))))))
(18)
Take the derivative of Eq. (18) w.r.t t, we obtain the new noise schedule 3'
B(((c))= - a (c 1)e (at (at-0.56t") (-a bt)
(19)
1 (c - 1)e (ab+)))
In the code implementation, apply Eq. (19) as the noise schedule to replace the original ((t) a + bt. When the scaling factor = 1, Eq. (19) degrades to (())
SNR Scaling for DPM-Solver. addition to applying (t), we need to update the inverse function of X(t) defined in DPM-Solver (Lu al., 2022) if we want to use it for sampling. The original A(t) is
X(t)
0.5 log (oN(S((()))
(20)
Now, given the updated S') we need to solve t from X(t) = 0.5 log (N).
A = 0.5 log (SN
(21)
e ve
21 logí
(22)
21
(23)
-
21
v
(24)
1 62
1 + 22
y/ = ln
(25)
22
Plugging Eq. (18) into Eq. (25), we get
14

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
= In| 1 2A
al + 0.5b Ince + In(1 + (c 1)e (at++.55()))
(26)
e2A c(1 c(l+e + e²a1
eat+0.5619 [1 + (c 1)e (ac+0..00)))
(27)
22
+ (c--1 = c(1 + e²a)
E a
(28)
22a
at + 0.5bl² |n| c(1 x2a)
+ d)
(29)
621
t= a
a² +
c(l 2bln
+1
-c
(30)
b
In practice we apply Eq. (30) update the inverse function of X(t). When the scaling factor c= 1, Eq. (30) degrades the original inverse function t -aty +2b 14,22
22
A.2.2. SNR SCALING IN DISCRETE-TIME DIFFUSION MODELS
Following DDPM (Ho al., 2020) and DiT (Peebles & Xie, 2023), the forward perturbation kernel is
(R &o (1 aie
(31)
where āe I a, and a 3r. So that the original SNR at time t is
a₁ SN = 1 - āt
(32)
R(t(() From which, we obtain a. SN ++ Now, scale the SNR by c, we have the updated signal schedule dy'
C X SNR(t)
āy
(33)
CX SN +1
Given the updated signal schedule az', we could iteratively solve of and 8/, and use 3/ for the implementation of SNR Scaling.
A.3. Image Diffusion Is Spectral Autoregression
In this section, we discuss how the data information changes during the diffusion process of diffusion model with the initial data being Zo and the data evolving into z,, For simplicity, we consider the scalar case of the diffusion process, since the original image diffusion is isotropic. Concretely, the forward diffusion SDE is
dz, = f(I,, t) + g(t)dwe,
(34)
where t E [0, T > 0 is constant, f(..)) and g(::)) :) are the drift and diffusion coefficients respectively, and we defines the standard Wiener process. We transform the image signal xt into (w) using the Fourier transform. Here, w represents the frequency. Thus, during the forward diffusion process, the signal x, can be represented in integral form as
It = xp + f g g(s)dw,
(35)
f(..)) = 0. After applying the Fourier transform, Eq. (35) becomes dz(w)) = 2o(w) + (e(w) where (ww is the Fourier transform of the Gaussian noise term. Obviously, the mean value E [er(w)]] = O. We now prove that [w)(] =
So ( dss. Consider the Fourier transform of (z)):
(w) = 1- e -iiie (x)dx
(36)
Since (x) is a random process with zero mean, its Fourier transform (w) is also random variable with zero mean.
15

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Calculate the variance of (,(w):
BiG((W) E [ rus (x)dxx
(37)
Expand the square the modulus:
lox e (x) (o)dr dr [ (eweerly) dy
E [[](w))]]] =
(38)
iila iula-viE (e(( dx dy.
Since (z) is spatially uncorrelated, we have
S/ (=(s))
when x =
(la(z)a(v))
(39)
0,
when x + y
This can be expressed as:
((c(())((V)) (f, (sss) 8(x y).
(40)
where 8() is the Dirac delta function. Substitute Eq. (40) into Eq. (38):
[[(é{(w))3] (%." 1o(s)
xo(e-v) v) o(x y)drdy
1/' (s(s)
-iwle-s)
(41)
(s(s)
dr.
The integral - dr means the integration region is infinite. In practice, we usually consider finite spatial range or normalize the density. For simplicity, consider unit-length spatial range that the integration result is 1, leading to
[[(2(w)) f." lo(s)
(42)
Then, we calculate the power density of various signals during the diffusion process. The power spectral density of signal z; is Sy, (w) E [w)( After expansion, obtain
Sz, (w) ( + 20 ([[))) + [[(W)]]]
(2 + 29R (wo(w) 0) [(([())]]]
(43)
= [2((w) + [
Since (w)) 0 and the cross term [29Re (fo(w) (w))) 0. have
Sz, (w) = 2 (w)) + E [[(
(44)
where E [[(w))]]] - (ds, which completes the proof of Theorem 5.1.
Moreover, we can evaluate how much information is damaged during the forward diffusion process from the SNR perspective. higher SNR implies that the signal is relatively purer and the degree of damage it undergoes lower. On the contrary, a lower SNR indicates that the noise has greater impact on the signal and the degree of information damage is also higher.
16

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Consequently, the SNR ratio can intuitively reflect the noise level contaminating the information during the forward diffusion SDE. The SNR can be expressed as
[Ro(w) SNR(W) = (w) Ld (s)s)
|Ro(w))
(45)
We find that the change of SNR with frequency is completely determined by the power spectral density |xo(w)] of the initial signal, while the noise power is the same all frequencies.
For natural images, it is generally the case that they possess low-pass characteristics. Moreover, their power spectra typically conform to power-law distribution (Turiel & Parga, 2000)), which can be expressed as (=)( x w where a >> 0 denotes the spectral attenuation degree of the signal. Therefore, as the frequency w increases, (²)( decreases rapidly.
This indicates that SNR(W) is low at high frequencies. As the diffusion time / increases, the denominator fo (s(s) increases, leading to an overall decrease in the SNR.
Given an SNR threshold y and a frequency w. the time ty (w) when the SNR reaches the threshold y satisfies:
|w
SNR(W) (2,(w) (ss
Y
(46)
From this, we can solve (w) to obtain the exact time required to reach SNR(w)= = 7. The right-side image of Figure 5 provides the illustrations of Eq. (46).
SNR vs. Frequerxy Different Times
Tine Reach SNR Threshuuld = 0.01
0-0
1-02
30
30'
10
30
100
10
10
7871
10-1
109
10
102
30:2
10
13a
30
Prequency
Frequency
Figure 5. Relationship between frequency w and SNR
A.4. Averaged Power Spectral Density in DCT space
Dieleman (2024) uses the radially averaged power spectral density (RAPSD) to analyze the frequency of images in Fourier space. Similarly, given the diffusion perturbation kernel N V( Zhe 1 Jr a(e)ds ,I - le- Sg B(s)de), we calculate the averaged power spectral density (APSD) for clear image øo. the noisy image &; and the isotropic Gaussian noise e;
(E: = -i= -- J 9(s)d=)e, wheree ~ I)) We use the Monte-Carlo method to estimate the APSD of zo. and E:. respectively with 50,000 samples from the FFHQ 256x256 dataset. Figure 6 shows the APSD curves at different times. Similar to the RAPSD figures under the Fourier transform in Dieleman (2024), APSD also shows a pattern of frequency autoregression.
A.5. Proof of DCT Upsampling Theorem
Consider a high-resolution image (c.g. 256 x 256) that consists of pixel blocks and each block is denoted as A E 22 then low-resolution (e.g. 128 128) image is derived from the average pooling of the 256 x 256 image and each pixel
17

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
I-1
10
1-50
10-4
10
15-
10
16.2
15
32"
19"1
12a
(a) 0=1
(b) 1'=10
L'=50
rolle
20--
21-4
30-
100
hequery
(d) 10=100
(e) 1'=200
(1) 1'=500
Figure 6. The averaged power spectral density (APSD) of Sa. æ; and the noise e; in the DCT space at time = 1' 1//0000.
block is denoted as À E R BxB, In this case, these two images have the same number of DCT blocks. Let D 2 be the DCT block converted from E 2B2B2 and D E TR. BxB be the DCT block converted from Á E Bx According to Eq. (5), we have
D(u,u) =
2
2
2B
Σ
12 128-1
Σ
A(z, y) cos
(2x + 1)uu
cos
(2y + 1)UT
(47)
2B
2B
4B
4B
2=0
0=
2
B-1 B B-1 Σ
(2i + 1)ux
(2j + 1)UT
D(u,v) = V B
A(i,j) cos
cos
(48)
B
2B
2B
F=
0=0
where D(u, v) is an element of D and Duu, v) is an element of D, respectively. Since À E R BxB the average pooling of E 2B2 we have
A(i,j) ΣΣΣ Σ A(2i m, 2
(49)
Plug Eq. (49) into Eq. (48), we obtain
D(u,v)
2 B
ΣΣ
A(ii m, n)) cos
(2i 1)ur
cos
(2j + 1)
(50)
2B
2B
m=0
Apply change variable 2i m and y 2j + n, Eq. (50) becomes
18

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
RGB
DCT block size=1
DCT
block size=2
DCT
block size 4
DCT block size=8
Figure The forward perturbation process of RGB and DCT using the forward SDE dæ. f o(t)dwr
10-128-1
(x
+ 1)TT
(y n + 1)UT
D(u, v)
Σ
Σ
A(r, v)
Σ
cos
cos
(51)
2B
2B
2B
m00
2B-12B 12-1 Σ
(xx 1) 1)UT
ZUT
(y + 1)Tr
you
A(x, 3)
cos
+ cos
COs
+ cos
(52)
2B T=0 y=0
2
2B
2B
2B
m==
n=
However, since i = ž-m m is an integer when applying the change of variable, r and m must both be odd or both be even (the same applies to y and n). Therefore, for any x, y in Eq. (52), only one term exists within each the two big parentheses. An approximation can obtained by taking the average of the two cosine terms within each bracket, which gives
2
D(u, v)
1 2B-12B-
Σ
Σ
A(x, y)
1
cos
(xr + 1)uT
+
CO8
1
cos
(yy + 1)UT
+ COs
yyu
(53)
2B
2B
2B
2
2B
2B
z=0
y=0
1 2B x=0
18--1281
(2x + 1)uT
(2y + 1)UT
Σ
Σ
A(c, y)
cos
cos
cos
+ 0o8
(54)
4B
L4B
4B
L4B
y=0
1
2B-1 28
(2x + 1)uT"
(2y + 1)UT
COS
COS
Σ
A(x, y) cos
cos
(55)
2
4B
4B
4B
4B
9=0
From Eq. (53) to Eq, (54) apply the trigonometric formulas
A B
A - B
cos(A) + cos( B) 2 cos(
cos(
2
2
Now comparing Eq. (55) with Eq. (47), we obtain
D(u,v)
COs
WT 4B
D(u,0) v)
(56)
L4B
which completes the proof of Theorem 5.2.
19

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
A.6. Training Parameters of UVIT, DiT and DCTdiff
We list the model and training parameters in Table 9 and Table 10 where the former compares UVTT and DCTdiff (inherited from UVIT) and the latter compares DiT and DCTdiff (inherited from DiT). We use the default training settings from UViT and DiT without any change.
Regarding the choice of DCTdiff parameters, we find that the fixed T 98 used for on all datasets, possibly due to the statistical consisteney of image frequency distributions. The block size B and SNR Scaling factor only depend on the image resolution, one can refer to Table to determine B and e given a new dataset. Finally, the frequency elimination parameter m² can be calculated from Eq. (6).
Scaling is effective
Table Training and network parameters of UVIT and DCTdiff on different datasets.
Transformer parameters
Learning parameters
DCTdiff parameters
Dataset
Model
parameters
patch size
# tokens
hatch size
learning
m"
UVIT DCTdiff
131M
4
64
256
0.0002
CIFAR-10
131M
(B
64
256
0.0002
98.25
UVIT DCTdiff
44M
4
256
256
0,0002
CelebA 64
44M
(
256
256
0.0002
98.25
UVIT DCTdif
44M
256
1024
0.0003
ImageNet 64
44M
4 (B
1024
0.0003
98.25
UVIT DCTdiff
44M
2.56
256
0,0002
FFHQ 128
44M
8 (B 1)
256
0.0002
98.25
UVIT (latent)
131M + 84M 131M
256
256
0.0002
FFHQ 256
DCTdiff
(B 4)
1024
256
0.0002
98.25
UVIT (latent)
131M + 84M 131M
128
0.0001
FPHQ 512
DCTdiff
= 8)
1024
128
0.0001
98.25
UVIT (latent)
131M + 84M
128
0.0001
AFHQ,512 512
DCTdiff
131M
(B = 8)
1024
128
0.0001
98.25
46
12
Table 10. Training and network parameters of DìT and DCTdiff on different datasets.
Transformer parameters
Learning parameters
DCTdiff parameters
Dataset
Model
# parameters
patch size
# tokens
batch size
learning rate
mn
DiT
58M
4
256
256
0.0001
CelebA 64
DCTdiff
58M
( 2)
256
256
0.0001
98.25
4
DIT DCTdiff
58M
8
256
256
0.0001
FFHQ 128
58M
7B 1)
256
256
0.0001
98.25
4
A.7. Evaluation of IS, Precision, Recall, and CMMD
In addition to the FID reported in the main paper, our evaluation of UVIT and DCTdiff also includes Inception Score (IS)
(Salimans et al., 2016), Recall, Precision (Kynkäänniemi et al., 2019), and CLIP Maximum Mean Discrepancy (CMMD)
(Jayasumana et al., 2024). As shown in Table 11, DCTdiff achieves better IS, CMMD, and mostly exhibits higher Precision and Recall than UVIT.
Table 11. Comparison between UVIT and DCTdiff on CIFAR-I0. 10, FFHQ 128 and AFHQ 512. We use DDIM sampler (NFE=100) for
CIFAR-10 10 and DPM-Solver (NFE=100) for the other two datasets.
CIFAR-10
FPHQ 128
AFHQ 512
Model
CMMD J
IS
Precision
Recall
CMMD
IS
Precision
Recall
CMMD
IS
Precision
Recall
UVI
0.052
7.08
0.668
0.589
0,610
3.54
0.648 0.668
0.485 0.512
0.373
11.00
0.547
0.496
DCTdai
0.043
7.70
0.660
0.606
0.470
3.67
0.335
11.00
0.632
0.496
20

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
A.8. Scalability of DCTdiff ImageNet 64x64
Similar the scalability experiments on FFHQ 128, we apply the network scaling strategy 'small, mid, and mid deep' on ImageNet 64 to further test the scalability of DCTdiff. Using the DDIM sampler and NFE= 100, summarize the FID-50k results:
DCTdiff (small), FI1D=8.69.
DCTdiff (mid), FID==4667.
DCTdiff (mid, deep). FID==.30.
FID comparison: 4.67 (DCTdiff, mid) vs. 5.85 (UViT, mid)
A.9. Inference Time and Generation Quality
In Section 6.3, we report the wall-clock inference time of DCTdiff and UVIT without considering their sampling quality. However, when considering inference time at comparable generation quality, DCTdiff shows clear advantages: latent UViT requires 20 minutes to achieve FID=11089, whereas DCTdiff achieves FID=8.04 in just minutes on FFHQ 512.
Table 12. Wall-clock inference time FFIQ 512x512. FID-50k is reported using DPM-Solver
NFE
Model
100
50
20
UVIT (latent)
20.2 min (FID 10.89)
13.4 min (FID 10.94)
9.3 min (FID 11.31)
7.9 min (FID 23.61) min (FID 19.67)
DCTdiff
47.8 min (FID 7.07)
23.9 min (FID 7.09)
9.6 min (FID 8.04)
Table 13, Wall-cloch inference time on AFHQ 512x512.
FID-50k is reported using DPM-Solver
NFE
Model
100
50
20
10
UVIT (latent)
20.2 min (FID 10.86)
13.4 (FID 10.86)
9.3 min (FID 11.94)
7.9 min (FID 28.31)
DCTdiff
47.8 min (FID 8.76)
23.9 min (FID 8.87)
9.6 min (FID 10.05)
4.8 min (FID 21.05)
A.10. Application of DCT Upsampling in Super-resolution Image Generation
ADM (Dhariwal & Nichol, 2021) proposed neural network- based upsampling method for super-resolution image generation. In detail, diffusion model D; generates a low-resolution (e.g., 64x64) image. This image is then upsampled to the target resolution (e.g., 256x 256) using pixel interpolation. The upsampled image, as a condition, is finally fed into another diffusion model Dr to learn the data distribution of the high-resolution images. Following this pipeline, we replace the pixel interpolation with DCT Upsampling to verify its feasibility in super-resolution generation. We evaluate the FID-10K using DDPM sampler (NFE=50) on ImageNet 256x256. Using the same pretrained model D₁. DCT Upsampling achieves FID 14.53, which is inferior to the pixel interpolation (FID 10.71). We believe this phenomenon is caused by the training bias, where Dù was only trained on the pixel interpolation images and had never seen the DCT upsampled images. A fair comparison is that D,, should be trained on the DCT upsampled images. We leave this to future work.
B. Ablation Study
In this section, we elaborate on the effect of each design factor of DCTdiff using the dataset FFHQ 128x 128 and the base model UVIT. We report the FID-10k using DPM-solver throughout the ablation section. Table 14 presents the ablation results and the first row shows FID-10k achieved by UVIT during training. We then progressively added each design element to the previous base model to examine the design space of DCTdiff:
UVIT (YCbCr) inherits from UViT and replaces the RGB pixel inputs with YCbCr inputs.
DCTdiff (ECS, m 0) integrates the DCT transformation and Hatropy-Consistens Scaling into UVIT (YCbCr).
21

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
DCTdiff (ECS, -) further eliminates 7 high-frequency coefficients.
DCTdiff (EBFR) adds Entropy-Based Frequency Reweighting DCTdiff (ECS, m 7),
DCTdiff (SNR) incorporates the SNR Scaling based on DCTdiff (EBFR)
Table 14, Ablation study of DCTdiff design factors: FID-10k of UVIT and DCTdiff during training on FFHQ 128.. The convergence
FID is marked as bold at each row.
Training steps
Row
Model
100k
150%
200k
250k
300%
350k
400k
450k
500k
550k
600k
650k
700k
750k
1
UVIT
70.67
40.64
24.72
17.88
14.73
13.65
12.64
12.10
11.27
11.17
11.02
10.84
10.58
10.60
UVIT (YChCr)
17.70
3
DCTdiff (ECS, m 0)
26.59
17.11
16.90
16.29
14.38
13.42
12.63
12.92
4
DCTdiff (ECS, m 8)
14.11
12.42
10.81
10.52
10.24
9.74
9.76
5
DCTdiT (ECS,
36.98
14.79
11.14
10.65
9.50
9.68
6
DCTaH (EBFR)
15.66
10.61
9.69
9.72
9.09
9.20
7
DCTdiff (SNR,
18.32
9.19
8.54
8.32
7.94
9.16
8
DCTdiff (SNR, 1)
20.60
18.90
8.49
8.07
7.64
7.73
B.1. Ablation Study: YCbCr Accelerates the Diffusion Training
To evaluate the effect of YCbCr color space transformation in DCTdifff we substitute the RGB inputs with YCbCr (2x chroma subsampling) input. The corresponding results shown in the second row of Table 14, indicating that YCbCr with chroma subsampling dramatically accelerates the diffusion training but the cost of generative quality (FID-10K increases from 10.58 to 14.79). We believe the chroma subsampling provides the training acceleration, but the reduced color redundancy causes a drop in generation quality.
B.2. A blation Study: DCT and Entropy-Consisten Scaling
As we mentioned in Section 4.4, Entropy-Consistern Scaling (ECS) is a key factor making the DCT generative modeling effective. In detail, DCTdiff (ECS) not only enjoys the training acceleration benefit of YCbCr subsampling, but also yields a better generative quality than UViT (YCbCr) (see Table 14). We attribute the improvement of generation quality to the DCT space where low-frequency coefficients occupy the majority of image information.
B.3. Ablation Study: Eliminating m High-frequency Coefficients
In Table 14, we show the effect eliminating m m refers to maintaining all coefficients for diffusion training and sampling. By comparing Row 3, Row and Row of Table 14, it is clear that ignoring a suitable amount of high-frequency signals increases the generative quality and boosts the training, too. The optimal m be decided via Eq. (6).
coefficients in each DCT block. The block size is 4 and
B.4. Ablation Study: Entropy-Based Frequency Reweighting
In Section 5.1, we highlight the frequency prioritization property of DCT image modeling in which some frequency coefficients can be modeled preferentially according to the task prior knowledge. We adopt the Entropy-Based Frequency Reweighting (EBFR) for image generative modeling tasks low-frequency coefficients have large entropy and contribute more to the visual quality of images than high-frequency signals. Row 5 and Row 6 of Table 14 demonstrate that EBFR improves the generation quality of DCTdiff without affecting the training convergence.
B.5. Ablation Study: SNR Scaling of Noise Schedule
Since the block size B affects the forward perturbation process of DCTdiff (detailed in Section 4.5), we propose SNR Scaling for DCTdiff to scale the noise schedule of UVIT by a constant e. Row 7 and Row 8 of Table 14 show that SNR Scaling significantly improves the generative quality of DCTdiff and wide range parameter e can yield the FID improvement. We also visualize the effect of c the perturbation process of DCTdiff in Figure where the image size is 128 128 and the block size is 4.
22

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
DCTdiff (c 1)
DCTdiff (c
Figure Visualization of the forward SDE process. UVIT and DCTdiff (c share the same noise schedule, while DCTdiff (c scales up the noise schedule by a factor of 4.
B.6. Ablation Study: Block Size B
Although have provided the choice of block size B for image resolutions ranging from 32x 512x512 in Table 9, it is worth discussing its effect and importance diffusion modeling in detail. Similar to the conclusion of patch size P mentioned in UVIT (Bao et al., 2023) and DiT (Peebles & Xie, 2023), also confirm that the block size B used in diffusion Transformers must be relatively small (smaller than the usual P = 16 in image classification). Since we always have the relationship P 2B, one can determine B based on the value of P used UVIT and DiT when the resolution below 256 256. Additionally, the value of greatly affects the generative quality and training convergence of DCTdifff: a larger B leads to faster training but sacrifices the generation quality. Table 15, we show the FID DCTdiff on the dataset FFHQ 256 with different B.
Table 15. Ablation study of block size B: FID-10k of DCTdiff during training on FFHQ 256 with different block sizes. The convergence FID is marked bold in each
Training steps
Model
50k
100k
150k
200k
250k
300k
350k
DCTdiff (B=4)
125.76
44.67
10.58
8.94
7.07
6.97
7.39
DCTdiff (B=8)
213.14
19.75
18.41
20.37
C. Qualitative Results
C.1. Qualitative Comparison between VAE Compression and DCT Compression
We randomly sample several images from ImageNet 256x256 dataset, then we perform VAE compression and DCT compression (4x compression ratio). The reconstructed images after compression are shown in Figure 9. From this, we clearly see that VAE compression loses image details and local image structure while DCT compression maintains most of the image information. Also, find that VAE compression not good reconstructing letters, digital numbers, and unseen images (not trained by VAE). In contrast, DCT compression training-free, computationally negligible, and insensitive image domains.
C.2. Qualitative Results of DCT Upsampling
In Figure 10, we show the visual differences between Pixel Upsampling by bicubic interpolation and DCT Upsampling.
C.3. Qualitative Results of DCTdiff
We show the uncurated samples generated by DCTdiff:
Figure 11, ImageNet 64x64, FID -4300, generated by DDIM sampler with NFE-100
Figure 12, FFHQ 128x 128, FID=4.98, generated by DPM-Solver with NFE= 100
23

---

12
11
1
10
2
WHICH COUNTRY HAS THE LONGEST NAME WITH NO LETTER REPEATED
7
6
SWTTZZLLAND
12
11
1
O WHOO COIATIT HAS TNEEON NAVE WITE VO J STTER REREATED
10
2
9
3
7
ONYTEZZUUNN
12
11
1
O WHICH COUNTRY HAS THE LONGEST LETTER REPEATED?
10
2
9
3
7
SWITTEERAND
Figure 9. Qualitative comparison between VAE compression and DCT compression. The first row shows the raw images (sampled from ImageNet 256x256). The sccond and third rows present the reconstructed images after VAE compression and DCT compression (4x).
Figure 13, FFHQ 256x256, FID=5.08, generated by DPM--olver with NFE= 100
Figure 14, FFHQ 512x512, FID=7.07, genneeae by DPM-Solver with NFE= 100
Figure 15, AFHQ 512x512, FID-8.76. generated by DPM-Solver with NFE- 100
24

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
(b) 128 128 (downsampled from 256x256)
(a) 256 (ground truth)
(c) 256x256 (Pixel Upsampling)
(d) 256 (DCT Upsampling)
Figure 10. Comparison between Pixel Upsampling and proposed DCT Upsampling. 10(c) and 10(d) are upsampled based 10(b).
Figure 11, Uncurated samples generated by dCT trained on the datasct ImageNet 64x64 (FID= 4.30).
25

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Figure 12, Uncurated samples generated by DCTdiff trained on the dataset FFHQ 128x 128 (FID= 4.98).
Figure 13. Uncurated samples generated by DCTdiff trained the dataset FFHQ 256x256 (FID= 5.08).
Figure 14. Uncurated samples generated by DCTdiff trained on the dataset 512x512 (FID= 7.07).
26

---

DCTdiff: Intriguing Propertics of Image Generative Modeling in the DCT Space
Figure 15. Uncurated samples generated by DCTdiff trained on the dataset AFHQ 512x512 (FID= 8.76).
27


Reviews:
[{"review_id": "NVZX8VtKEp", "reviewer": "Reviewer_VaLT", "summary": "The paper propose DCTDiff that models images in the discrete cosine transform (DCT) space. The paper discusses the design space of DCTdiff and reveals interesting properties of image modeling in the DCT space such as spectral autoregression nature of pixel diffusion models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments which help improve the paper.\n\n**Q1: The paper claims that \"DCT Upsampling Outperforms Pixel Upsampling\". However, it is only compared against interpolation methods**\n\nThe upsampling we mean in the paper is indeed interpolation. We will make this statement clearer.\n\n**Q2: What is the inference time comparison**\n\nPixel diffusion and DCTdiff share the same GFLOPs and inference time (Table 1)\n\nTable 1. wall-clock time (NFE=100, 10k samples, one A100)\n\n|  | CIFAR-10 | CelebA 64 | FFHQ 128 |\n| --- | --- | --- | --- |\n| UViT | 2m 54s | 5m 06s | 5m 14s |\n| DCTdiff | 2m 58s | 5m 11s | 5m 12s |\n\nComparing latent diffusion and DCTdiff, the inference time is different because\n\n- GFLOPs of UViT (latent) = 34*NFE (diffusion) + 1240 (Decoder)\n- GFLOPs of DCTdiff = 133*NFE (diffusion)\n\nThe decoder of the VAE is expensive, but DCTdiff has a larger complexity in diffusion as DCTdiff has 1024 tokens and UVIT has 256 tokens. Table 2 shows that DCTdiff is faster in low NFE but is slower in the high NFE condition.\n\nTable 2. Wall-clock inference time on AFHQ 512 (10k samples, A100 GPU). GFLOPs is appended in the brackets.\n\n|  | NFE=100 | NFE=50 | NFE=20 | NFE=10 |\n| --- | --- | --- | --- | --- |\n| UViT (latent) | 20m 14s (4640) | 13m 24s (2940) | 9m 18s (1920) | 7m 57s (1580) |\n| DCTdiff | 47m 50s (13300) | 23m 53s (6650) | 9m 34s (2660) | 4m 47s (1330) |\n\n**Q3: The performance gap diminishes as NFE increases**\n\nIn most cases, our DCTdiff has significantly lower FID (20%~30%) than the base model. We believe the small improvement on ImageNet 64 is due to the small model capacity since we have limited GPU to scale on ImageNet. However, we do provide the scaling experiments on other datasets (Table 6,7)\n\n**Q4: How about evaluation using IS, Precision/Recall and CMMD**\n\nTable 3. Comparison between UViT and DCTdiff on CIFAR-10 using DDIM sampler (NFE=100)\n\n|  | FID ↓ | CMMD ↓ | IS ↑ | precision ↑ | recall ↑ |\n| --- | --- | --- | --- | --- | --- |\n| UViT | 5.05 | 0.052 | 7.08 | 0.668 | 0.589 |\n| DCTdiff | 4.25 | 0.043 | 7.70 | 0.660 | 0.606 |\n\nTable 4. Comparison between UViT and DCTdiff on FFHQ 128 using DPM-solver (NFE=100)\n\n|  | FID ↓ | CMMD ↓ | IS ↑ | precision ↑ | recall ↑ |\n| --- | --- | --- | --- | --- | --- |\n| UViT | 9.18 | 0.610 | 3.54 | 0.648 | 0.485 |\n| DCTdiff | 6.50 | 0.470 | 3.67 | 0.668 | 0.512 |\n\nTable 5. Comparison between UViT and DCTdiff on AFHQ 512 using DPM-solver (NFE=100)\n\n|  | FID ↓ | CMMD ↓ | IS ↑ | precision ↑ | recall ↑ |\n| --- | --- | --- | --- | --- | --- |\n| UViT (latent) | 10.86 | 0.373 | 11.00 | 0.547 | 0.496 |\n| DCTdiff | 8.76 | 0.335 | 11.00 | 0.632 | 0.496 |\n\n**Q5 Missing experiments of model scalability**\n\nWe performed scaling experiments on CIFAR-10 and FFHQ 128 during the short rebuttal period.\n\nTable 6. FID-50k on CIFAR-10 using DDIM sampler, patch_sz=4\n\n|  | NFE=100 | NFE=50 | NFE=20 |\n| --- | --- | --- | --- |\n| UViT (small) | 7.25 | 8.45 | 21.18 |\n| DCTdiff (small) | 6.51 | 6.62 | 7.87 |\n|  |  |  |  |\n| UViT (mid) | 6.23 | 7.88 | 20.48 |\n| DCTdiff (mid) | 5.02 | 5.21 | 6.81 |\n|  |  |  |  |\n| UViT (mid, deep) | 6.05 | 7.33 | 20.27 |\n| DCTdiff (mid, deep) | 4.25 | 4.54 | 5.96 |\n\nTable 7. FID-50k on FFHQ 128 using DPM sampler\n\n|  | NFE=100 | NFE=50 | NFE=20 |\n| --- | --- | --- | --- |\n| DCTdiff (small) | 6.50 | 6.55 | 7.72 |\n| DCTdiff (mid) | 5.13 | 5.20 | 6.19 |\n| DCTdiff (mid, deep) | 4.98 | 5.05 | 5.94 |\n\n**Q6 Compare DCTransformer with DCTdiff**\n\nDCTdiff differs from DCTransfomer in several key aspects: probabilistic modeling, image representation, network and image tokenization. We summarize their differences in Table 8. Overall, DCTdiff exhibits a straightforward way for image frequency generative modeling. Performance-wise, DCTdiff achieves FID 7.28 on FFHQ while DCTransformer has FID 13.06.\n\nThe only overlap between DCTransformer and DCTdiff is using the DCT transform and YCbCr color transform. But these well-known techniques of JPEG compression cannot be credited to DCTransformer.\n\nTable 8. Differences between DCTransformer and DCTdiff\n\n|  | DCTransformer | DCTdiff |\n| --- | --- | --- |\n| Probability modeling | Autoregression (each conditional distribution is further factorized into 3 distributions, see their Eq(2)) | Diffusion |\n| Image representation | tuples (channel, position, value) | Y, Cb, Cr |\n| Network | a Transformer with 1 encoder and 3 decoders used for predicting channel, position, value, respectively | a single diffusion Transformer |\n| use quantization | yes (has information loss) | no |\n| DCT block size | fixed (8/16) | flexible for different resolution generation |\n\n**Q7: What is the limitation of this paper?**\n\nThe limitation is that we did not explore other generative applications (e.g. image inpainting) and discriminative tasks. Also, frequency-oriented Transformer architecture and super-resolution image generation were not covered in this paper. We think these are promising directions for future work.", "claims_and_evidence": "The paper claims that \"DCT Upsampling Outperforms Pixel Upsampling\". However, it is only compared against interpolation methods and not any super resolution methods.\n\nTable 4 shows that the training cost for DCTdiff is lower than UViT in terms of GFLOPs. However, the actual wall clock time will be different due to GPU optimizations. What is the inference time comparison for the two models?\n\nThe gap in the performance difference between the two methods diminishes as the NFE increases, especially for class conditional generation (Table 2). So, the advantage of the proposed method is not clear when scaling the test time compute.", "methods_and_evaluation_criteria": "The proposed method is quantitatively evaluated only using FID metric. How about Inception Score and Precision/Recall [1] scores? \nThere have been better metrics proposed in recent years such as CMMD [2]. How about the performance comparisons using CMMD?\n\n[1] Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. CVPR, 2019\n[2] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, Sanjiv Kumar, Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR 2024", "theoretical_claims": "Yes, no major concerns", "experimental_designs_or_analyses": "Missing experiments on the scalability of the proposed method. Does the performance improve with increasing size of the model?", "relation_to_broader_scientific_literature": "The key contribution of the method is to use DCT space for image modeling instead of the pixel space. However, prior work has already shown that DCT space is effective for image generation and so the advantage of the proposed method over the prior work is not clear.", "essential_references_not_discussed": "The results do not compare against the prior work DCTransformer [1] that also uses DCT space for image modeling. How is the proposed method better? A detailed comparison of the similarities and differences with [1] will elucidate the advantages of the proposed DCTdiff.\n\n[1] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. ICML, 2021", "other_strengths_and_weaknesses": "The paper does not discuss the limitations of the proposed method.", "other_comments_or_suggestions": "Typo - Table 5 should be DiT", "questions_for_authors": "I would like to see answers specifically to the following questions in the rebuttal.\n\n1. How is the proposed method compared to closely related DCTransformer? Is training a DiT-based model with the dense DCT image from DCTransformer better?  \n\n2. How about performance comparisons using Inception Score. Precision/Recall scores and CMMD metric?\n\n3. What is the wall clock inference time comparison?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DX4ahwdyTq", "reviewer": "Reviewer_vBsQ", "summary": "The paper introduces an end-to-end diffusion modeling framework in the frequency space, instead of in the original pixel space. It shows that the DCT (discrete cosine transform) space could be an effective and near-lossless compression for diffusion modeling, mitigating pixel redundancy and enabling efficient scaling to 512×512 image generation without requiring auxiliary VAEs.\n\nThe authors propose a pipeline for token preparation for Diffusion Transformers, accompanied by adjustments to hyperparameters such as noise schedules and loss re-weighting. Experimental results on UViT and DiT architectures show that DCT-based diffusion models outperform pixel-space and latent-space counterparts on FID scores and training efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the detailed and helpful reviews.\n\n**Q1: The paper only evaluates some generative tasks, while the capability of DCT space on other generative and discriminative tasks is still unknown**\n\nWe will first rephrase this sentence to avoid any misunderstanding. Our work primarily aims to explore the feasibility of performing diffusion in the DCT space for image generation, with a particular focus on generative capabilities. While we acknowledge the potential discriminative properties of DCT-based modeling, we believe these properties are primarily derived from the generative pre-training mechanism rather than the information format itself. This could also explain why both pixel and latent diffusion models exhibit similar characteristics. We leave the investigation of the discriminative, editing, inpainting and restoration ability of DCT-based diffusion models as future work.\n\n**Q2: The comparison is restricted to latent diffusion using SD-VAE, an outdated compression model, Can the DCT space outperform these modern tokenizer?**\n\nFirst, we will rewrite this claim by clarifying that the latent diffusion is SD-VAE based, making the scope of our claim clearer. Second, Our goal is not to beat these emerging tokenizers with dedicated designs, we explore how far the image diffusion modeling can go without a pretrained tokenizer. Moreover, our work does not conflict with image tokenizers, e.g. we totally agree that exploring DCT tokenizer is promising, and the technique of these new tokenizers can also be used to develop DCT tokenizer.\n\n**Q3: Can you provide some results on ImageNet-256?**\n\nGiven the limited GPUs, we have prioritized our experiments on the other tasks: scaling up, extra evaluations, and explorations of UNet-based DCTdiff.\n\n**Q4: The baseline on CIFAR-10 appears under-optimized**\n\nFor fair comparison, our implementation of UViT and DCTdiff on CIFAR-10 used patch size (ps=4) (different from ps=2 in original UViT paper). If we apply ps=2 on DCTdiff, block size will be 1, yielding only the DC frequency and preventing frequency reweighting. To address your concern, we have implemented another baseline of UViT using ps=2 (256 tokens). Correspondingly, we increase the model size of DCTdiff to ensure the same computational complexity for a single network forward pass. Results on Tables 9 and 10 show that DCTdiff is significantly better than UViT in terms of FID. We acknowledge that the FID 5.05 on the UViT is still higher than the 4.5 you tested (might be due to the warmup steps, machine and software difference etc.), but our code and all trained models will be released for reproducibility.\n\nTable 9. Extra CIFAR-10 benchmark, FID-50k using DDIM sampler\n\n|  | 100 | 50 | 20 | 10 |\n| --- | --- | --- | --- | --- |\n| UViT (small, 256 tokens) | 5.05 | 6.24 | 17.83 | 73.05 |\n| DCTdiff (mid_deep, 64 tokens) | **4.25** | **4.54** | **5.96** | **11.17** |\n\nTable 10. Extra CIFAR-10 benchmark, FID-50k using DPM-solver\n\n|  | 100 | 50 | 20 | 10 |\n| --- | --- | --- | --- | --- |\n| UViT (small, 256 tokens) | 4.82 | 4.85 | 4.92 | 10.72 |\n| DCTdiff (mid_deep, 64 tokens) | **4.40** | **4.43** | **4.56** | **8.82** |\n\n**Q5: Can the proposed DCT-based diffusion be adapted to UNet backbones and also outperform pixel-space counterparts?**\n\nIt is an interesting question. Although we think that Transformer has many advantages than UNet regarding the image DCT implementation: ease of (1) Cb Cr 2x subsampling, (2) elimination of high frequencies, and (3) frequency loss reweighting, we have investigated the possibility of UNet-based DCTdiff using the ADM codebase. Concretely, we remove the Cb Cr 2x subsampling and loss reweighting in the preliminary experiment, and just convert the RGB channel to YCbCr followed by DCT transform, the resulting frequencies yield the input tensor with shape (32, 32, 3) on CIFAR-10. We trained the UNet-based DCTdiff for 400k steps, the FID is shown in Table 11 and the results are promising. We believe further exploration of UNet-based DCTdiff will lead to better generation quality.\n\nTable 11. FID of UNet-based DCTdiff on CIFAR-10 (NFE=100).\n\n| training steps | 200k | 300k | 400k |\n| --- | --- | --- | --- |\n| DCTdiff (UNet) | 5.06 | 4.88 | 4.48 |\n\n**Q6: ViT-based architecture is more preferable when scaling to larger network capacities**\n\nWe provide the scaling experiments below.\n\nTable 6. FID on CIFAR-10 using DDIM sampler, ps=4\n\n|  | NFE=100 | NFE=50 | NFE=20 |\n| --- | --- | --- | --- |\n| UViT (small) | 7.25 | 8.45 | 21.18 |\n| DCTdiff (small) | 6.51 | 6.62 | 7.87 |\n|  |  |  |  |\n| UViT (mid) | 6.23 | 7.88 | 20.48 |\n| DCTdiff (mid) | 5.02 | 5.21 | 6.81 |\n|  |  |  |  |\n| UViT (mid, deep) | 6.05 | 7.33 | 20.27 |\n| DCTdiff (mid, deep) | 4.25 | 4.54 | 5.96 |\n\nTable 7. FID on FFHQ 128 using DPM sampler\n\n|  | NFE=100 | NFE=50 | NFE=20 |\n| --- | --- | --- | --- |\n| DCTdiff (small) | 6.50 | 6.55 | 7.72 |\n| DCTdiff (mid) | 5.13 | 5.20 | 6.19 |\n| DCTdiff (mid, deep) | 4.98 | 5.05 | 5.94 |", "claims_and_evidence": "- \"*suggesting its potential for both discriminative and generative tasks*\" (Lines 67-68).\n  - The paper only evaluates some generative tasks (unconditional or class-conditional image synthesis), while the capability of DCT space on other generative (e.g. image editing, inpainting and restoration) and discriminative (e.g. image classification and segmentation) tasks is still unknown.\n  - Recent studies show that intermediate representations within diffusion networks are effective for discriminative tasks through generative pre-training, observed in both pixel-space (DDAE, arXiv:2303.09769; DDPM-seg, arXiv:2112.03126) and latent-space models (l-DAE, arXiv:2401.14404; REPA, arXiv:2410.06940). Does DCT-based modeling retain similar properties?\n\n- \"*outperforms the pixel-based and latent diffusion models regarding generation quality and training speed*\" (Lines 62-64).\n  - The comparison is restricted to latent diffusion using SD-VAE, an outdated compression model. Many modern image tokenizers (VA-VAE, arXiv:2501.01423; MAETok, arXiv:2502.03444) can also achieve near-lossless reconstruction (rFID < 0.5) and provide compact, diffusion-optimized spaces. Can the DCT space outperform these modern tokenizer specifically designed for \"*more diffusible*\" latent representations?\n  - I understand that complete training on large datasets like ImageNet256x256 is too resource-intensive. However, as mentioned above, recent tokenizer advancements have reduced costs (e.g. 10-20x faster convergence). Therefore, please, as much as you can afford (e.g. even with limited training in a few hundred epochs), provide some results on standard benchmarks like ImageNet-256x256.\n  - The baselines also appear under-optimized. For example, I can achieve an FID of 4.5 on unconditional CIFAR-10 with a 100-NFE DDIM sampler, whereas Table 2 reports FIDs of 5.29-6.23. This suggests the pixel- and latent-space models may not be fully converged, undermining the claimed outperformance.", "methods_and_evaluation_criteria": "The proposed method is technically sound and novel, but its applicability appears limited to ViT-based backbones (e.g. UViT, DiT).\n- UNet-based architectures remain competitive, particularly for unconditional and class-conditional image generation on datasets like CIFAR-10, FFHQ, and ImageNet-64 (EDM2, arXiv:2312.02696). Can the proposed DCT-based diffusion be adapted to UNet backbones and also outperform pixel-space counterparts?\n- ViT-based architecture, on the other hand, is more preferable when involving text modalities (e.g. text-to-image synthesis) and scaling to larger network capacities. However, the provided evaluations align more closely with UNet strengths rather than ViT advantages.", "theoretical_claims": "I do not find any obvious errors in the theoretical analysis.", "experimental_designs_or_analyses": "Please refer to the second discussion in \"Claims And Evidence\" section.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "The essential related works are mostly discussed.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Typo: \"UViT\" in Table 5 should be \"DiT\".", "questions_for_authors": "After reading the paper carefully, I acknowledge that the DCT-based space is indeed a more preferable replacement for the raw pixel-space (if the claimed \"*potential for both discriminative and generative tasks*\" is correct and sound). However, as mentioned above, I doubt whether it is better than modern VAE/AE-based image tokenizers. Those tokenizers compress images with higher compression ratio, offering a more compact and \"*diffusible*\" latent space for diffusion modeling, and may optionally preserve semantic information for unified generation-and-understanding tasks, particularly for multi-modal models. Btw, I think training an auxiliary tokenizer on the DCT space (instead of pixel-space) may also sound reasonable.\n\n\n**2025/04/04: [Replying to Reply Rebuttal Comment by Authors]**:\n\nThank you for the additional discussion.\n\nRegarding the performance and efficiency comparison, I acknowledge the authors' point that evaluating DCT-space designs against latent-space sampling involves complexities influenced by factors such as model size, patch size, and NFEs. The revised Tables 2a & 2b offer improved clarity. For the final version, I suggest incorporating more direct visualizations, similar to the performance trade-off curves (e.g., Fig. 1, 8, 9 presented in EDM2 [arXiv:2312.02696]), to further enhance the presentation.\n\nI also accept the explanation that the implementation is particularly well-suited for Transformer-based models. The current UNet-based comparison, while noted as incomplete, is acceptable and does not critically impact my overall assessment.\n\nRegarding the UViT baselines, I am glad the authors confirm that the reimplementation is strong and valid. The patch_size in UViT on 512x512 is indeed 4 (instead of the common practice ps=2 in modern architectures). I apologize for the false alarm.\n\nBased on the clarifications provided, I am raising my rating to Weak Accept (actually not very curcial since other reviews are all positive). I strongly recommend that the authors dedicate to reorganizing the results presentation and refining the training details in the final version for improved clarity.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7RaNwdBcCV", "reviewer": "Reviewer_5JtJ", "summary": "In this work authors introduce a novel idea to model images in their frequency spaces with diffusion models. Authors show that they can use Diffusion Transformer architectures to model the frequencies of images in a smart way without changing the architecture. There are several new observations on how to achieve this task regarding the need for new scaling etc. With the series of experiments it is shown that the method outperforms pixel-based alternative baseline approach achieving good level of generation quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the insightful suggestions and comments provided by the reviewer.\n\n**Q1: The benchmarks used for the evaluation are sufficient in size, authors even include some high-resolution datasets. Nevertheless the method is only compared with the baseline model with the same architecture run in the pixel space**\n\nIn addition to the comparison with pixel-based diffusion models, we also compared DCTdiff with the latent diffusion model UViT which utilizes the VAE of StableDiffusion for image compression. Please refer to Table 3 of our paper for the details.\n\n**Q2: Theorem 5.1 might be a bit relevant to the proposed approach, but it does not provide theoretical grounds for the proposed method.**\n\nTheorem 5.1 provides a theoretical foundation for understanding how noise affects different frequency components in the diffusion process. It formalizes the observation that high-frequency signals are more susceptible to noise perturbations due to the energy concentration properties of the DCT. This insight is crucial for justifying why DCTdiff exhibits a faster noise accumulation process than pixel diffusion, ultimately motivating the introduction of the SNR scaling method. We will improve the writing of Section 5.3 to highlight the connection.\n\n**Q3: The evaluation is a comparison to the baseline UViT on a number of different datasets including some high-resolution ones. In those experiments DCTdiff consistently outperforms pixel-space UViT. However, the observed results are far from state-of-the-art (e.g. FID on FFHQ256 2.19 for StyleGAN from 2022 vs 5.08 from this work).**\n\nThanks for the interesting observation. GANs have been a strong class of models that generate high fidelity images (often measured by FID) (more evaluations can be found in https://paperswithcode.com/sota/image-generation-on-ffhq-256-x-256) yet do not produce diverse images. The main advantages of diffusion models are that they are stable to train and have higher diversity (often measured by recall) than GANs (see papers [1][2]). Our DCTdiff actually achieves better FID than many other diffusion-based models, e.g [3] and [4]).\n\n[1] Dhariwal, Prafulla, and Alexander Nichol. \"Diffusion models beat gans on image synthesis.\" NIPS. 2021.\n\n[2] Boutin, Victor, et al. \"Diffusion models as artists: are we closing the gap between humans and machines?.\" ICML 2023.\n\n[3] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" *CVPR*. 2022.\n\n[4] Kim, Dongjun, et al. \"Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation.\" ICML. 2022.\n\n**Q4 I think the one important reference missing is [1] where authors combine a hierarchical VAE with a diffusion model by using a Discrete Cosine Transform (DCT) to create low-dimensional pseudoinputs, which are then modeled via a diffusion-based prior to efficiently approximate the aggregated posterior and enhance latent space utilization**\n\nExcept for explicit signal compression, it is inspiring to see that DCT can be treated as a low-dimensional latent representation and applied to VampPrior framework to obtain the flexible prior distribution modeled by diffusion. We will add the discussion of [1] to our literature review.\n\n[1] Kuzina, Anna, and Jakub M. Tomczak. \"Hierarchical VAE with a Diffusion-based VampPrior.\" Transactions on Machine Learning Research.\n\n**Q5: Have you considered using different terminal distribution for the diffusion model instead of gaussian? Maybe something like Pareto distribution would be more suitable for the unusual distribution defined by DCT? Such an approach might also help for the SNR Scaling**\n\nWe sincerely appreciate the reviewer providing this insightful suggestion. In our DCTdiff work, we follow the Gaussian prior of diffusion due to the fair comparison. But we do think that Pareto distribution which exhibits the power-law (same as image spectral distribution) is very likely to be a great solution for DCT distribution modeling. Moreover, Laplace distribution is also worth exploring for DCT modeling. We will investigate this topic in our next work.", "claims_and_evidence": "The submission introduces a new method that uses in a brilliant way several observations from compression. Those combinations of known tricks from JPEG with generative modelling are not trivial, and compelling:\n- The observation that DCT blocks correspond to VIT patches is clever and insightful\n- The zigzag flattening (originally introduce in JPEG) together with reconstruction FID metric that is used for estimating the best trade-off between generation quality and compression rate (speed) introduces great controllability of those two crucial elements in generative modeling with diffusion models.", "methods_and_evaluation_criteria": "The benchmarks used for the evaluation are sufficient in size, authors even include some high-resolution datasets. Nevertheless the method is only compared with the baseline model with the same architecture run in the pixel space. This is fine regarding that the contribution is solely limited to the change of the modeling space, but might be seen as a limitation of the evaluation.", "theoretical_claims": "There are 3 Theorems with \"Sketch of Proofs\" in the submission, while there are detailed proofs in the appendix. I did not check the correctness of all proofs, but I am unsure if Theorem 5.1 which discusses connection between pixel-based diffusion and autoregressive modeling is really necessary for this work. I see how it might be a bit relevant to the proposed approach, but it does not provide theoretical grounds for the proposed method.", "experimental_designs_or_analyses": "- The evaluation of upsampling is very limited to comparison with baseline pixel upsampling. This is fine as a proof of concept, but please note that novel diffusion architectures (e.g. DeepFloyd IF) use more advanced approaches also employing several diffusion steps. \n- The final evaluation is a comparison to the baseline UViT architecture on a number of different datasets including some high-resolution ones. In those experiments DCTdiff consistently outperforms pixel-space UViT. However, the observed results are far from state-of-the-art (e.g. FID on FFHQ256 2.19 for StyleGAN from 2022 vs 5.08 from this work).", "relation_to_broader_scientific_literature": "This submission well discusses the broader scientific literature", "essential_references_not_discussed": "I think the one important reference missing is [1] where authors combine a hierarchical VAE with a diffusion model by using a Discrete Cosine Transform (DCT) to create low-dimensional pseudoinputs, which are then modeled via a diffusion-based prior to efficiently approximate the aggregated posterior and enhance latent space utilization.\n\n[1] Kuzina, Anna, and Jakub M. Tomczak. \"Hierarchical VAE with a Diffusion-based VampPrior.\" Transactions on Machine Learning Research.", "other_strengths_and_weaknesses": "Strengths:\n- The proposed method is novel, sound, and makes a lot of sense-\n- This work introduces several smart observations from frequency space analysis and compression methods into generative modelling.\n\nWeaknesses:\n- The evaluation of the methods is limited to the comparison to the baseline approach in the pixel space", "other_comments_or_suggestions": "Small suggestions:\n- I don't know how was it defined, but the bold math letters (e.g. x'_y and so on in line 121 right) are hard to read", "questions_for_authors": "- The scaling method introduced in the work is a challenging task. The proposed Entropy-Consistent Scaling is reasonable, but have you considered using different terminal distribution for the diffusion model instead of gaussian? Maybe something like Pareto distribution would be more suitable for the unusual distribution defined by DCT? Such an approach might also help for the SNR Scaling", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ePzx6NR3A", "reviewer": "Reviewer_guDg", "summary": "This paper introduces a new paradigm in diffusion models by using DCT coefficients, specifically low-frequency components, as operands instead of pixel or latent representations. Inspired by JPEG compression, this method aims to improve efficiency. The model achieves 512x512 resolution without latent representations. The paper also claims that diffusion models can be interpreted as spectral autoregression.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the reviews and constructive suggestions.\n\n**Q1: the main paper lacks qualitative figures, and the appendix only includes Figs. 9-11, missing key results for FFHQ 512**\n\nWe will add more qualitative samples in the appendix, including the randomly drawn ones, and compare them with the baseline as well. Surely, the qualitative results on FFHQ 512x512 will be added to the paper, too.\n\n**Q2: Theorems 5.1 and 5.2 appear correct but do not significantly contribute to the main claims. Theorem 5.1: States that diffusion removes high-frequency components first and low-frequency components later. This is well known (Patashnik et al., ICCV 2023) and not specific to DCTdiff. Theorem 5.2: Reformulates existing DCT-based upsampling methods in a mathematical form.**\n\nTheorem 5.1 provides a theoretical foundation for understanding how noise affects different frequency components in the diffusion process. It formalizes the observation that high-frequency signals are more susceptible to noise perturbations due to the energy concentration properties of the DCT. This insight is crucial for justifying why DCTdiff exhibits a faster noise accumulation process than pixel diffusion, ultimately motivating the introduction of the SNR scaling method. Regarding Theorem 5.2, we present an alternative derivation that differs from the existing approach in the literature. We believe that DCT-based upsampling holds significant potential in deep learning applications. We consider moving Theorem 5.2 to the appendix while emphasizing its potential in the main paper. In summary, we will refine the writing in these two paragraphs by adding proper citations and explicitly highlighting the message we want to deliver in this paper.\n\n**Q3: The metrics and baselines are limited (only FID and UViT), which may not ensure a fair comparison. A broader evaluation with models trained on the same datasets (even if #Parameters, GFLOPs, and training steps differ) would improve the study. Additional metrics, such as Inception Score, could be included.**\n\nIn addition to UViT, we have also provided the results on the baseline DiT (please refer to our paper). The reason for choosing these two base modes is that they are well-known models and widely used in the community. Evaluating DCTdiff on more base models is valuable as long as the base model is easy to implement. For example, we have preliminarily implemented DCTdiff using the UNet architecture, the results are promising (Table 11). We believe that DCTdiff is a general method to be applied in different diffusion networks.\n\nTable 11. FID of UNet-based DCTdiff on CIFAR-10 (NFE=100).\n\n| training steps | 200k | 300k | 400k |\n| --- | --- | --- | --- |\n| DCTdiff (UNet) | 5.06 | 4.88 | 4.48 |\n\nRegarding extra metrics, we have added IS, precision, recall, CMMD (as suggested by reviewer VaLT) to evaluate the models. The results are shown in Tables 3, 4, 5.\n\nTable 3. Comparison between UViT and DCTdiff on CIFAR-10 using DDIM sampler (NFE=100)\n\n|  | FID ↓ | CMMD ↓ | IS ↑ | precision ↑ | recall ↑ |\n| --- | --- | --- | --- | --- | --- |\n| UViT | 5.05 | 0.052 | 7.08 | 0.668 | 0.589 |\n| DCTdiff | 4.25 | 0.043 | 7.70 | 0.660 | 0.606 |\n\nTable 4. Comparison between UViT and DCTdiff on FFHQ 128 using DPM-solver (NFE=100)\n\n|  | FID ↓ | CMMD ↓ | IS ↑ | precision ↑ | recall ↑ |\n| --- | --- | --- | --- | --- | --- |\n| UViT | 9.18 | 0.610 | 3.54 | 0.648 | 0.485 |\n| DCTdiff | 6.50 | 0.470 | 3.67 | 0.668 | 0.512 |\n\nTable 5. Comparison between UViT and DCTdiff on AFHQ 512 using DPM-solver (NFE=100)\n\n|  | FID ↓ | CMMD ↓ | IS ↑ | precision ↑ | recall ↑ |\n| --- | --- | --- | --- | --- | --- |\n| UViT (latent) | 10.86 | 0.373 | 11.00 | 0.547 | 0.496 |\n| DCTdiff | 8.76 | 0.335 | 11.00 | 0.632 | 0.496 |\n\n**Q4. Latent diffusion models using VAEs typically produce high-quality images but introduce issues with encoder-decoder invertibility. Hong et al., propose a correction algorithm for this, suggesting that DCTdiff could bypass such issues entirely**\n\nThank for suggesting the case of encoder-decoder invertibility. We do agree that DCT is a flexible and invertible method of image representation, which differs from the neural network-based image tokenizers. We will include this discussion with paper [1] in the final version of our paper.\n\n[1] Hong, Seongmin, et al. \"Gradient-free Decoder Inversion in Latent Diffusion Models.\" NIPS (2024)", "claims_and_evidence": "The core idea is interesting, which is the primary reason why I recommend this paper, but some claims lack convincing theoretical or experimental support.\n\n**Lack of qualitative evidence**: Unlike related works such as VAR (Tian et al., 2024) and LCM (Luo et al., 2023), this paper does not provide enough qualitative results in the main text. Typically, such studies include extensive uncurated results to prevent cherry-picking concerns. However, the main paper lacks qualitative figures, and the appendix only includes Figs. 9-11, missing key results for FFHQ 512.\n\n**Overclaim on spectral autoregression**: Unlike VAR, which is truly autoregressive in nature, the proposed model is not structurally autoregressive. Diffusion models naturally generate low-frequency components early and high-frequency components later, a well-known phenomenon (Patashnik et al., ICCV 2023). This claim lacks novelty and does not provide practical benefits for DCTdiff.", "methods_and_evaluation_criteria": "The core idea of using DCT coefficients as diffusion model inputs is strong. However, evaluation has weaknesses.\nThe metrics and baselines are limited (only FID and UViT), which may not ensure a fair comparison. UViT was chosen for parameter consistency, but there is no clear reason why other pretrained models were not evaluated. A broader evaluation with models trained on the same datasets (even if #Parameters, GFLOPs, and training steps differ) would improve the study. Additional metrics, such as Inception Score, could be included, but more qualitative results would be the most impactful improvement.", "theoretical_claims": "Theorems 5.1 and 5.2 appear correct but do not significantly contribute to the main claims.\n\nTheorem 5.1: States that diffusion removes high-frequency components first and low-frequency components later. This is well known and not specific to DCTdiff.\n\nTheorem 5.2: Reformulates existing DCT-based upsampling methods in a mathematical form. This principle is already widely used and not novel.\n\nOverall, these theorems do not offer strong contributions.", "experimental_designs_or_analyses": "The main experiments and analyses are generally sound.", "relation_to_broader_scientific_literature": "The paper mainly compares DCTdiff with UViT-based latent diffusion using FID scores. However, the advantages of DCTdiff may extend beyond what is discussed.\n\nFor example, latent diffusion models using VAEs typically produce high-quality images but introduce issues with encoder-decoder invertibility. Hong et al., NeurIPS 2024, propose a correction algorithm for this, suggesting that DCTdiff could bypass such issues entirely.\n\n### refs:\nHong, Seongmin, et al. \"Gradient-free Decoder Inversion in Latent Diffusion Models.\" Advances in Neural Information Processing Systems 37 (2024): 82982-83007.", "essential_references_not_discussed": "The claim that \"diffusion models are spectral autoregression\" is not novel. Unlike true autoregressive models like VAR, diffusion models do not have structural autoregression. They just *tend* to make low-frequency in the early stages, and then *tend* to refine high-frequency in the later stages. The authors should cite Or Patashnik et al. (ICCV 2023) and tone down this claim. \n\n### ref:\nPatashnik, Or, et al. \"Localizing object-level shape variations with text-to-image diffusion models.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.", "other_strengths_and_weaknesses": "The study tackles an important problem, and its ideas could extend to many future works, such as Video diffusion models. This makes it a promising research direction.", "other_comments_or_suggestions": "The authors should provide more qualitative results, and it would be better that they go to the first figure.", "questions_for_authors": "1. Why are qualitative results so limited in the main paper and appendix? Can you provide more uncurated samples?\n\n2. Why were only UViT-based baselines used? Can you compare DCTdiff against more diverse models, even if their training settings differ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a DCT space based diffusion model design, and show that it yields strong results without having to rely on pretrained latent codes. It also provides theoretical analysis, linking it to the comparison between diffusion and autoregressive modeling in frequency space. After extensive discussions, reviewers unanimously agree that this is a strong submission, which the AC agrees with.
Decision: Accept (poster)