Deconstructing Denoising Diffusion Models for Self-Supervised Learning
Authors: Xinlei Chen
Zhuang Liu
Saining Xie
Kaiming He
Venue: ICLR 2025 Poster (2025)
Keywords: denoising diffusion models
denoising autoencoder
self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: deconstructing a modern denoising diffusion model to a classical denoising autoencoder for self-supervised learning

Abstract: In this study, we examine the representation learning abilities of Denoising Diffusion Models (DDM) that were originally purposed for image generation. Our philosophy is to deconstruct a DDM, gradually transforming it into a classical Denoising Autoencoder (DAE). This deconstructive process allows us to explore how various components of modern DDMs influence self-supervised representation learning. We observe that only a very few modern components are critical for learning good representations, while many others are nonessential. Our study ultimately arrives at an approach that is highly simplified and to a large extent resembles a classical DAE. We hope our study will rekindle interest in a family of classical methods within the realm of modern self-supervised learning.

Published as a conference paper at ICLR 2025
DECONSTRUCTING DENOISING DIFFUSION MODELS FOR SELF- SELF-SUPERVISED LEARNING
Xinlei Chen!
Zhuang Liu
Saining Xie
Kaiming Hel
FAIR, Meta
Princeton University
3 New York University
*MIT
ABSTRACT
In this study, we examine the representation learning abilities of Denoising Dif- fusion Models (DDM) that were originally purposed for image generation. Our philosophy is to deconstruct DDM, gradually transforming it into classical Denoising Autoencoder (DAE). This deconstructive process allows us to explore how various components of modern DDMs influence self-supervised representation learning. We observe that only a very few modern components are critical for learning good representations, while many others are nonessential. Our study ultimately arrives at an approach that is highly simplified and to large extent resembles classical DAE. We hope our study will rekindle interest in family of classical methods within the realm of modern self--supervsedd learning.
INTRODUCTION
Denoising is the core in the current trend of generative models in computer vision and other areas. Popularly known as Denoising Diffussio Models (DDM) today, these methods (Sohl- Dickstein et al.
2015; Song & Ermon 2019 Song al.) 2020) Ho et al. 2020 Nichol & Dhariwal] [202] Dhariwal & Nichol 2021) learn Denoising Autoencoder (DAE) Vincent et al. 2008) that removes noise of
multiple levels driven by diffusion process. These methods achieve impressive image generation quality, especially for high-resolution, photo-realistic images (Rombach et al.) [022] Peebles & Xie 2023) in fact, these generation models are so good that they appear to have strong recognition power for understanding the visual content.
While DAE powerhouse of today's generative models, it was originally proposed for learning representations (Vincent et al.) 2008) from data in self-supervised manner. In today's community of representation learning, the arguably most successful variants of DAEs are based on "masking noise" Vincent et al. 2008)) such as predicting missing text in languages (e.g.. BERT (Devlin et al., 2019 or missing patches in images (e.g.. Masked Autoencoder, MAE (He et al. 2022)). ). However, in concept, these masking- masking-based based variants remain significantly different from removing additive (e.g., Gaussian) noise: while the masked tokens explicitly specify unknown vs. known content, no clean signal is available in the task of separating additive noise. Nevertheless, today's DDMs for generation are dominantly based on additive noise. implying that they may learn representations without explicitly marking unknown/known content.
Most recently, there has been an increasing interest (Xiang et al. 2023 Mukhopadhyay et al. 2023
in inspecting the representation learning ability of DDMs. In particular these studies directly lake off-the-helff pre-trained DDMs (Ho et al. 2020 Peebles & Xie 2023 Dhariwal & Nichol 2021). which are originally purposed for generation, and evaluate their representation quality for recognition. They report encouraging results using these
models. However, these pioneering
studies obviously leave open questions: these off-the-shelf models were designed for generation, not recognition; it remains largely unclear whether the representation capability is gained by denoising-dres -divvv process, or diffusion-drive process.
In this work, we take much deeper dive into the direction initialized by these recent explorations (Xiang et al., 2023 Mukhopadhyay et al. 2023). Instead of using an off-the-shelf DDM that is generation-oriemteed
train models that recognition-orientt to deconstruct a DDM, changing it step-by-step into classical DAE. Through this deconstructive process, examine every single aspect (that we can think of) of modern DDM, with the goal of
At the core of our philosophy is

---

Published as a conference paper at ICLR 2025
image
PCA
Jtt PCA
noised
encoder
decoder
deviisal
2o
(a) latent Denoising Autoencoder (I-DAE)
(b) clean, noised and denoised image
Figure 1: (a) The I-DAE we have ultimately reached, after thorough exploration of deconstructing Denoising Diffusion Models (DDM) (Ho et al., 2020), to approach classical Denoising Autoencoders (DAE) (Vincent et al. 2008). Crucial its self sele-super representations. I-DAE adds noise in the latent space. (b) A clean image (left) is projected onto the latent space with PCA, in which noise added. It is then mapped back to pixels via inverse PCA (middle). An autoencoder learned to predict a denoised image (right).
learning representations. This research process gains new understandings what are the critical components for a DAE to learn good representations.
Surprisingly, discover that the main critical component is tokenizer (Rombach et al. creates a low-dimensional latent space. Interestingly, this observation is largely independent of the specifics of the tokenizer-we explore a standard VAE (Kingma & Welling 2013). a patch-wise VAE, a patch-wise AE, and a patch-wise PCA encoder. We discover that it is the low-dimensional latent space, rather than the tokenizer specifics, that enables a DAE to achieve good representations.
2022 that
Thanks to the effectiveness of PCA, our deconstructive trajectory ultimately reaches a simple ar- chitecture that highly similar to the classical DAE (Fig. [ We project the image onto latent space using patch-wise PCA, add noise, and then project it back by inverse PCA. Then we train an autoencoder to predict a denoised image. We call this "latent Denoising Autoencoder" (I-DAE).
Our deconstructive trajectory also reveals many other intriguing properties that lie between DDM and classical DAE. For one example, we discover that even using a single noise level (i.e., not using the noise scheduling of DDM) can achieve a decent result with our I-DAE. The role of using multiple levels of noise is analogous to form of data augmentation, which can be beneficial, but not an enabling factor. With this and other observations, argue that the representation capability of DDM is mainly gained by the denoising-drine
process, not a diffusion
process.
Finally, we compare with previous baselines. To make the comparisons fair and signals clean, adopted minimal enhancements to I-DAE- only ones related to data and model size scaling. On one hand, our results are substantially better than the off-the-shelf counterparts (following the spirit of Xiang et al. (2023); Mukhopadlyay et al. (2023)) ): this is as expected, because these are our starting point of deconstruction. On the other hand, while I-DAE falls short in certain aspects, e.g. linear probing against baseline contrastive learning methods (Chen et al.) 2021)) or model size scaling against masking-based methods (He et al. (022). but outshines others (e.g., fine-tuned transfer with ViT-B). These suggest opportunities and potential for further research along this direction.
2 RELATED WORK
In the history of machine learning and computer vision, the generation of images (or other content) has been closely intertwined with the development of unsupervised or self-supervised learning. Approaches in generation are conceptually forms of un-/selff trained without labeled data, learning capture the underlying distributions of the input data.
learning, where models were
There has been a prevailing belief that the ability of a model to generate high-fidelig data is indicative of potential for learning good representations. Generative Adversarial Networks (GAN) (Good. fellow et al. 2014). for example, have ignited broad interest in adversarial representation learning (Donahue et al. 2017 Donahue & Simonyan, 2019). Variational Autoencoders (VAE) (Kingma & Welling 2013), originally conceptualized generative models for approximating data distributions, have evolved to become a standard learning localized representations ("tokens")) e.g., VQVAE (Oord et al. 2017: Esser et al.) 2021). Image inpainting (Bertalmio et al. (000), essentially form of conditional image generation, has led to a family of modern representation learning methods, including Context Encoder (Pathak et al. 2016) and MAE (He et al. 2022).
2

---

Published as a conference paper at ICLR 2025
Dose
nobse
noised
encodar
desodes
denvied
image
laterr
nolted
cueoder
decoder
denoised
(a) a classical Denoising Autoencoder (DAE)
(b) modern Denoising Diffusion Model (DDM)
Figure 2: A classical DAE and modern DDM. (a) A classical DAE that adds and predicts noise on the image space. (b) State-of-the-art DDMs (e.g., LDM (Rombach et al. 2022). DIT (Peebles & Xie 2023
that operate latent space, where the noise is added and predicted.
Analogously, the outstanding generative performance of DDMs (Sohl-Dickstein et al. 2015: Song & Ermon 2019 Song et al. 2020; Ho et al. [020 Dhariwal & Nichol 2021) has drawn attention for their potential in representation learning. Pioneering studies (Xiang et al. 2023 Mukhopadhyay et al.) 2023) have begun to investigate this direction by evaluating existing pre-trained DDMs. However, we note that while a model's 's generation capability suggests certain level of understanding, it does not necessarily translate to representations useful for downstream tasks. Our study delves deeper into these issues, with deconstructive trajectory different from Hudson al. (2024).
On the other hand, although classical DAES (Vincent et al. 2008 have laid the groundwork for autoencoding- baased representation learning, their success has been mainly confined to scenarios involving masking- baasen corruptions (e.g.. He et al. ) Xie et al. (2022); Fang et al. (2022); Chen al. (2023). To the best of our knowledge, little research dives into classical DAE variants with additive Gaussian noise alone (e.g., not with masking (Wei et al. (2023). We believe the underlying
reason is that a simple DAE bascline (Fig. aaa) performs poorly (e.g.. ~00% in Fig.
BACKGROUND: DENOISING DIFFUSION MODELS
Our deconstructive research starts with a Denoising Diffusion Model (DDM) (Ho al. 2020 Dhariwal & Nichol, 2021). We briefly describe the DDM we use, following Dhariwal & Nichol Peeblles Xie (2023).
A diffusion process starts from clean data point (zn) and sequentially adds noise to it. At specified time step t, the noised data 2 is given by:
21 70 + drC,
(I)
where e~~N co,NO0,1 I) is a noise map sampled from Gaussian distribution, and Te and o, define the scaling factors of the signal and of the noise, respectively. By default, it is set 7? + of (Nichol & Dhariwal 2021 Dhariwal & Nichol 2021).
A DDM is learned to remove the noise, conditioned on the time step t. Unlike the original DAE (Vincent et al. 2008) that predicts a clean input, modern DDMs (Ho et al. 2020 Nichol & Dhariwal 2021 often predict the noise e, minimizing:
- (==))l
(2)
where net( is the network output. The model is trained for multiple noise levels given noise schedule conditioned on l. For generation, a trained model is iteratively applied until it reaches 20-
types of input spaces. One is the image space (Dhariwal & Nichol| 2021) where the raw image zo is directly used as 20. The other option is to build DDMs on latent space produced by a tokenizer, following Rombach al. (2022). See Fig. 2 In this case, pre-trained tokenizer f (which is often another autoencoder, e.g., VQVAE (Oord et al. 2017)) )) is used to map the image Zg into latent 20 = (xo).
DDMs can operate
Diffusion Transformer (DIT). Our study begins with the Diffusion Transformer (DiT) (Peebles & Xie, 2023). We choose this for several reasons: (i) Unlike other UNet-based DDMs (Dhariwal & Nichol 2021 Rombach al. 2022), Transformer-bau
provide fairer comparisons to other sepevvuperv learning baselines driven by Transformers (e.g., Chen et al. (202T); ; He et al.
architectures
(2022)); (ii) DiT has clearer distinction between the encoder and the decoder, while UNet's
3

---

Published as a conference paper at ICLR 2025
acc. (1)
FID (L)
DiT baseline (DDAE Xiang al. (023))
57.5
11.6
remove class-conditining
62.5
30.9
remove VQGAN perceptual loss
58.4
54.3
+ remove VQGAN adversarial loss
59.0
75.6
replace noise schedule
63.4
93.2
Table 1: Reorienting DDM for self-supervised learning, We begin with the DiT (Peebles & Xie 2023) baseline and evaluate its linear probe accuracy (acc.) on ImageNet. This conceptually follows recent studies (Xiang et al. 2023 Mukhopadhyay et al. (023) (more specifically, DDAE (Xiang ct al. 2023)), which evaluate off-the-shelf DDMs with the same protocol. Each line based on modification of the immediately preceding line. The entries in gray, in which class labels are used, are not legitimate results for self-supervised learning. See Sec. 4.1 for descriptions.
encoder and decoder connected by skip connections and may require extra efforts network surgery when evaluating the encoder; (iii) DiT also trains much faster than other UNet-based DDMs (see Peebles & Xie 2023)) while achieving better generation quality.
Specifically, we use Dar-L (DiT-L) (Peebles & Xie 2023 In DiT-L, the encoder and decoder together have the size of VIT-L (Dosovitskiy et al. 2022 (24 blocks). We evaluate the representation quality the encoder, which has 12 blocks, referred to as " "LL" (half large).
Tokenizer. DiT instantiated in Peebles & Xie (2023)
form of Latent Diffusion Models (LDM) (Rombach et al., 2022), which uses VQGAN tokenizer (Esser et al. 2021). Specifi- cally, this VQGAN tokenizer transforms the 256x256x3 input image (heightx width channels) into 32x32xx latent map, with a stride of 8.
Starting baseline. We train DīT-L for 400 epochs on ImageNet (Deng et al. 2009). The baseline results reported in Tab. | (line 1). With DiT-L, we report a linear probe accuracy of 57.5% 5% using its L encoder. The generation quality (Fréchet Inception Distance (Heusel et al. 2017 FID-50K) of this DiT-L model is 11.6. This is the starting point of our destructive trajectory.
Despite differences in implementation details (see Appendix A), our starting point largely follows recent studies (Xiang et al. 2023 Mukhopadhyay et al. 2023 which evaluate DDMs the-- with linear probing.
DECONSTRUCTING DENOISING DIFFUSION MODELS
Our deconstruction trajectory is divided into three stages. We first adapt the generation-focusel settings in DiT' be more oriented toward self-supervised learning (Sec. [. Next, we deconstruct and simplify the tokenizer step by step (Sec. 4.2). Finally, we attempt to reverse many DDM- motivated designs as possible, pushing the models towards classical DAE (Vincent et af.) 2008)
(Sec. 4.3
4.1 REORIENTING DDM FOR SELF-SUPERVISED LEARNING
While a DDM is conceptually a form of DAE, it was originally developed for image generation. Many designs in a DDM are oriented toward the generation task. Some designs are not legitimate for self-supervised learning (e.g., involving class labels); some others are not necessary if visual quality not concerned. We thus first reorient our baseline for self-super learning (Tab. T
Remove class-conditioinng A high-quality DDM is often trained with conditioning class labels. which can largely improve the generation quality. But the usage of class labels is simply not legitimate in self-supervised learning. As the first step, remove class-conditioning in our baseline.
Surprisingly, this change substantially improves the linear probe accuracy from 57.5% to 62.1% 1% (Tab. [ even though generation is greatly hurt as expected (FID from 11.6 to 34.2). We hypothesize that directly conditioning the model on class labels may reduce the model's demands on encoding the label-related information. Removing this conditioning can force the model learn more semanties.
4

---

Published as a conference paper at ICLR 2025
Figure 3: Noise schedule comparisons. The original schedule (Ho et al. 2020 Peebles & Xie 2023). which sets " -II
original kne
3.)
with linear schedule of B, spends many time steps on very noisy images (small 7). Instead, we use a simple schedule that is linear on 7², which provides less noisy images.
20
880 3X0
time step
Deconstruct VQGAN. The VQGAN tokenizer inherited from LDM (Rombach et al. 2022) is trained with multiple loss terms: (i) AE reconstruction loss; (ii) KL-divergence regularization loss (Rombach
et al.
2022 (iii) perceptual loss (Zhang et al. 2018b); and (iv) adversarial loss (Goodfellow et al.
2014
Esser al.
202T) with a discriminator. We ablate the latter two terms next.
The perceptual loss (Zhang et al. 2018b) is based on supervised VGG net (Simonyan & Zisserman
2015) trained for ImageNet classification. Therefore, using the VQGAN trained with this loss is again not legitimate. Instead, train another VQGAN (Rombach al. (0022) where this loss is removed. Using this tokenizer reduces the linear probe accuracy significantly from 62.5% to 58.4% (Tab. D which, however, provides the first legitimate entry thus far. This comparison reveals that a tokenizer trained with the percepiual loss (with class labels) in itself provides semantic representations. From now on, will longer use perceptual loss.
We train the next VQGAN tokenizer that further removes the adversarial loss. It slightly increases the accuracy from 58.4% to 59.0% (Tab. D At this point, tokenizer is essentially VAE, which we move on to deconstruct next. We also note that removing either loss harms generation quality.
Replace noise schedule. In the task of generation, the goal is to progressively turn a noise map into an image. As result, the original noise schedule allocates many time steps on very noisy images (Fig. B.. This is not necessary our model not generation-oriented.
We study a simpler noise schedule for the purpose of self-supervised learning. Specifically, we let oo linearly decay in the range of 1>>2> (Fig. 3). This allows the model to spend more capacity on cleaner images. This change greatly improves the linear probe accuracy from 59 9,00% to 63.4% (Tab. [. suggesting that the original schedule is too focused on noisier regimes. On the other hand, expected, it further hurts generation, leading to FID of 93.2.
Summary, Overall, the results in Tab. [ reveal that self-supervised learning performance is not correlated to generation quality. The representation capability of DDM not necessarily the outcome of its generation capability.
4.2 DECONSTRUCTING THE TOKENIZER
Next, we further deconstruct the VAE tokenizer by making substantial simplifications. We compare the following four variants as tokenizers, each of which a simplified version of the preceding one:
Convolutional VAE. Our deconstruction thus far leads us to a VAE tokenizer. As common practice (Kingma & Welling, 2013 Rombach et al. 2022), the encoder ((-) and decoder g(-) of this VAE are deep convolutional (conv) neural networks (LeCun et al. 1989). This convolutional VAE minimizes the following loss function:
||
Here, x the full input image. The first term is the reconstruction loss, and the second term the K-L divergence (Bishop & Nasrabadi 2006 Esser et al. 2021) between the latent distribution of fx) and unit Gaussian distribution.
Patch-wise VAE. Next we simplify so that the VAE encoder and decoder are both linear projections, and the VAB input is patch. The training process of this patch-wise VAE minimizes this loss:
|| -
T Vx|| + KIL [Vz[W].
The KI. form in Rombach al. 2022) does not perform explicit vector quantization (VQ).
5

---

Published as a conference paper at ICLR 2025
Table 2: Linear probe accuracy vs. latent dimension. With a DiT model, we study four variants of to- kenizers for computing the latent space. We vary the dimensional- ity d (per token) of the latent space. The table is visualized by the plot above. All four variants of the tokenizers exhibit similar trends, despite their differences in architec- tures and loss functions. The 63.4% entry of "conv. VAE" is the same entry as the last line in Tab. [
canv. VAE
patichwise VAE
50
wutch-wise AE
putch-wise PCA
latent dim per token (log-acalle)
lntent dim..
8
16
32
64
128
cony. VAE (haseline)
54.5
63.4
57.0
patch-wise VAE
58.3
64,9
64.8
56.8
patch-wise AE
59.9
64.7
64.6
59,9
patch wise PCA
$6.0
63.4
65.1
60.0
53.9
Here x denotes patch flattened into D-dimensional vector. Both U and V are dx D matrices, where d is the latent space dimension. Patch size is 16x 16, following Dosovitskiy et al. (2011)
Patch-wise AE. We further simplify VAE by dropping regularization:
| - Vx
As such, this tokenizer is essentially an AE on patches, with a linear encoder and decoder.
Patch-wise PCA. Finally, we consider simpler variant which performs PCA on the patch space. It is easy to show that PCA is equivalent to a special case of AE:
| VTVx
Here V satisfies VVT=I (dxd identity matrix). The PCA bases can be simply computed by eigen-decompositton large set of randomly sampled patches without gradient-based training.
Tab. 2 summarizes the linear probe accuracy of DiT using these four variants of tokenizers. We show
the results w.r.t. the latent dimension "per token" .2
We draw the following observations.
Latent dimension of the tokenizer is crucial for DDM to work well in self-supervised learning.
As shown in Tab. 2 all four variants of tokenizers exhibit similar trends, despite their differences in architectures and Toss functions Interestingly, the optimal dimension is relatively low (d is 16 or 32), even though the full dimension per patch is much higher (16x 16x3=768).
Surprisingly, the convolutional VAE tokenizer is neither necessary nor favorable; instead, all patch- based tokenizers, in which each patch is encoded independently, perform similarly with each other and consistently outperform conv. VAE. In addition, the KL regularization term is unnecessary, as both the AE and PCA variants work well.
To our further surprise, even the PCA tokenizer works well. Unlike the VAE or AE counterparts, PCA does not require gradicnt- based training. With pre-computed PCA bases, the application of the PCA tokenizer is close "image pre-processing" step, rather than "network architecture". Its effectiveness largely helps us push the modern DDM towards a classical DAE, as we will show next.
High-resoluttin, pixel- DDMs are inferior for self- supervised learning.
Before we move we report an extra ablation that is consistent with the aforementioned observation. Specifically, we consider "naïve tokenizer" that performs identity mapping on patches extracted from resized images. In this case, a "token" is the flatten vector consisting all pixels of a patch.
In Fig. we show the results of this ppixel-based tokenizer, operated on an image size of 256, 128. 64, and 32, respectively with patch size of 16, 8, 4, 2. The "latent" dimensions of these tokenized spaces are 768. 192, 48, and 12 per token. In all case, the sequence length of the Transformer is kept unchanged (256).
For patch-wise VAE/AE/PCA (patch stride is 16). treat each patch token, the latent dimension is simply d for each patch. For the convolutional VAE that has stride of 8, DřT (Peebles Xie 2023 treats cach x 2 patch on the latent space as a "token": as a result, its latent dimension "per token" should be multiplied by 4 for calibration.
For the pasch-wise tokenizers, we find they also learn different filters. See Appendix B for visualizations.

---

Published as a conference paper at ICLR 2025
Figure 4: Linear probe accuracies on pixel- based tokenizer, operated on an image size of 256, 128, 64, and 32, respectively with patch size of 16, 8, 4, 2. The "latent" dimensions of these tokenized spaces are 768, 192, 48, and 12 per token. Similar to others, this pixel-based tokenizer exhibits a similar trend: a relatively small dimension of the latent space is optimal.
63.4
65.1
60.0
30
$6.0
5K.5
$3.5
44.7
pulchwwise PCA
pioci-based
23.6
20
12
192
768
lateral dim per token (log-scale)
Interestingly, this pixel-based tokenizer exhibits similar trend with previous tokenizers, although the optimal dimension is shifted. In particular, the best dimension is d=48, which corresponds to an image size of 64 with a patch size of 4. With image size 256 and patch size 16 (d=768), the accuracy drops dramatically to 23.6%.
These comparisons show that the tokenizer and the resulting latent space are crucial for DDM/DAE to work competitively in the sulf-super learning scenario. In particular, applying classical DAE with additive Gaussian noise on the image space leads to poor results.
4.3 TOWARD CLASSICAL DENOISING AUTOENCODERS
Next, we go with our deconstruction trajectory and aim get as close as possible to the classical DAE (Vincent et al., 2008). We attempt to remove every single aspect that still remains different between our current PCs-b DDM and the classical DAE practice. Via this deconstructive process. we gain better understandings on how every modern design may influence the classical DAE. Tab. lists the results.
Predict clean data (rather than noise). While modern DDMs commonly predict the noise (see Eq. 2), the classical DAE predicts the clean data instead. We examine this difference by minimizing:
>e n=)(2))
Here 2g is the clean data (in the latent space), and net( is the network prediction. Xe is a t- dependent loss weight, introduced to balance the contribution of different noise levels (Salimans & Ho 2022). It is suggested to set de = /?//a? as per [Salimans & Ho (2022). We find that setting d "h works better in our scenario. Intuitively, it simply puts more weight the loss terms of the cleaner data (larger 7x.).
With the change of predicting clean data, the linear probe accuracy degrades from 65.1% to 62.4% (Tab. 3 This suggests that the choice of the prediction target influences the representation quality.
Even though we suffer from degradation in this step, will stick to this modification from now on, as our goal is to get as close as possible to a classical DAE
Remove input scaling, In modern DDMs, the input scaled by a factor of Y (Eq. (.). This is not common in a classical DAE and next we remove it, i.e., set = 1. As "/e is fixed, we need to define a noise schedule directly on a, We simply set as linear schedule on a, from 0 to v2, and empirically set the weight in Eq. B as X 1/(1 + of). This again puts more emphasis on cleaner data.
After fixing = 1, we achieve a decent accuracy of 63.6% (Tab. 3). which compares favorably with the varying 7t counterpart's 's 62.4%.
Operate on the image space with inverse PCA. So far, for all entries we have explored (except Fig. 4.. the model operates on the latent space produced by tokenizer (Fig. 2 (b)). Ideally, we hope our DAE can work directly on the image space while still having good accuracy, With the usage of PCA, we can achieve this goal by inverse PCA.
The idea is illustrated in Fig. [ Specially, we project the input image into the latent space by the PCA bases (i.e., V), add noise in the latent space, and project the noisy latent back to the image space by the inverse PCA bases (VT"). Fig. lb (middle) shows example image with noise added in the latent space. With this noisy image as the input to the network, we can apply a standard ViT network (Dosovitskiy et al. 2021) that directly operate on images, if there is no tokenizer.
We have revisited undoing this change in our final entry, in which we have observed this degradation.
7

---

Published as a conference paper at ICLR 2025
patch-wise PCA baseline
65.1
predict clean data (rather than noise)
remove input scaling (fix Y 1)
63.6
+ operate on image input with inv. PCA
63.6
operate image output with inv. PCA
63.9
+ predict original image
64.5
Table 3: Moving toward classical DAE from our patch-wise PCA tokenizer. Each line adds modification to the immediately preceding line. See Sec. 4.3 for details.
Applying this modification on the input side (while still predicting the output on the latent space) has 63.6% accuracy (Tab. 3 Further applying it to the output side (i.e., predicting the output on the image space with inverse PCA) has 63.9% accuracy. Both results show that operating on the image space with inverse PCA can achieve similar results as operating the latent space.
Predict original image. While inverse PCA can produce a prediction target in the image space, this target the original image. This is because PCA lossy encoder for any reduced dimension d. In contrast, it is a more natural solution to predict the original image directly.
When we let the network predict the original image. the "noise" includes two parts: (i) the additive Gaussian noise, whose intrinsic dimension is d, and (ii) the PCA reconstruction error, whose intrinsic dimension is D (D 768). We weight the loss of both parts differently.
Formally, with the clean original image and network prediction net( we can compute the residue projected onto the full PCA space: r - V(ro net((;))) Here V is the D-by-D matrix representing the full PCA bases. Then minimize A, Di=1 wir?, where i denotes the i-th dimension of the vector r. The per-dimension weight w; for i S d, and 0.1 I for d < i < D. Intuitively, w; down-weights the loss of the PCA reconstruction error. With this formulation, predicting the original image achieves 64.5% linear probe accuracy (Tab. 3h.
Conceptually, this variant is very simple: its input is noisy image whose noise is added the PCA space, its prediction is the original clean image (Fig.
DIT baseline
57.5
Single noise level. Lastly, out of curiosity, we further study variant with single-level noise. We note that multi-level noise, given by noise scheduling, is a property motived by the diffusion process in DDMs; it is conceptually unnecessary in a classical DAE.
remove cls cand-
62.5
remove VQGAN less
38.4
VQGAN
59.0
replace noise
conv. VAE to patch-wise VAE
to patch-wise AE
64.7
fix the noise level a as a constant (v 1/3). Using this single-level noise achieves decent accuracy of 61.5%. a 3% degradation vs. the multi-level noise counterpart (64.5%). This mild degeneration sug- gests multiple levels of noise is analogous to a form of data augmentation in DAE: it is beneficial, but not an enabling factor. This also implies that the rep- resentation capability of DDM is mainly gained by the serbissiinn-iiieen process, not a diffusion- process.
to patch-wise PCA predict clean data remove input scaling use image input
65.1
62.4
63.6
use image output- predict original Image
63.9
64.5
56 58
66
linear probing accuracy
Figure 5: The overall deconstructive trajec- tory from a modern DDM I-DAF, summa- rizing Tab. [ Tab. 2 and Tab. [ Fach line is based on a modification of the immediately preceding line.
As multi-level noise is useful and conceptually sim- ple, we keep for our final comparisons.
4.4 SUMMARY
In sum, have deconstructed modern DDM and pushed it towards classical DAE (Fig. 5 We undo many of the modern designs and retain only two designs inherited from modern DDMs: (i) low-dimessional latent space in which noise is added; and (ii) multi-level noise.
We use the entry at the end of Tab, 3 as our final DAE instantiation (illustrated in Fig. T We refer to this method "latent Denoising Autoencoder". I-DAE.
8

---

Published as a conference paper at ICLR 2025
Figure 6: Visualizing pixel noise vs. latent noise. (a) clean image, 256x256. (b) noise added to the image space. (c) noise added to the PCA latent space, visualized by back projection to the image space using inverse PCA. a= /1/3 in both cases.
clean
(b) pixel
(c) latent
ANALYSES AND COMPARISONS
Visualizing latent noise. Conceptually, /-DAE a form of DAE that learns to remove noise added to the latent space. Thanks the simplicity of PCA, PCA. Fig. 6 compares the noise added to pixels vs. to the latent. Unlike the pixel noise, the latent noise is largely independent of the resolution of the image. With patch-wise PCA as the tokenizer, the pattern of the latent noise is mainly determined by the patch size. Intuitively, we may think of it as using patches, rather than pixels, resolve the image. This behavior resembles MAE (He et al. 2022). which masks out patches instead of individual pixels.
easily visualize the latent noise by inverse
Denoising results. Fig. 7 shows examples of denoising results based on /-DAE, Our method produces reasonable predictions despite the heavy noise. We note that this is less of surprise, because neural network-based image restoration (Burger et al. 2012 Dong et al. 2014) has been intensively studied.
Nevertheless, the visualization may help better understand how I-DAE can learn good represen- tations. The heavy noise added to the latent space creates a challenging pretext task for the model to solve. It is nontrivial (even for human beings) predict the content based one few noisy patches locally; the model is forced to learn higher-level, more holistic semantics to make sense of the underlying objects and scenes.
Beyond qualitative visualizations, we perform three more quantitative analyses on I-DAE:
Data augmentation. All models we present thus far have no data augmentation: only the center crops of images are used, with no random resizing or color jittering, following Dhariwal & Nichol 202 Peebles Xie (2023). We further explore mild data augmentation (random resized crop) our final I-DAE, which has slight improvement (Tab, 4ap. This suggests that the representation learning ability of I-DAE is largely independent of its reliance on data augmentation. A similar behavior was observed in MAE (He et al. 2022). which sharply differs from the behavior of contrastive learning methods (e.g., Chen et al. (2020)).
Training epochs. All experiments thus far are based on 400-epoch training. Following MAE (He et al. 2022), we also study training for 800 and 1600 epochs (Tab. 4b). As reference, MAE has
18:2
18:32
Figure 7: Denoising results of I-DAF, evaluated on ImageNet validation images. This denoising problem, serving pretext task, encourages the network to learn meaningful representations in supervied manner. For each case, we show: (left) clean image; (middle) noisy image as the input to the network, where the noise is added to the latent space; (right) denoised output.

---

Published as a conference paper at ICLR 2025
aug.
center crop
random crop
epochs
400
800
1600
encoder
VIT-B
VIT- 2
VIT-L
acc.
64.5
65.0
acc.
65.0
67.5
69.6
acc.
60.3
65.0
70.9
(a) augmentation
(b) training epochs
(c) model size
Table Analysis on I-DAE. Specifically, we show: I-DAE requires minimal data augmentation in
(a) and scales reasonably well with training epochs in (b) and model size in (c).
acc.
VIT-B
VIT-L
acc.
VIT-B
VIT-L
Aphex
ViT-B
VIT-L
MoCo v3
76.7
77.6
MoCo v3
83.2
84.1
Supervised
47.6
49.6
MAE
68.0
75.8
MAE
83.6
85.9
MAE
51.2
54.6
I-DAE 66.6
75.0
I-DAE
83.7
84.7
I-DAE
51.6
54.4
(a) linear probing
(b) end-to-end fine-tuning
(c) object detection transfer
Table System-level comparisons with previous baselines on standard Transformer encoders. For ImageNet classification, we compare MoCo v3 (Chen et al. (021) and MAE (He et al. (202) in two protocols: (a) linear probing and (b) End-to-end fine-tuning, For object detection transfer (c). we use ViTDet (Li al., 2022) and report AP on COCO (Lin et al. (014). Different methods show different strengths: 1-DAE shines in fine-tuned transfer with ViT-B; and even for linear probing, the accuracy
gap is drastically closed (from 20% in Fig. 4 to 70+% in Tab. 5ap.
significant gain (4%) from 400 800 epochs, and MoCo v3 (Chen al. 2021) has nearly no gain (0.2%) from 300 to 600 epochs. Our 1-DAE falls in-between (+4.6% from 400 to 1600 epochs).
Model size, So far, our all models are based on the DiT-L variant (Peebles & Xie 2023). whose encoder and decoder are both "ViT- A (half depth of VIT-L). We further train models of different sizes, whose encoder VIT-B or ViT-L (decoder always has the same size). We observe a good scaling behavior w.r.t model size (Tab, 4ch: from ViT-B to ViT-L it has large, 10.6% gain. A similar scaling behavior also observed in MAE (He et al. 2022). which gains 7.8% from ViT-B to ViT-L.
Comparison with previous baselines on ImageNet. We compare I-DAE to previous baselines in Tab. 55 and Tab. 5b We consider MoCo v3 (Chen et al., 2021), which belongs to contrastive learning methods, and MAE (He et al., 2022). which belongs to masking- based methods. Besides linear probing, we also report end-to-end fine-tuning results, with details listed in Appendix
While J-DAE falls short in linear probe accuracy especially against MoCo v3, it behaves similarly to MAE, and even achieves the best overall fine-tuning results with ViT-B. We note that here the training settings are made as fair as possible between MAE and I-DAE: both are trained for 1600 epochs and with random crop augmentation. With our deconstructive efforts, the accuracy gap between MAE and a DAE-driven method is drastically closed (from 200 in Fig. Aoo 70+ % in Tab. 5a).
Comparison object detection transfer. Finally, transfer the representations learned with I-DAE for object detection and segmentation. We use the ViTDet framework (Li et al. 2022) and evaluate on the COCO benchmark (Lin et al., (2014) against MAE and supervised pre-training. Again, we find I-DAE behaves similarly to MAE, and significantly outperforms supervised pre-training. With MAE, I-DAE performs well with ViT-B, while slightly falls short when the model size gets larger. More details and segmentation results are found in Appendix A and Appendix
6 CONCLUSION
We have reported that I-DAE, which largely resembles the classical DAE, can perform competitively in self-supervised learning. The critical component is a now-dimensional latent space on which noise is added. We hope our discovery will reignite interest in denoising- based methods in the context of today's 's self-supervised learning research.
Limitations. Our work studies DDM and DAE empirically, with limited theoretical analysis; while we have locally shuffled the deconstruction order and verified our major claims, it's impossible to experiment all the orders; and while (-DAE outshines prior baselines in certain settings, it falls short in others. Nonetheless, we hope our study provides clean and valuable signals to the community, which can help guide future research along this direction.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. We thank Pascal Vincent, Mike Rabbat, and Ross Girshick for their discussion and feedback.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXivv1607.06450, 2016.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In SIGGRAPH, 2000.
Christopher Bishop and Nasser Nasrabadi. Pattern recognition and machine learning. Springer, 2006.
Harold Christopher Burger, Christian Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with BM3D? In CVPR, 2012.
Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit- Yan Yeung. Mixed autoencoder for self-supervised visual representation learning. In CVPR, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised Vision Transformers. In ICCV, 2021.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurlPS, 2021.
Jeff Donahue and Karen Simonyan. 10Xi:01977.0244.. 2019.
Large
scale
adversarial representation learning.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image nuper-resolution. In ECCV, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Yuxin Fang, Dong, Hangho Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. arXiv:2202.03388, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv: /70..0767007770000 2017.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local Nash equilibrium. In NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In ECCV, 2016.
Drew A Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. In CVPR, 2024.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In ICML, 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2013.
Yann LeCun, Bernhard Boser. John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, 1989.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. ECCV, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion models beat GANs on image classification. WrXiv:2307.08702 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurlPS, 2017.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
William Peebles and Saining Xie. Scalable diffusion models with Transformers. In ICCV, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Jascha Sohl-Dickstell Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 2019.
Yang Song, Jascha Sohl-Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv:2011. 13456, 2020.
12

---

Published as a conference paper at ICLR 2025
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. ICML, 2008
Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models masked autoencoders. In ICCV, 2023.
Weilai Xiang, Hongyu Yang. Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In ICCV, 2023.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling. In CVPR, 2022.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features, In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018b.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
The DiT architecture similar to the original ViT (Dosovitskiy et al. (021), with extra modifications made for conditioning. Each Transformer block accepts an embedding network two-layer MLP) conditioned on the time step f. The output of this embedding network determines the scale and bias parameters of LayerNorm (Ba et al., 2016), referred to as adaLN (Peebles & Xie 2023). Slightly different from Peebles & Xie (2023), we set the hidden dimension of this MLP as 1/4 of its original dimension, which helps reduce model sizes and save memory, at no accuracy cost.
DîT architecture. We follow the DIT architecture design (Peebles & Xie 2023
Training. The original DiTs Peebles & Xie 2023) are trained with a batch size of 256. To speed up our exploration, we increase the batch size to 2048. We perform linear learning rate warm up (Goyal et al. 2017) for 100 epochs and then decay it following half-cycle cosine schedule. We use a base learning rate blr le-4 (Peebles & Xie 2023) by default, and set the actual Ir following the 256. No weight decay is used (Peebles & Xie 2023). We train for 400 epochs by default. On a 256-core TPU-v3 pod, training DiT-L takes 12 hours.
linear scaling rule (Goyal al. 2017): blr batch size /
Linear probing. Our linear probing implementation follows the practice of MAE (He et al. 2022). We use clean, x56 2i6-sized images for linear probing training and evaluation. The ViT output feature map is globally pooled by average pooling. It is then processed by parameter- free Batch- Norm (Toffe Szegedy 2015) layer and a linear classifier layer, following He et al. (2022). The training batch size 16384, learning rate is 6.4x10 (cosine decay schedule), weight decay is O, and training length 90 epochs. Randomly resized crop and flipping are used during training and a single center crop is used for testing. Top-1 accuracy is reported.
While the model is conditioned on t in self-supervised pre-training, conditioning is not needed in transfer learning (e.g., linear probing). We fix the time step & value in linear probing training and evaluation. The influence of different / values (out of 1000 time steps) is shown as follows:
fixed t
0
10
20
40
80
w/ clean input
64.1
64.5
64.1
63.3
62.2
w/ noisy input
64.2
65.0
65.0
65.0
64.5
We note that the t value determines: (i) the model weights, which are conditioned on f, and (ii) the noise added in transfer learning, using the same level Both are shown in this table. We use 10 and clean input in all our experiments, except Tab. 5 where we use the optimal setting.
Fixing also means that the t-dependent MLP layers, which are used for conditioning, are not exposed in transfer learning. because they can be merged given the fixed t. As such, our model has the number parameters just similar to the standard ViT (Dosovitskiy et al., 2021). as reported in Tab. 5
The DIT-L (Peebles & Xie 2023) has 24 blocks where the first blocks are referred to as "encoder" (hence ViT- and the others the "decoder". This separation of the encoder and decoder artificial. In the following table, we show the linear probing results using different numbers of blocks in the encoder, using the same pre-tr model:
enc. blocks acc.
10
11
13
58.5
62.0
64.1
64.5
63.6
61.9
59.7
The optimal accuracy is achieved when the encoder and decoder have the same depth. This behavior different from MAE's (He et al. 2022), whose encoder and decoder are asymmetric.
End-to-end fine-tuning. We closely followed MAE'S protocol (He et al. 2022). We use clean, 256 256-sized images as the inputs to the encoder. Globally average pooled outputs are used as features for classification. The training batch size is 1024, initial learning rate is 4x 10-3 weight decay is 0.05, drop path (Huang et al. 2016 is 0.1, and training length is 100 epochs. We use a layer- wise learning rate decay of 0.85 (B) 0.65 (L). MixUp (Zhang et al. 2018a) (0.8), CutMix (Yun) jet al. (219)) (1.0), RandAug (Cubuk et al. 2020 (9, 0.5). and exponential moving average (0.9999)
used, similar to He al. (2022).
14

---

Published as a conference paper at ICLR 2025
Object detection and segmentation transfer. We adopted the COCO recipe from ViTDet (Li et al. 2022), except: window attention size is set to 8x8, as the pre- pre-training size for I-DAE is 256x256; layer-wise learning rate decay of 0.95; drop path of 0.1 (B) or 0.2 (L).
B VISUALIZATIONS OF THE LATENT SPACE
Thanks to the simplicity of using patches, for the three patch-wiss tokenizers (VAE/AE/PCA), can visualize their filters in the patch space. We show below in Fig.
(a) patch-wise VAE
(h) patch-wise AE
(c) patch-wise PCA
Figure 8: Visualization of the patch-wiss tokenizer. Each filter corresponds to row of the linear projection matrix V (d)D), reshaped to 16x 16x for visualization. Here d-16.
C OBJECT SEGMENTATION RESULTS
Besides Average Precision (AP) for bounding box detection, the evaluation of ViTDet (Li et al. 2022) also includes results object segmentation detection. We list them in Tab. 6 below.
VIT-B
VIT-L
method
Aphox
Ap
Aph
Apmask
Supervised
47.6
42.4
49.6
43.8
MAE
51.2
45.5
54.6
48.6
I-DAE
51.6
45.8
54.4
48.2
Table 6: Object detection and segmentation results on COCO using ViTDet (Li et al. 2022). I-DAE performs similarly MAE and outperforms supervised pre-training.
15


Reviews:
[{"review_id": "wKnaFF4nV3", "reviewer": "Reviewer_Lfx6", "summary": "In this paper, the author studied the representation learning abilities of denoising-autoencoder-based diffusion models (DDMs). Throughout extensive ablation studies, they explored how various components of modern DDMs influence self-supervised representation learning. At the core of their philosophy is to deconstruct a DDM, changing it step-by-step into a classical DAE. This research process demonstrates that the main critical component for a DAE to learn good representations is a tokenizer that creates a low-dimensional latent space.", "questions": "**1.** The authors mainly focused on investigating the representation learning abilities of DiTs. Are there similar observations on U-Net-based diffusion models? \n\n**2.** Based on the experimental results, can we conclude that adding noise primarily impacts the generation capabilities of diffusion models rather than their representation learning ability? Are there any insights for this observation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1.** In general, this paper contributes significantly to the intersections of diffusion models and representation learning.  The findings open up new avenues for future research in leveraging diffusion processes to enhance representation quality across diverse applications\n\n**2.** The authors conducted extensive experiments to support their results. The paper is well-written and easy to follow.", "weaknesses": "The reviewer has the following major concerns about this paper: \n\n**1.** It is not comprehensive to study the representation ability of diffusion models only by considering the classification of downstream tasks. The authors should provide more experiments on other tasks to support their conclusions. \n\n**2.** Although the observations of this work are really new and interesting, the authors seem to not fully discuss the implications of these findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zKIA8vHldh", "reviewer": "Reviewer_SR52", "summary": "This paper studies the representation learning ability of denoising diffusion models. Through a set of ablation studies that deconstruct a denoising diffusion model into a classical denoising autoencoder (DAE), the authors observe that only a very few modern components (such as a low-dimensional latent space) are critical for learning good representations. Experiments also show that a latent DAE, which largely resembles the classical DAE, can perform competitively in self-supervised learning.", "questions": "1. What are the FID scores for other modifications beyond Table 1, such as operating in the image space with PCA?\n\n2. Does better classification accuracy always lead to worse FID scores? In other words, are the tasks of generation and representation learning (for recognition) fundamentally contradictory to each other? Or could we unify them?\n\n3. Why is the middle layer of the transformer chosen as the representation for linear probing?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Studying the representation learning ability of denoising diffusion models is important and could provide useful insights for unifying models for generation and discrimination.\n\nOverall, the paper is very well written. The experiments are solid, and the message is clear. The observation that a low-dimensional latent space is a critical component for representation learning is useful.", "weaknesses": "1. While the study begins with denoising diffusion models, it ultimately leads to models that demonstrate strong representations for classification but not for generation. The FID is reported only in Table 1, which reveals a significant contradiction between classification accuracy and FID. \n\n3. For the goal of representation learning for classification without fine-tuning, the obtained latent DAE achieves slightly worse performance than MAE and contrastive learning.\n\n4. The representation is extracted from the middle layer of the transformer for linear probing. Previous studies have found that the middle layer may not provide the best representation of a diffusion model for classification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hAZTGtwvLq", "reviewer": "Reviewer_Lm16", "summary": "The paper studies the representation ability of generative denoising diffusion models (DDM). It specifically aims to identify crucial components for DDMs' representation ability during the process of removing modern components in DDMs until it becomes a simpler model very similar to classic Denoising Autoencoder (DAE). Notably, DAE is originally proposed for representation learning. The paper is highly empirical, conducting various experiments on different components, such as different loss terms, different tokenizers, class-conditioning, noise schedule, whether to predict clean data, etc. It can remove many components designed for generation, and show high-level representation abilities are not strictly related to generation ability.", "questions": "1. L 247: The paper claims \"self-supervised learning performance is not correlated to generation quality\". However, the selected tasks such as linear probes only consider coarse representations that are useful for high-level tasks. What about more fined representations such as segmentation, and positions of a specific object, etc, would it be correlated to generation quality?\n2. L 369: why the DAE is expected to work directly on the image space, could you please explain the importance of working on the image space?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) The paper has identified a thorough set of different components between classical DAE and modern diffusion models.\n\n(2) The paper sheds light on a potential framework for understanding the representation ability of diffusion models - DAE.\n\n(3) The paper is the first work trying to identify key representation components from generative models and could inspire future works.\n\n(4) The paper has some interesting findings, such as the low-rank tokenizer is important, and the high-level representation ability may not correlate with the generation ability.", "weaknesses": "(1) The paper only presents empirical findings, with no theoretical analysis or practical applications.\n\n(2) The complex possible choices of components make the experiment order not strictly natural and logical. \n\n(3) Missing experiments on some possible choices of components make the conclusions of the paper not that strong, for example, it's hard to conclude whether predicting clean images is more helpful than predicting noise for representation learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u1bRxYgtBk", "reviewer": "Reviewer_NrMn", "summary": "In this paper, the authors perform an extensive ablation study on modern Denoising Diffusion Models (DDMs) to identify the critical components for effective representation learning, comparing these models to traditional denoising autoencoders (DAEs). The authors begin by comparing DDMs and DAEs on their design and purpose, indicating that generation (DDM objective) is not directly connected with good representations (DAE objective), and suggesting a possible trade-off between these them.\n\nThen the authors test various modifications to make DDMs more similar to DAEs while preserving high-quality representations. Notably, they find that the tokenizer is beneficial mainly for its dimensionality reduction role and that a simplified approach, like patch-wise PCA, can serve this function without compromising performance. They also identify less critical components: noise scheduling (analogous to data augmentation), class conditioning (which may lessen the model’s need to capture class-specific, fine-grained semantics), and predicting noise versus the original image.\n\nBased on these findings, the authors propose **l-DAE**, a DAE that operates in the latent domain. They validate the effectiveness of l-DAE through experiments on classification and object detection tasks, showing its competitive performance.", "questions": "1. Could you explain on why noise scheduling can be considered a form of data augmentation? Is there any ablation study showing that the effects of noise scheduling and data augmentation are comparable?\n2. Regarding the section on predicting the original image (Lines 392–406), shouldn't the matrix $V$ be of size $D \\times d$? Also, if the weights $w_i$ are all 1, the loss would again become the reconstruction loss on the latent space. This suggests that the relative scale of these weights is important. Do you have any ablation studies on this aspect?\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors aim to bridge modern self-supervised learning with classical methods by deconstructing diffusion denoising models (DDMs) and propose a simple but effective representation learner similar to traditional denoising autoencoders (DAEs).\n\n2. Through this novel deconstructing process, the authors provide key insights. For instance, they highlight the importance of the low-dimensional latent space into which the tokenizer maps the patches, as it plays a crucial role in learning robust representations. Additionally, they demonstrate that adding noise in the latent space is more effective than in the pixel space, a finding that invites further exploration of latent space structures.", "weaknesses": "1. In the paper, the authors make numerous claims but test them only on a specific task (ImageNet classification) and model (DiT-L). Furthermore, the deconstructing process faces a limitation: the components may be correlated, making a sequential analysis potentially inadequate.\n2. Some of the claims may be overlooked. For instance, the statement that 'multiple levels of noise is analogous to a form of data augmentation' (lines 416–418) may be overly simplified. Prior research (https://arxiv.org/abs/1406.3269) has shown that combining representations at different noise levels can lead to significant improvements.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)