Deconstructing Denoising Diffusion Models for Self-Supervised Learning

Authors: Xinlei Chen, Zhuang Liu, Saining Xie, Kaiming He
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: denoising diffusion models, denoising autoencoder, self-supervised learning

Abstract:
In this study, we examine the representation learning abilities of Denoising Diffusion Models (DDM) that were originally purposed for image generation. Our philosophy is to deconstruct a DDM, gradually transforming it into a classical Denoising Autoencoder (DAE). This deconstructive process allows us to explore how various components of modern DDMs influence self-supervised representation learning. We observe that only a very few modern components are critical for learning good representations, while many others are nonessential. Our study ultimately arrives at an approach that is highly simplified and to a large extent resembles a classical DAE. We hope our study will rekindle interest in a family of classical methods within the realm of modern self-supervised learning.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730593721054

thread_type:
official_review

<thread_object>
creator: reviewer_nrmn

<content>
summary:
In this paper, the authors perform an extensive ablation study on modern Denoising Diffusion Models (DDMs) to identify the critical components for effective representation learning, comparing these models to traditional denoising autoencoders (DAEs). The authors begin by comparing DDMs and DAEs on their design and purpose, indicating that generation (DDM objective) is not directly connected with good representations (DAE objective), and suggesting a possible trade-off between these them.

Then the authors test various modifications to make DDMs more similar to DAEs while preserving high-quality representations. Notably, they find that the tokenizer is beneficial mainly for its dimensionality reduction role and that a simplified approach, like patch-wise PCA, can serve this function without compromising performance. They also identify less critical components: noise scheduling (analogous to data augmentation), class conditioning (which may lessen the model’s need to capture class-specific, fine-grained semantics), and predicting noise versus the original image.

Based on these findings, the authors propose **l-DAE**, a DAE that operates in the latent domain. They validate the effectiveness of l-DAE through experiments on classification and object detection tasks, showing its competitive performance.

strengths:
1. The authors aim to bridge modern self-supervised learning with classical methods by deconstructing diffusion denoising models (DDMs) and propose a simple but effective representation learner similar to traditional denoising autoencoders (DAEs).

2. Through this novel deconstructing process, the authors provide key insights. For instance, they highlight the importance of the low-dimensional latent space into which the tokenizer maps the patches, as it plays a crucial role in learning robust representations. Additionally, they demonstrate that adding noise in the latent space is more effective than in the pixel space, a finding that invites further exploration of latent space structures.

weaknesses:
1. In the paper, the authors make numerous claims but test them only on a specific task (ImageNet classification) and model (DiT-L). Furthermore, the deconstructing process faces a limitation: the components may be correlated, making a sequential analysis potentially inadequate.
2. Some of the claims may be overlooked. For instance, the statement that 'multiple levels of noise is analogous to a form of data augmentation' (lines 416–418) may be overly simplified. Prior research (https://arxiv.org/abs/1406.3269) has shown that combining representations at different noise levels can lead to significant improvements."

questions:
1. Could you explain on why noise scheduling can be considered a form of data augmentation? Is there any ablation study showing that the effects of noise scheduling and data augmentation are comparable?
2. Regarding the section on predicting the original image (Lines 392–406), shouldn't the matrix $V$ be of size $D \times d$? Also, if the weights $w_i$ are all 1, the loss would again become the reconstruction loss on the latent space. This suggests that the relative scale of these weights is important. Do you have any ablation studies on this aspect?"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615018973

thread_type:
official_review

<thread_object>
creator: reviewer_lm16

<content>
summary:
The paper studies the representation ability of generative denoising diffusion models (DDM). It specifically aims to identify crucial components for DDMs' representation ability during the process of removing modern components in DDMs until it becomes a simpler model very similar to classic Denoising Autoencoder (DAE). Notably, DAE is originally proposed for representation learning. The paper is highly empirical, conducting various experiments on different components, such as different loss terms, different tokenizers, class-conditioning, noise schedule, whether to predict clean data, etc. It can remove many components designed for generation, and show high-level representation abilities are not strictly related to generation ability.

strengths:
(1) The paper has identified a thorough set of different components between classical DAE and modern diffusion models.

(2) The paper sheds light on a potential framework for understanding the representation ability of diffusion models - DAE.

(3) The paper is the first work trying to identify key representation components from generative models and could inspire future works.

(4) The paper has some interesting findings, such as the low-rank tokenizer is important, and the high-level representation ability may not correlate with the generation ability.

weaknesses:
(1) The paper only presents empirical findings, with no theoretical analysis or practical applications.

(2) The complex possible choices of components make the experiment order not strictly natural and logical. 

(3) Missing experiments on some possible choices of components make the conclusions of the paper not that strong, for example, it's hard to conclude whether predicting clean images is more helpful than predicting noise for representation learning.

questions:
1. L 247: The paper claims "self-supervised learning performance is not correlated to generation quality". However, the selected tasks such as linear probes only consider coarse representations that are useful for high-level tasks. What about more fined representations such as segmentation, and positions of a specific object, etc, would it be correlated to generation quality?
2. L 369: why the DAE is expected to work directly on the image space, could you please explain the importance of working on the image space?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701187695

thread_type:
official_review

<thread_object>
creator: reviewer_sr52

<content>
summary:
This paper studies the representation learning ability of denoising diffusion models. Through a set of ablation studies that deconstruct a denoising diffusion model into a classical denoising autoencoder (DAE), the authors observe that only a very few modern components (such as a low-dimensional latent space) are critical for learning good representations. Experiments also show that a latent DAE, which largely resembles the classical DAE, can perform competitively in self-supervised learning.

strengths:
Studying the representation learning ability of denoising diffusion models is important and could provide useful insights for unifying models for generation and discrimination.

Overall, the paper is very well written. The experiments are solid, and the message is clear. The observation that a low-dimensional latent space is a critical component for representation learning is useful.

weaknesses:
1. While the study begins with denoising diffusion models, it ultimately leads to models that demonstrate strong representations for classification but not for generation. The FID is reported only in Table 1, which reveals a significant contradiction between classification accuracy and FID. 

3. For the goal of representation learning for classification without fine-tuning, the obtained latent DAE achieves slightly worse performance than MAE and contrastive learning.

4. The representation is extracted from the middle layer of the transformer for linear probing. Previous studies have found that the middle layer may not provide the best representation of a diffusion model for classification.

questions:
1. What are the FID scores for other modifications beyond Table 1, such as operating in the image space with PCA?

2. Does better classification accuracy always lead to worse FID scores? In other words, are the tasks of generation and representation learning (for recognition) fundamentally contradictory to each other? Or could we unify them?

3. Why is the middle layer of the transformer chosen as the representation for linear probing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730858847562

thread_type:
official_review

<thread_object>
creator: reviewer_lfx6

<content>
summary:
In this paper, the author studied the representation learning abilities of denoising-autoencoder-based diffusion models (DDMs). Throughout extensive ablation studies, they explored how various components of modern DDMs influence self-supervised representation learning. At the core of their philosophy is to deconstruct a DDM, changing it step-by-step into a classical DAE. This research process demonstrates that the main critical component for a DAE to learn good representations is a tokenizer that creates a low-dimensional latent space.

strengths:
**1.** In general, this paper contributes significantly to the intersections of diffusion models and representation learning.  The findings open up new avenues for future research in leveraging diffusion processes to enhance representation quality across diverse applications

**2.** The authors conducted extensive experiments to support their results. The paper is well-written and easy to follow.

weaknesses:
The reviewer has the following major concerns about this paper: 

**1.** It is not comprehensive to study the representation ability of diffusion models only by considering the classification of downstream tasks. The authors should provide more experiments on other tasks to support their conclusions. 

**2.** Although the observations of this work are really new and interesting, the authors seem to not fully discuss the implications of these findings.

questions:
**1.** The authors mainly focused on investigating the representation learning abilities of DiTs. Are there similar observations on U-Net-based diffusion models? 

**2.** Based on the experimental results, can we conclude that adding noise primarily impacts the generation capabilities of diffusion models rather than their representation learning ability? Are there any insights for this observation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732480655868

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Shared response from authors

comment:
We sincerely thank all the reviewers for their time, efforts, and thoughtful feedback. We are delighted to see all the reviews are positive about our work, with remarks highlighting various aspects:

- **Novelty**: “the observations of this work are really new and interesting” (Lfx6), “the paper is the first work trying to identify key representation components from generative models and could inspire future works” (Lm16), “interesting findings” (Lm16), “novel deconstructing process” (NrMn)
- **Writing**: “the paper is well-written and easy to follow” (Lfx6), “the paper is very well written … the message is clear” (SR52)
- **Significance**: “​​contributes significantly to the intersections of diffusion models and representation learning … open up new avenues for future research” (Lfx6), “Studying the representation learning ability … is important and could provide useful insights for unifying models for generation and discrimination” (SR52), “the paper sheds light on a potential framework for understanding the representation ability of diffusion models” (Lm16), “provide key insights” (NrMn)
- **Experiments**: “extensive ablation studies … conducted extensive experiments to support their results” (Lfx6), “the experiments are solid” (SR52), “highly empirical … conducting various experiments on different components” (Lm16), “extensive ablation study” (NrMn)

We have carefully addressed each reviewer’s comments and questions individually below, and we hope that our responses address all remaining concerns. Should there be any further clarifications needed, we would be happy to provide them.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper the authors conduct a sequence of experiments which progressively transforms a Denoising Diffusion Model in a Denoising Autoencoder to understand how the various aspects of the model impact the overall performance and to identify which components are essential (or not) for the model.

Overall, the reviewers are largely in agreement that the paper makes an interesting contribution.  While several weaknesses are noted by the reviewers, none of the issues appear to be critical flaws and instead appear to be more of avenues for potential improvement of the work, and I believe this work would be of interest to the community.