Title: Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Models

Representation Learning

Model Property Study
Abstract: Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at https://github.com/Darkbblue/generic-diffusion-feature.

Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features

Abstract

Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at this url.

1 Introduction

DiffusionActivations

Inter-Module

ViT Basic Block Output

ViTAttentionActivations

PreviousStudiesConsider

Property Analysis & Qualitative Filtering

The complex architecture of the diffusion U-Net provides many activations that can serve as features. However, these activations inherently vary in quality, inducing significant performance gaps on discrimination. Hence, selecting a small yet effective subset from these activations has become a fundamental problem. In the early stage, Baranchuk et. al [2] perform a large-scale quantitative comparison among activations within Guided Diffusion [10]. Later, the activations they select are followed by most studies in this field, pursuing other improvements [29, 56, 58, 52, 60, 53, 24, 27].

However, we find that this fundamental issue is far from solved. On one hand, Baranchuk et. al [2] only consider activations between neighboring modules that comprise the main residuals. This means that many potential activations are excluded from the candidate pool, such as the queries and keys in the self-attention blocks. Moreover, recent developments of diffusion architecture, such as cross-attention [37] or embedded deep vision transformers (ViTs) [36], have introduced additional types of activations. Hence, as shown in upper Figure 1, only a small fraction of potential activations in modern diffusion models have been evaluated for their discriminative ability, which might hinder future work in this direction. For example, lower Figure 1 shows that Stable Diffusion XL (SDXL) [36], which is more advanced than Stable Diffusion v1.5 (SDv1.5) [37], fails to achieve better performance with the feature selection solution proposed in [2].

In this paper, we revisit the fundamental problem of feature selection, considering a more comprehensive candidate pool of activations. Due to its large volume, a full-scale quantitative comparison is no longer operational, urging us to modify the previous research methodology in [2]. As illustrated in Figure 1, instead of a direct quantitative comparison, we first explore the properties of diffusion U-Nets. These properties allow us to qualitatively and efficiently filter out many activations that are highly likely to be sub-optimal, shrinking the candidate pool for the quantitative comparison. More importantly, we find these properties are universal among diffusion models, making it possible to generalize our findings to more models beyond those covered in this paper.

In summary, the contributions of this work are three-fold:

Revisit of a fundamental problem : To the best of our knowledge, we are the first to point out that the fundamental issue of feature selection remains unresolved in the realm of diffusion feature.

Generic insights : The properties we find are universal among different diffusion U-Nets, which can provide valuable insights for future work.

Extensive validation : Extensive experiments on three discriminative tasks validate the effectiveness of the feature selection solutions induced by our insights.

2 Related Work

Diffusion models for discrimination. We discuss three mainstream ways to use diffusion models for discrimination. (i) Diffusion classifier [26, 7] utilizes Bayes' theorem to transform a pre-trained diffusion model into an image classifier. This method enjoys a theoretical guarantee and does not need additional training. However, it is limited to image-level tasks. (ii) The second way is to model discriminative tasks as image-to-image generation tasks with diffusion models [22, 23]. This method is suitable for various dense vision tasks but requires heavy training. (iii) Diffusion feature [2, 58, 56, 41, 29], the focus of this paper, follows the traditional practice of feature extraction to pursue the balance between wider applicability and less training. This makes it adaptable for different tasks and alleviates the training needs for the diffusion model. Only small downstream models may require training.

The diffusion feature approach has seen various improvements. Some techniques toggle the input settings of diffusion models, such as seeking better timesteps [60, 29]. The others add trainable parameters outside the diffusion model to refine the outputs or provide an efficient fine-tuning alternative [58, 27, 46]. Additionally, some studies explore completely training-free methods that utilize spatial attention information [51, 43], and some attempts focus on text-free diffusion models [32, 33].

Despite the progress, previous methods only consider activations between neighboring modules, leaving many potential candidates unevaluated. Our work shows that such an overlook can hinder model performance. By introducing a more comprehensive feature selection solution, our method could generically enhance both existing and future diffusion feature approaches.

Analysis on model properties. The inner properties of neural networks have consistently received much attention [5, 34]. For transformers, Geva et al. [15] show that the feed-forward layers act as key-value memories and are interpretable for humans. Amir et al. [1] extend this insight to vision transformers and put it into practical applications such as image classification, while Vilas et, al. [45] try to make more detailed interpretation of ViT activations.

The methodology of these studies provides valuable guidance for our research. However, since diffusion models are trained for generation, our study relies more on qualitative analysis and feature visualization compared to previous work.

3 Preliminaries: Architecture of Diffusion U-Nets

Diffusion models typically involve a forward pass and a reverse pass [21]. During the forward pass, noise is gradually added to a clean image x 0 ∈ R 3 × w × h until the image resembles Gaussian noise, where w and h denote the width and height, respectively. This process can be denoted by x t ∼ q ( x t | x 0 ) , where q represents the noise posterior and t is the timestep. In the reverse pass, an end-to-end neural network ϵ θ , as parameterized by θ , learns to predict the noises and thus reconstruct the image. One such denoising step can be denoted as ϵ = ϵ θ ( x t , t, c ) , where ϵ is the predicted noise, and c is the condition that describes the expected image content. Although there are alternative formulations for diffusion models [39, 28, 40], they all rely on this neural network backbone ϵ θ . This study focuses on this backbone, typically implemented as U-Net [38], rather than other components of diffusion models. Next, we will detail the architecture of the diffusion U-Net and standardize the terminology referring to different parts of the model, as shown in Figure 2.

We start with an overview. Following the initial U-Net design [38], the U-Net has three main stages : down-stage, mid-stage, and up-stage. The down-stage reduces the resolution of activations, while the up-stage increases it. Both stages contain multiple resolutions , in each of which the activations share

Down-Stage

Mid-Stage

Up-Stage

In/Out

Conv

Module

10101

ViT

Module

M

Down/Up

Resolution

Modules

InViT

Self-Attention

Basic

Cross-Attention

Block

Feed-Forward

the same resolution. Furthermore, each resolution includes several modules , including ResModule (convolutional ResNet [20] structures), ViT Module [12], and Downsampler/Upsampler (simple convolutional layers). Previous diffusion feature approaches only consider activations between these adjacent modules, i.e. , inter-module activations.

We next dive into the details below the module level. Among these modules, ResModule and ViTs adopt residual connection [20], where an increment activation is added element-wise to the residual activation to refine it. Specifically, ResModule uses simple convolutional layers to produce increments, whereas ViTs use complex attention mechanisms, which will be further explained next. Typically, ViT operates as a standalone model followed by a decoder that produces the output predictions for visual tasks [12, 25, 9, 18]. However, in the diffusion U-Net, multiple ViTs are embedded into the network, and their outputs serve as increment activations.

Furthermore, each ViT module consists of several stacked basic blocks . A basic block typically has a self-attention layer to perform attention on the image itself and a feed-forward layer , essentially a two-layer MLP [15]. Modern diffusion U-Net introduces an additional cross-attention layer between the two layers, enabling the fusion of the image and additional textual prompts. Besides, each layer includes a residual connection, meaning that the increment activation added to the outer residual is also the internal residual.

In a nutshell, the architecture described above provides abundant activations that can serve as dense features. Given the massive activations, it is no longer operational to perform a full-scale quantitative comparison. Hence, we next make a comprehensive analysis of model properties to better understand these activations, which can help the qualitative filtering.

4 Distinct Properties of Diffusion U-Nets

The diffusion U-Net has many interesting properties, but now we only focus on those distinct from the knowledge of traditional U-Net [38] or ViTs [44]. As shown in Figure 3, this section highlights three noticeable properties, each of which corresponds to a different level of the diffusion U-Net architecture described in Section 3. Notably, these properties can be universally observed in different samples and diffusion models, though all visualization in the main content is conducted on the same image and SDXL model for consistency . Additional visualization is provided in Appendix A. Besides, the omitted properties are available in Appendix E.

(b, In-Resolution Level) There are in-resolution granularity changes

PreviousKnowledgeof

Overall U-Net Granularity

(c, Sub-Module Level) Without positional embeddings,

thereisstilllocalityinself-attentionquery&key

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency.

butitschangeindown/up-stageis asymmetric

Noisy

→StillClean

Down-Stage

Mid-Stage

Up-Stage

4.1 Asymmetric Diffusion Noises

The first property is at the macro level and closely related to the overall diffusion process. It is common that high-frequency signals are typically noisy [5, 34]. This phenomenon can also be observed in diffusion U-Nets, especially within the increment activations of residual connection. However, this does not mean that low-frequency signal activations in diffusion U-Nets are free from noises. As shown in Figure 3(a), the diffusion process introduces a new type of noise that also impacts low-frequency signals. This is not surprising since the diffusion process requires the backbone to process noisy inputs and predict noises as outputs. As a result, activations near the inputs or outputs, regardless of their frequency, also suffer from such noises. Considering the special cause, we name such noises diffusion noises .

How does the influence of diffusion noises spread across diffusion U-Nets? As shown in Figure 3(a), diffusion noises exist throughout the entire down-stage, with a decreasing magnitude. Remarkably, during the early half of the up-stage, activations are rather clean, with no perceivable diffusion noises. Only in the later half do diffusion noises start to resurface. The existence of this asymmetric behavior can also be indirectly supported by the ablation curves in many fellow studies such as [2, 32, 33]. Furthermore, as shown in Appendix A, such an asymmetric pattern is consistent across activations in different diffusion models.

Similar to common high-frequency noises, diffusion noises can degenerate feature quality. Hence, this property can serve as a criterion for identifying and filtering out sub-optimal activations, which we will delve into in Section 5.

4.2 In-Resolution Granularity Changes

The second property is at the in-resolution level and closely related to recent advances in diffusion architectures. Specifically, the design of U-Net follows the idea of resolution hierarchy [38]. Consequently, the overall architecture displays a fine-coarse-fine granularity trend, looking like the alphabet 'U'. Traditional U-Nets implement this architecture with a relatively large number of resolutions, while each resolution is typically small, equipped with two or three simple convolutional layers.

Hence, the understanding of traditional U-Net focuses on the granularity changes across resolutions, implicitly assuming that the change within a single resolution is negligible.

However, diffusion U-Nets become much 'fatter'. In other words, modern diffusion U-Nets typically have much fewer resolutions, but each resolution is significantly enlarged. For example, SDv1.5 only has four resolutions [37], and SDXL further decreases this number to three [36], as shown in Figure 2. Meanwhile, each resolution can produce much more activations, primarily thanks to the embedded ViT modules. This architectural evolution makes the granularity change within a single resolution more significant, as depicted in Figure 3(b).

Different granularity carries varied information and quality, resulting in different discriminative performance on downstream tasks [2, 29]. Hence, this discovery of in-resolution granularity changes highlights the necessity to evaluate more activations, especially those within the embedded ViTs, as they are of different granularity.

4.3 Locality without Positional Embeddings

For the third distinct property, we delve into the sub-module level, i.e. , the blocks in embedded ViT modules. Positional embeddings, which are widely used in language transformers [44] and ViTs [12], aim to provide spatial information for each input token. Consequently, the activations of traditional ViTs display strong positional information, where the latent pixel resembles nearby pixels more than those that are semantically similar but far away [1]. This phenomenon is significant for the layers close to the inputs. When the layer goes deeper, the tokens are refined with semantic information, making the activations display less positional information. However, only in the last few layers, such positional information becomes negligible.

In contrast, ViT modules in diffusion U-Nets do not use positional embeddings [37, 36], perhaps because the other U-Net components have provided sufficient spatial cues. This change results in distinct properties of the activations. On one hand, positional information is negligible for most activations despite how near they are to the inputs. For example, even in the first basic block, crossattention query activations contain no perceivable positional information. On the other hand, the queries and keys of self-attention still display non-negligible positional information, marked with orange circles in Figure 3(c). Specifically, the latent pixel on the horse's neck is a light blue color, similar to the pixels to its left that actually represent the background. In contrast, the pixels above the circle are in purple color, though they also represent the horse's neck. Such comparison shows that a latent pixel is more similar to other pixels that are spatially near it than those semantically closer to it.

We name this phenomenon locality since it has a different mechanism from that induced by positional embeddings. As pointed out in [12], self-attention allows ViT to integrate global and local information even in the shallow layers, and the attention scope enlarges w.r.t. layer depth. Even without positional embeddings, self-attention activations are generally consistent with this insight, leading to the existence of locality. Nevertheless, the magnitude has indeed greatly reduced, compared to the visualization of conventional ViT activations in [1]. As shown in Figure 3(c), in shallow activations, locality exists but is inherently weaker, as much semantic information is still reserved. In addition, locality quickly diminishes as the layer goes deeper.

Since positional information can degrade the quality of activations [1], its absence has the potential to enhance the activations in ViT modules. Moreover, locality can play a special role in activation filtering, as presented in Section 5.

4.4 Universality of Three Properties

Although the visualization in Figure 3 is conducted only on SDXL, the scope of these properties is not limited to the specific architecture. Further supporting evidence is available in Appendix A.

(i) Diffusion noises directly arise from the diffusion process. Hence, it is promising to extend this property to other diffusion backbones.

(ii) In-resolution granularity changes come from the 'fatness' of U-Nets, making it potentially applicable to more traditional U-Nets.

(iii) Locality originates from the self-attention mechanism in ViT architectures, so it is broadly applicable to standalone or embedded ViTs where positional embeddings are absent.

(a)Resolution#O: Diffusion Noises Are Negligible

(b)

Block#0

Block#5

Block#9

Self-Atten.

Resolution#1:DiffusionNoisesinLaterHalf

36.79

58.59

61.99

Cross-Atten.

80

Resolution#0

59.31

64.70

59.56

60

(c)

Self-Atten.

OtherActivations

40

Visualization

20

Resolution#1

0

41.41

34.58

23.10

5 Enhanced Feature Selection from Diffusion U-Nets

So far, we have had a more comprehensive understating of the properties of diffusion U-Nets. All these properties, especially in-resolution granularity changes, encourage us to reconsider the feature selection solution, with a special emphasis on the activations in ViT modules. With these properties, we are also able to filter out many low-quality activations qualitatively, followed by a thus simplified quantitative comparison.

5.1 Qualitative Filtering

Avoiding Diffusion Noises. As shown in Figure 4(a), diffusion noises tend to degrade the quality of activations. Hence, it is natural to filter out the activations severely affected by diffusion noises from the candidate pool. Specifically, according to the asymmetric trend of diffusion noises, we only consider activations in the early half of the up-stage , which are rather clean. This approach will significantly reduce the number of candidate activations and simplify the quantitative comparison.

Avoiding Self-Attention Locality. The locality in self-attention modules is another important factor that can degrade the activations. The empirical evidence in Figure 4(b) demonstrates that these activations are generally inferior to the others, such as those from cross-attention layers or the outputs of ViT basic blocks. Consequently, it is rather safe to filter out most activations in self-attention modules from our candidate pool.

Using Locality to Suppress Diffusion Noises. So far, the activations in the candidate pool are clean and free from locality. However, all these activations are low-resolution ones since highresolution activations are generally noisy and thus filtered out. This is unfavorable since some detailed information might only exist in high-resolution activations. To address this issue, we exploit a side effect of self-attention locality. Specifically, as indicated in Figure 4(c), locality can help suppress diffusion noises via its focus on spatial structures. Although locality is sub-optimal, it is still superior to severe diffusion noises. In view of this, the candidate pool reserves self-attention activations extracted from the later half of the up-stage .

We have filtered out many candidate activations based on the distinct properties of diffusion U-Nets. Additionally, all increment activations in residual connection can be further filtered out since they introduce high-frequency noises [5, 34]. After such qualitative filtering, a small but high-quality candidate pool is available. Taking SDXL as an example, the number of candidates decreases from 279 to 63, i.e. , a 78% reduction. Next, we explain how to conduct this quantitative comparison briefly.

5.2 Quantitative Comparison

The quantitative comparison follows the protocol of [2]. Specifically, given an input image x 0 , we first perform the forward pass with a pre-defined timestep t to generate the noisy image x t . Then, the U-Net backbone conducts one denoising step. Instead of collecting the model output ϵ , we gather U-Net activations and consolidate them to the candidate pool, as described in Section 5.1. Afterward, each activation is individually fed to a downstream model to evaluate its discriminative potential, with the details of the model described in Appendix C. Such comparison is made fair by using the same dataset for all activations. Finally, we can obtain the capability ranking of activations from each resolution, according to which features can be selected wisely. Notably, we conduct such comparisons on multiple datasets to guarantee generalizability, since the best activations may differ among different scenes [2]. Thanks to qualitative filtering, the time cost for each (model, dataset) pair has been reduced by more than one week , equipped with Nvidia(R) RTX 3090 GPUs.

Given the capability ranking, feature selection is in fact flexible, as it is possible to combine multiple activations and specifically tune the choice for a task. For practicality, we provide off-the-shelf feature combinations for SDv1.5 and SDXL that are likely to generically perform well in Appendix B, according to the results detailed in Appendix D. Each of them consists of four activations mainly from ViT modules rather than inter-module positions . Taking SDv1.5 as an example, one of the four selected features is from one self-attention layer in the highest resolution, which utilizes our observation in Section 4.

6 Experimental Validation on Multiple Discriminative Tasks

To validate the effectiveness of our feature selection solution, the experiments are conducted on three popular discriminative tasks: semantic correspondence, semantic segmentation, and label-scarce segmentation. The SOTA methods for each task are selected as competitors. Besides, we also compare with two baselines that select the conventional inter-module activations as features [2], named Legacy-v1.5 and Legacy-XL . For our method, we provide the following implementations:

Ours-v1.5 & Ours-XL : Features extracted from SDv1.5 and SDXL, respectively.

Ours-XL-t : For fairness, we further enhance SDXL features with some additional techniques that are also adopted by SOTAs. The techniques we select, i.e. , attention maps [58] and feature amalgamation [2, 29, 60], are relatively simple and lightweight.

More experimental details are in Appendix C, such as task information, evaluation metrics, and implementation details. Besides, Appendix D presents additional results not covered here.

6.1 Empirical Results on Semantic Correspondence

We present the experimental results for semantic correspondence in Table 1. From the results, we have the following observations:

(i) Ours-v1.5 outperforms Legacy-v1.5 (77.78 v.s. 75.14 on PCK@0.1img). The main reason is that Legacy-v1.5 fails to effectively handle the diffusion noises in high-resolution activations. Previous approaches either reserve the noisy activations and thus suffer from performance degradation [2, 52], or simply discard high-resolution activations and thus suffer from information loss [58, 46]. In contrast, our approach uses self-attention locality to suppress diffusion noises and harvest better high-resolution activations.

(ii) Legacy-XL is inferior to Legacy-v1.5 (66.00 v.s. 75.14 on PCK@0.1img). At first glance, this result is counter-intuitive since SDXL is more advanced than SDv1.5. However, the analysis in Section 4 can unravel the mystery. Specifically, since SDXL has more ViT modules, the more valuable activations shift from inter-module positions to these embedded ViTs. Since the baseline does not consider ViT modules, Legacy-XL fails to achieve better performance. In contrast, Ours-XL shows improvement over Ours-v1.5 (81.72 v.s. 77.78 on PCK@0.1img). This is consistent with the advance in model architecture and again validates our analysis.

(iii) Ours-XL-t significantly outperforms the SOTA method with a similar amalgamation technique, i.e. , DHF [29] ( 83.90 v.s. 72.56 on PCK@0.1img). This performance gain again validates the effectiveness of our method.

6.2 Empirical Results on Semantic Segmentation

As shown in the left part of Table 2, Ours-XL-t and Ours-XL achieve state-of-the-art performance on the semantic segmentation task ( 45.71 and 43.45 v.s. 40.89 on ADE20K), demonstrating its effectiveness and generalizability. Furthermore, the most competitive SOTA, Meta Prompts [46], introduces a large number of trainable parameters and uses the diffusion U-Net recurrently, which is rather time-consuming. In contrast, our method delivers superior results with efficiency maintained.

Unlike the results in semantic correspondence, Legacy-XL outperforms Legacy-v1.5 on CityScapes (71.67 v.s. 64.01), and the performance gap between Legacy-v1.5 and Ours-v1.5 is narrow (64.01 v.s. 64.10). This is because this task utilizes a relatively large-scale downstream model, which can significantly refine the input features and thus reduce the gap in feature quality. Nevertheless, Ours-XL still achieves a significant improvement over Legacy-XL (74.47 v.s. 71.67).

6.3 Empirical Results on Label-Scarce Segmentation

One advantage of diffusion features is the applicability to label-scarce scenarios [2]. For validation under such conditions, we experiment on the label-scarce segmentation task, with results presented in the right part of Table 2. The observations are generally similar to those on the semantic correspondence task. For example, Ours-v1.5 outperforms Legacy-v1.5 (60.2 v.s. 59.4), Legacy-XL is inferior to Legacy-v1.5 (53.0 v.s. 59.4), and Ours-XL is better than Ours-v1.5 (62.7 v.s. 60.2).

Next, we focus on the comparison with the SOTA method, DDPM [2]. Although DDPM is a relatively early study, it outperforms the other competitors and our most implementations. This result has two reasons. On one hand, DDPM performs the diffusion process directly in the image space

rather than the currently common practice of compressed latent space. Although inefficient, such an implementation yields better discriminative features. On the other hand, DDPM utilizes diffusion models specifically trained on each dataset. In comparison, we use pre-trained general-purposed SDv1.5 or SDXL to be more efficient and thus consistent with the motivation of diffusion feature. Hence, it is rather challenging to surpass this SOTA method. Fortunately, our best implementation, Ours-XL-t , achieves this goal with the help of additional lightweight techniques ( 66.3 v.s. 65.0).

7 Conclusion and Future Work

In this study, we revisit the fundamental problem of feature selection from diffusion U-Nets. We point out that prior arts only consider a limited range of potential activations. In contrast, we consider a much wider range of activations as candidates, especially those extracted from the embedded ViT modules. Given the large volume of the candidate pool, we first analyze the properties of diffusion U-Nets. The properties we find are universal such that our observations are not limited to the specific diffusion architecture. Based on these properties, we qualitatively filter out many activations with low quality, facilitating the following quantitative comparison. On top of this, concrete feature selection solutions are proposed for two popular diffusion models, i.e. , SDv1.5 and SDXL. Finally, extensive experiments on three discriminative tasks validate the effectiveness of our method.

However, we are not sure whether our observations can generalize well to recently-developed DiT models [35] since they have a markedly different architecture from U-Net-based diffusion models. Thus, analyzing DiT models is a promising topic for future research.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, U21B2038, U23B2051, 61931008, 62122075 and 62025604, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680000, in part by the Innovation Funding of ICT, CAS under Grant No.E000000, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20240384.

References

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCV Workshop on What is Motion For , 2022.

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In The Tenth International Conference on Learning Representations , 2022.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Annual Conference on Neural Information Processing Systems , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision, ICCV , pages 9630-9640, 2021.

Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill , 4(3):e15, 2019.

Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. CoRR , abs/2401.09047, 2024.

Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Your diffusion model is secretly a certifiably robust classifier. CoRR , abs/2402.02316, 2024.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition , pages 3213-3223, 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Annual Conference on Neural Information Processing Systems , pages 8780-8794, 2021.

Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskclip. In International Conference on Machine Learning, ICML , volume 202 of Proceedings of Machine Learning Research , pages 8090-8102, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations , 2021.

Niladri Shekhar Dutt, Sanjeev Muralikrishnan, and Niloy J. Mitra. Diffusion 3d features (diff3f) decorating untextured shapes with distilled semantic features. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4494-4504, 2024.

Raphael Antonius Frick and Martin Steinebach. Diffseg: Towards detecting diffusion-based inpainting attacks using multi-feature segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3802-3808, 2024.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5484-5495, 2021.

Rui Gong, Martin Danelljan, Han Sun, Julio Delgado Mangas, and Luc Van Gool. Prompting diffusion representations for cross-domain semantic segmentation. CoRR , abs/2307.02138, 2023.

Boyu Han, Qianqian Xu, Zhiyong Yang, Shilong Bao, Peisong Wen, Yangbangyan Jiang, and Qingming Huang. Aucseg: Auc-oriented pixel-level long-tail semantic segmentation, 2024.

Shijie Hao, Yuan Zhou, and Yanrong Guo. A brief survey on semantic segmentation with deep learning. Neurocomputing , 406:302-321, 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15979-15988, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Annual Conference on Neural Information Processing Systems , 2020.

Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. DDP: diffusion model for dense visual prediction. In IEEE/CVF International Conference on Computer Vision , pages 21684-21695, 2023.

Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. CoRR , abs/2312.02145, 2023.

Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães, and Pietro Perona. Text-image alignment for diffusion-based perception. CoRR , abs/2310.00031, 2023.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In IEEE/CVF International Conference on Computer Vision , pages 2206-2217, 2023.

Xinghui Li, Jingyi Lu, Kai Han, and Victor Prisacariu. Sd4match: Learning to prompt stable diffusion model for semantic matching. CoRR , abs/2310.17569, 2023.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations , 2023.

Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. In Annual Conference on Neural Information Processing Systems , 2023.

Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. CoRR , abs/1908.10543, 2019.

Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for visual correspondence. In Computer Vision - ECCV 2020 - 16th European Conference , volume 12360, pages 346-363. Springer, 2020.

Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion models beat gans on image classification. CoRR , abs/2307.08702, 2023.

Soumik Mukhopadhyay, Matthew Gwilliam, Yosuke Yamaguchi, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Tianyi Zhou, and Abhinav Shrivastava. Do text-free diffusion models learn discriminative visual representations? CoRR , abs/2311.17921, 2023.

Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill , 2(11):e7, 2017.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision , pages 4172-4182, 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR , abs/2307.01952, 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10674-10685, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention , volume 9351, pages 234-241. Springer, 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations , 2021.

Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In The Eleventh International Conference on Learning Representations , 2023.

Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In Annual Conference on Neural Information Processing Systems , 2023.

Kashvi Taunk, Sanjukta De, Srishti Verma, and Aleena Swetapadma. A brief review of nearest neighbor algorithm for learning and classification. In 2019 international conference on intelligent computing and control systems (ICCS) , pages 1255-1260. IEEE, 2019.

Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar González-Franco. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. CoRR , abs/2308.12469, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Annual Conference on Neural Information Processing Systems , pages 5998-6008, 2017.

Martina G. Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. In Annual Conference on Neural Information Processing Systems 2023 , 2023.

Qiang Wan, Zilong Huang, Bingyi Kang, Jiashi Feng, and Li Zhang. Harnessing diffusion models for visual perception with meta prompts. CoRR , abs/2312.14733, 2023.

Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Openauc: Towards auc-oriented open-set recognition. In Annual Conference on Neural Information Processing Systems , 2022.

Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Optimizing partial area under the top-k curve: Theory and practice. IEEE Trans. Pattern Anal. Mach. Intell. , 45(4):5053-5069, 2023.

Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re-weighting and logit-adjustment for imbalanced learning. In Annual Conference on Neural Information Processing Systems , 2023.

Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. In Annual Conference on Neural Information Processing Systems , 2023.

Changming Xiao, Qi Yang, Feng Zhou, and Changshui Zhang. From text to mask: Localizing entities using the attention of text-to-image diffusion models. CoRR , abs/2309.04109, 2023.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2955-2966, 2023.

Xingyi Yang and Xinchao Wang. Diffusion model as representation learner. In IEEE/CVF International Conference on Computer Vision , pages 18892-18903, 2023.

Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. CoRR , abs/1506.03365, 2015.

David Junhao Zhang, Mutian Xu, Chuhui Xue, Wenqing Zhang, Xiaoguang Han, Song Bai, and Mike Zheng Shou. Free-atm: Exploring unsupervised learning on diffusion-generated images with free attention masks. CoRR , abs/2308.06739, 2023.

Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements DINO for zero-shot semantic correspondence. In Annual Conference on Neural Information Processing Systems , 2023.

Zhengxin Zhang, Qingjie Liu, and Yunhong Wang. Road extraction by deep residual u-net. IEEE Geosci. Remote. Sens. Lett. , 15(5):749-753, 2018.

Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In IEEE/CVF International Conference on Computer Vision , pages 5706-5716, 2023.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In IEEE Conference on Computer Vision and Pattern Recognition , pages 5122-5130, 2017.

Jingyi Zhou, Jiamu Sheng, Jiayuan Fan, Peng Ye, Tong He, Bin Wang, and Tao Chen. When hyperspectral image classification meets diffusion models: An unsupervised feature learning framework. CoRR , abs/2306.08964, 2023.

Contents

A Additional Visualization for Distinct Properties of Diffusion U-Nets

A.1 Visualization of Other Sample Images

We have provided activation visualization from SDXL in a simple outdoor scene presenting a horse in the wild. Next, we will show visualization from the same model in different scenes to demonstrate the universality of the observed properties. Another simple outdoor scene presenting a cat is in Figure 5. Two simple indoor scenes are visualized in Figure 6 and Figure 7. Two complex outdoor scenes of urban streets are visualized in Figure 8 and Figure 9. Two complex indoor scenes are shown in Figure 10 and Figure 11.

(b,In-ResolutionLevel)Therearein-resolutiongranularitychanges

(c,Sub-ModuleLevel)Without positional embeddings,

there is still locality inself-attention query&key

SDXL,SimpleOutdoor

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency.

butitschangeindown/up-stageisasymmetric

Noisy

→Still Clean

Down-Stage

Mid-Stage

Up-Stage

(b,In-Resolution Level) There are in-resolution granularity changes

22

SDXL,

Simple

(c, Sub-Module Level) Without positional embeddings,

Indoor

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency.

but its change in down/up-stage is asymmetric

Noisy

→StillClean

Down-Stage

Mid-Stage

Up-Stage

(b,In-Resolution Level) There are in-resolution granularity changes

HITES

STAR

(c, Sub-Module Level) Without positional embeddings,

thereisstilllocalityinself-attentionquery&key

SDXL,SimpleIndoor

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency. but its change in down/up-stage is asymmetric

Noisy

→StillClean

Down-Stage

Mid-Stage

Up-Stage

(b, In-Resolution Level) There are in-resolution granularity changes

(c,Sub-Module Level)Without positional embeddings,

thereisstilllocalityinself-attentionquery&key

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency. butitschangeindown/up-stageisasymmetric

Noisy

→StillClean

Down-Stage

Mid-Stage

Up-Stage

A.2 Visualization from Other Diffusion Models

Playground v2 Activations. Playground v2 2 shares the same architecture as SDXL but is trained independently, and it is claimed to be more powerful in generation. In Figure 12, we present its activation visualization using the same horse image as the primary SDXL visualization. Compared to SDXL, Playground v2 activations are less noisy, particularly in the down-stage. This supports the claim that Playground v2 is a stronger model.

SDv1.5 Activations. SDv1.5, as an older model, has a slightly different architecture from SDXL. Specifically, SDv1.5 has four resolutions instead of three, but its ViTs contain only one layer. Moreover, in SDXL, only the highest resolution lacks ViT due to efficiency concerns, while in

(b, In-Resolution Level) There arein-resolution granularity changes

(c,Sub-ModuleLevel)Withoutpositionalembeddings,

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency.

but itschangeindown/up-stage is asymmetric

Noisy←

→StillClean

Down-Stage

Mid-Stage

Up-Stage

(b, In-Resolution Level) There are in-resolution granularity changes

(c,Sub-ModuleLevel)Without positional embeddings,

SDXL, Complex Indoor

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency.

butitschangeindown/up-stageisasymmetric

Noisy

→StillClean

Down-Stage

Mid-Stage

Up-Stage

SDv1.5, only the lowest resolution lacks ViT due to its low resolution. Despite the architectural changes, the three unique properties still apply to SDv1.5, as shown in Figure 13.

Video Diffusion Activations. To further demonstrate the universality, we even select a diffusion model for video generation [6] for visualization. This model is based on the SDv1.5 architecture, with an additional temporal attention layer inserted after cross-attention to enable sequential generation. Although this model is designed for a different task, we can still observe the three unique properties in Figure 14.

Conventional U-Net Activations. As comparison to diffusion U-Net, we also visualize some activations from a conventional U-Net in Figure 15, where the three properties of diffusion U-Net can hardly be observed.

(b,In-ResolutionLevel)There arein-resolutiongranularity changes

(c,Sub-ModuleLevel)Withoutpositional embeddings,

SDXL, Complex Indoor

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency. but itschangeindown/up-stage is asymmetric

Noisy

StillClean

Down-Stage

Mid-Stage

Up-Stage

(b,In-Resolution Level) There are in-resolution granularity changes

(c,Sub-ModuleLevel)Without positional embeddings,

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency.

butitschangeindown/up-stageisasymmetric

Noisy

→StillClean

Down-Stage

Mid-Stage

Up-Stage

(b,In-ResolutionLevel)There arein-resolutiongranularity changes

(c,Sub-ModuleLevel)Withoutpositional embeddings,

SDv1.5

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency. butitschangeindown/up-stageisasymmetric

Noisy

→StillClean

Down-Stage

Mid-Stage

Up-Stage

(b,In-Resolution Level)There arein-resolutiongranularity changes

(c,Sub-ModuleLevel)Withoutpositionalembeddings,

thereisstilllocalityinself-attentionquery&key

(a, Macro Level) Diffusion noises are induced by the diffusion process and exist despite signal frequency. butitschangeindown/up-stageisasymmetric

Noisy

StillClean

15.7573

Down-Stage

Up-Stage

Down

Up

Mid

B Details of Our Method

In this section, we index all diffusion U-Net components in the order in which they are activated during a network forward run. Besides descriptions in natural language, we also index the activations using the same notations as used in our code implementation. These notations denote stages using down/mid/up , resolutions as level , and module index as repeat . Index starts to count from 0 instead of 1, following the convention in coding.

B.1 Feature Selection Solution for SDv1.5

We select four activations from SDv1.5, maintaining the same total feature channels as the conventional way to extract all output activations from each resolution.

(i) The cross-attention query activation from the 2nd ViT, the 2nd resolution. This provides coarse information for simple scenes (up-level1-repeat1-vit-block0-cross-q).

(ii) The inter-module activation after the 3rd ResModule, the 2nd resolution. This provides coarse information for complex scenes (up-level1-repeat2-res-out).

(iii) The cross-attention query activation from the 2nd ViT, the 3rd resolution. This provides finer information with a higher resolution (up-level2-repeat1-vit-block0-cross-q).

(iv) The self-attention key activation from the 1st ViT, the 4th resolution. This extracts features from the highest resolution, harnessing the noise suppression effect of self-attention locality (up-level3-repeat0-vit-block0-self-k).

We omit the index of basic blocks in ViTs, as SDv1.5 only contains one-layer ViTs. Additionally, we totally ignore the lowest resolution, as its activations have very low resolution ( 8 × 8 if the input image is 512 × 512 ), suggesting inferiority, as supported by the quantitative comparison.

B.2 Feature Selection Solution for SDXL

Four activations are selected from SDXL, trying to get similar total feature channels to the SDv1.5 feature selection solution.

(i) The output activation after the 8th basic block, the 1st ViT, the 1st resolution. This provides relatively coarse information for simple scenes (up-level0-repeat0-vit-block7-out).

(ii) The output activation after the 6th basic block, the 1st ViT, the 1st resolution. This provides relatively coarse information for complex scenes (up-level0-repeat0-vit-block5-out).

(iii) The cross-attention query activation from the 1st basic block, the 1st ViT, the 2nd resolution. This provides relatively fine information for simple scenes (up-level1-repeat0-vit-block0-crossq).

(iv) The output activation after the 1st basic block, the 1st ViT, the 2nd resolution. This provides relatively fine information for complex scenes (up-level1-repeat0-vit-block0-out).

We ignore the highest resolution since it is affected by diffusion noises and lacks ViTs from which we can extract self-attention activations. However, if the downstream model is strong enough to learn to suppress noises, it may be possible to additionally extract inter-module activations from this resolution to harness more information.

B.3 Feature Selection Solution with Additional Techniques

For the setting Ours-XL-t , we mainly utilize two simple techniques that are also adopted in some SOTA methods: (i) We additionally extract attention maps, i.e. , the similarity scores of cross-attention query and key, as dense features [58, 52]. Such attention maps are closely related to the semantics of prompts, thus providing important supplementary information. Despite its usefulness, this technique only adds a few additional channels to the features. (ii) We amalgamate features from different models [2, 29, 60] through simple concatenation. This technique is a common practice and can be seen as an extension of amalgamating different activations together as a whole feature. We next explain what activations are selected to implement the two techniquess.

We first extract features according to the feature selection solution for SDXL. Afterward, we extract additional features from SDv1.5:

(i) The cross-attention query activation from the 2nd ViT, the 2nd resolution (up-level1-repeat1vit-block0-cross-q).

(ii) The cross-attention query activation from the 2nd ViT, the 3rd resolution (up-level2-repeat1vit-block0-cross-q).

(iii) The upsampler output activation from the 3rd resolution, to harness high-resolution information (up-level2-upsampler-out).

(iv) The self-attention key activation from the 1st ViT, the 4th resolution, to harness high-resolution information (up-level3-repeat0-vit-block0-self-k).

(v) The attention maps averaged over all cross-attention layers in the up-stage.

We also extract one feature from Playground v2: the output activation after the 4th basic block, the 1st ViT, the 1st resolution (up-level0-repeat0-vit-block3-out).

B.4 Alternative Feature Selection Solution with Additional Techniques

Large-scale datasets for semantic segmentation mostly consist of images of complex scenes. In such cases, we find attention maps can be too noisy to be useful. Therefore, we discard the attention map technique and the entire SDv1.5 model, as it is weaker compared to the newer SDXL and Playground v2 models. To compensate for the loss of activations, we select additional activations from SDXL and Playground v2.

From SDXL, we select all the activations as described in the feature selection solution for SDXL and select one additional activation: the upsampler output activation from the 2nd resolution, to harness high-resolution information (up-level1-upsampler-out). From Playground v2, we select the following activations:

(i) The output activation after the 6th basic block, the 1st ViT, the 1st resolution (up-level0-repeat0vit-block5-out).

(ii) The cross-attention query activation from the 1st basic block, the 1st ViT, the 2nd resolution (up-level1-repeat0-vit-block0-cross-q).

(iii) The upsampler output activation from the 2nd resolution (up-level1-upsampler-out).

C Experimental Details

C.1 Semantic Correspondence

Task and Dataset. Semantic correspondence [18] involves finding a pixel in an image that semantically matches another keypoint pixel in a reference image, such as the hind legs of two different cats. We conduct experiments on the SPair-71k dataset [30].

SOTA Competitors. We provide the results from four SOTA methods: DINO [4] and DHPF [31] as non-diffusion-feature methods, as well as DIFT [41] and DHF [29] as diffusion feature methods.

Implementation Details. The semantic correspondence task can be done via the nearest neighbor algorithm [42], which is unsupervised and training-free [30]. We add one additional trainable convolutional layer before applying the nearest neighbor algorithm to refine the input features, which is also adopted by some SOTAs including DHF. The model is trained for two epochs, each containing 5,000 sample pairs, following conventional settings. Our implementation is derived from DHF, and we keep all hyper-parameters at their default settings.

C.2 Semantic Segmentation

Task and Dataset. Semantic segmentation [18] is essentially pixel-level classification. For this task, we choose the ADE20K dataset [59] with over 20k annotated images of 150 semantic categories, and the CityScapes dataset [8], which contains 5,000 fine annotated images of urban street scenes.

Evaluation Metric. We use mIoU metric, which is the mean over the IoU performance across all semantic classes [18]. For each image, IoU (Intersection over Union, ↑ ) is defined by #(overlapped pixels between the prediction and the ground truth) / #(union pixels of them).

SOTA Competitors. We choose three diffusion feature SOTA methods as competitors. ODISE [52] is an early method with a simple implementation. VPD [58] is another early study of this field, which introduces additional text adapter modules for improvement. Meta Prompts [46] is a newer method and shows significant improvements. We also report the performance of MaskCLIP [11], which is included as a competitor in the ODISE study.

Implementation Details. Both VPD and Meta Prompts perform full-scale fine-tuning on the diffusion U-Net using feedback from the discriminative task. This heavy fine-tuning does not entirely comply with the motivation of diffusion feature, which seeks a balance between wider applicability and less training, and is hard to extend to the larger SDXL model. Therefore, we keep the entire diffusion model frozen instead. As the setting has been changed, the performance of VPD and Meta Prompts in Table 2 is based on our experiments, not the reported results from the original papers. Our implementation directly uses the hyper-parameters reported in Meta Prompts.

C.3 Label-Scarce Segmentation

Task and Dataset. Using features from a pre-trained diffusion model ensures good performance even when labeled training data is scarce [2]. For this setting, we use a dataset collected in [2] and experiment on its Horse-21 subset, the data of which is sourced from LSUN [54]. This subset contains only 30 labeled training images to be consistent with the intuition. The semantic segmentation in the label-scarce scenario also uses the mIoU metric.

SOTA Competitors. We select the SOTA diffusion feature approach, DDPM [2], as the major competitor. We also include other representative segmentation methods: DatasetDDPM, MAE [19], SwAV [3], which are all reported in [2].

Implementation Details. Following DDPM [2], the downstream model is an ensemble of ten simple MLP networks, each conducting pixel-wise classification. The simplicity of the model is intended to demonstrate the innate capability and generalizability of diffusion models. Our implementation is derived from DDPM with only batch size changed among all hyper-parameters. We use a larger batch size for faster experiments as a smaller one does not improve performance. Additionally, this is also the setting for the quantitative comparison, as the compact size of the dataset can enhance efficiency .

D Additional Experimental Results

D.1 Generalizability across Different Scenes

As stated in Section 5, it is preferable to conduct the quantitative comparison across multiple datasets and choose activations that are optimal for each. This approach can enhance the generalizability of the selected features. To evaluate the generalizability of our features, we conducted an additional experiment, with results presented in Table 3.

In this experiment, we design an alternative feature selection solution for SDXL, based solely on quantitative results from a single dataset consisting of simple scenes. In this solution, we extract both optimal and slightly sub-optimal activations, maintaining the same total number of feature channels as the standard solution. The alternative solution achieves higher performance on the simple scene it is based on but performs significantly worse on the other scene. Therefore, we conclude that our standard feature selection solution achieves generalizability across different scenes, albeit with a slight performance drop compared to features specifically selected for each scene.

D.2 Quantitative Comparison Results

In this part, we present all the quantitative comparison results obtained following the protocol described in Section 5. These results are from a dataset of simple scenes (Horse-21 [2]) and a dataset of complex scenes (Bedroom-28 [2]). Since SDXL and Playground v2 share the same U-Net architecture, their results are shown in the same tables. We display the results from the lowest resolution in Table 4 and the results from the middle resolution in Table 5. We have also done a quantitative comparison on SDv1.5, and the results are shown in Table 6.

CommonActivations

ResModuleIncrement

ViTFeed-Forward

ViTSelf-Attention

(No Noises)

(ModerateNoises)

(Moderate Noises)

(Strong Noises)

Coarse

Fine

OverlyFine

E Other Properties of Diffusion U-Nets

E.1 High-Frequency Noises in Increment Activations

Typically, high-frequency signals are usually noisy [5, 34]. This applies to the diffusion U-Net, and we can further examine what activations are more vulnerable to such noises. To be specific, the diffusion U-Net contains many residual connection structures, and their increment activations are high-frequency signals prone to noises. Such increment activations include:

(i) The increment branch of ResModule. These activations are moderately noisy and thus usually less effective than inter-module activations.

(ii) The feed-forward layer activations within ViTs. These activations are also moderately noisy. However, this results from their significantly more channels compared to other activations, which can reduce their noise magnitude.

(iii) The self-attention value activations within ViTs. These activations are severely noisy and suffer significant degradation as they are the nested inner increments within embedded ViTs.

These activations are visualized in Figure 16 for a clearer illustration. Additionally, the residual activations within ViTs are at the same time also increments to the main inter-module residual. However, these activations are not obviously affected by high-frequency noise, possibly due to their dual role as ViT residuals.

E.2 Detailed In-Resolution Granularity Change

We have previously described the existence of in-resolution granularity changes, and this section will further detail the pattern of one such change. In one resolution of the up-stage, activations gradually shift from coarse to fine granularity, which aligns with intuition. However, the diffusion U-Net tends to overly refine the inter-module activations, resulting in slight noises in the last few inter-module activations due to excessive detail. This can be observed from the visualization of Figure 17. Consequently, a drop in discriminative performance is often seen near the end of one resolution.

E.3 Collaboration between Embedded ViTs

Multiple ViTs exist within one resolution; for example, one resolution in the up-stage typically contains three ViTs. These ViTs collaborate in refining the main inter-module residual. During this process, each ViT exhibits an inner granularity change as it produces the increment activation, and the

ViT

#1

ViT

#2

ViT

#3

Shifting&Overlapped

Granularity Spectra

changes of collaborating ViTs form a certain pattern. To be specific, the change of one ViT overlaps with the previous one to some extent but also shifts as a whole towards finer granularity, which is shown in Figure 18.

F Future Direction

It might be a good future direction to focus on more challenging discrimination scenarios. Specifically, long tail and out-of-distribution are important problems in discrimination, which can greatly hinder the performance of models that work well under i.i.d. settings [49]. There have been a few attempts to address this more challenging problem with diffusion models. For example, the disentanglement property of prompts might grant diffusion features cross-domain capability [16]. It is also possible to utilize diffusion models to synthesize training samples to adjust training data distribution [50, 55]. Given these attempts, more efforts are still required to be put in this direction. Moreover, there is a recent study [17] that might boost long tail and out-of-distribution discrimination using diffusion models. To be specific, AUC is an evaluation metric as well as a loss function that promotes good performance on both head and tail samples, which is an important tool for long tail and out-ofdistribution studies [47, 48]. Conventionally, AUC is applicable to image classification but not semantic segmentation and other pixel-level tasks, but the aforementioned study manages to adopt AUC for semantic segmentation. Given this new tool, it is now made more viable to attempt to enhance diffusion feature on long tail and out-of-distribution problems.

G Computation Resources

We use Nvidia(R) RTX 3090 and Nvidia(R) RTX 4090 GPUs for the experiments, all with 24GB VRAM. Most of our experiments, except label-scarce segmentation, require no additional storage besides the necessary space for model checkpoints and datasets. The label-scarce segmentation task first extracts features and stores them on the disk, and then loads them for the downstream task, which takes about 4GB.

The codes are designed to be able to run on a single GPU or less than 4 GPUs, while multiple experiments can run simultaneously if more GPUs are provided. Each experiment on the semantic correspondence task takes about 5 hours. Each experiment on the large-scale semantic segmentation task takes 2 to 3 days, depending on the dataset. Each experiment on the label-scarce segmentation task takes about 1 hour, but we repeat on 5 random splits following [2], which increases the overall time. All the experiments in sum can be done within two weeks.

Our quantitative comparison is based on the label-scarce segmentation task, each run taking about 40 minutes. The time is shorter because the quantitative comparison evaluates each activation individually. This comparison uses large storage, which is about 45GB per model per dataset. With the help of qualitative filtering, we can finish the quantitative comparison of one model on one dataset within 2 days.

H Asset License

SPair-71k: Available at https://cvlab.postech.ac.kr/research/SPair-71k/ .

Label-scarce segmentation datasets (sourced from LSUN): Available at https://github.com/ fyu/lsun .

ADE20K: Custom (research-only, non-commercial), at https://groups.csail.mit.edu/ vision/datasets/ADE20K/terms/ .

CityScapes: Custom, at https://www.cityscapes-dataset.com/license/ .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state our contributions in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Section 7, which are mainly related to DiT models [35].

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not make theoretical contributions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The most important information to reproduce our results is the exact activations we select as features, and it is detailed in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the codes after acceptance. Besides, we have provided the necessary information for reproducing the experiments in this manuscript.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have included the experimental details in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide error bars for some experimental results. Others are omitted as it can too computationally expensive, such as about two days per run.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no socetal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers where related assets are first mentioned and list their license in Appendix H.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the critical task of enhancing feature selection in diffusion models to improve performance in discriminative tasks such as semantic correspondence, semantic segmentation, and label-scarce segmentation. Previous methods often overlooked many potential activations within diffusion models, leading to suboptimal performance and limitations due to ignoring high-resolution activations and not effectively handling diffusion noises. The authors propose a comprehensive feature selection solution that leverages distinctive properties of diffusion U-Nets, including diffusion noises, in-resolution granularity changes, and locality without positional embeddings, to filter and select the most relevant features. Experimental results demonstrate that their method outperforms state-of-the-art techniques, achieving significant improvements in various metrics across multiple datasets.\n\nSTRENGTHS:\n- This paper introduces a novel approach to feature selection in diffusion models, focusing on qualitative analysis to filter out suboptimal activations before performing quantitative comparisons. This methodology shifts from the traditional full-scale quantitative comparison, making the process more efficient and potentially more accurate.\n- The authors identify three distinct properties of diffusion U-Nets that are leveraged for feature selection:\n1.  **Asymmetric Diffusion Noises**: The diffusion process introduces unique noises affecting both low- and high-frequency signals.\n2.  **In-Resolution Granularity Changes**: Modern diffusion U-Nets exhibit significant granularity changes within a single resolution due to fewer but larger resolutions.\n3. **Locality without Positional Embeddings**: Self-attention modules in diffusion U-Nets show a new type of locality that enhances activation quality without traditional positional embeddings.\n- The discovered properties means that the findings can be generalized to various diffusion models, providing valuable insights and a solid foundation for future research. This can guide the development of more effective and efficient discriminative models and applications in other fields.\n- The proposed method achieves SOTA performance across multiple discriminative tasks, including semantic correspondence, semantic segmentation, and label-scarce segmentation.\n\nWEAKNESSES:\n- Some more comparisons are recommended. While the authors claim to find unique properties in diffusion U-Nets, it lacks a detailed comparison with traditional U-Nets. A more comprehensive analysis of how these properties differ would provide better context.\n- Some visualizations, such as Figure 3(c) showing positional information in self-attention activations, could be clearer and better explained to enhance understanding of the findings.\n\nQUESTIONS:\nSee the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper highlights the importance of considering a broader range of activations within diffusion models. The authors propose three universal properties of diffusion U-Nets that aid in qualitatively filtering out activations that are clearly sub-optimal. On top of this, the authors can improve the efficiency of feature selection. By leveraging these properties, the proposed method demonstrates superior performance across multiple discriminative tasks, such as semantic correspondence and segmentations. This comprehensive approach to activation selection addresses a fundamental issue in diffusion models, enhancing their applicability and performance in various tasks.\n\nSTRENGTHS:\n1) The authors propose an efficient method for selecting high-quality activations.  With the help of qualitatively filtering out sub-optimal activations before quantitative comparison, the computational costs can be significantly reduced, compared with the prior arts.\n2) The authors present their observations in a clear and organized manner, making it easy for readers to understand the complex concepts and follow the research process. Besides, the main content and the appendix include numerous visual samples and activation visualizations, which help illustrate the properties and effectiveness of the proposed activation selection method.\n3) Extensive experimental results across multiple discriminative tasks validate the superiority of the proposed method over the competitors.\n\nWEAKNESSES:\nW1: This paper could benefit from introducing more notations to clearly define and differentiate between various activations and components within the diffusion models, which would further enhance the clarity of the descriptions.\nW2: While the empirical results are robust, the paper could include more theoretical analysis to provide deeper insights.\nW3: Some more latest related work are commended to be cited:\n[1] Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features, CVPR 2024. [2] DiffSeg: Towards Detecting Diffusion-Based Inpainting Attacks Using Multi-Feature Segmentation, CVPR 2024.\n\nQUESTIONS:\nI have no more questions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDiffusion models have achieved significant success in image generation and show great potential for various discriminative tasks. The authors rethink the foundational problem of feature selection within these models. To this end, they analyze the properties of diffusion models, including asymmetric diffusion noises, in-resolution granularity changes, and locality without positional embeddings, to filter out suboptimal activations. The results demonstrate that their proposed feature selection method outperforms state-of-the-art techniques across multiple discriminative tasks, achieving superior performances.\n\nSTRENGTHS:\n- This paper points out three unique properties of diffusion models—asymmetric diffusion noises, in-resolution granularity changes, and locality without positional embeddings—that provide valuable insights into improving discriminative tasks.\n- The authors propose an off-the-shelf solution for feature selection. Besides, the findings and methodologies presented are not limited to the specific models studied (SDv1.5 and SDXL).\n- The paper conducts extensive experiments across multiple discriminative tasks, including semantic correspondence, semantic segmentation, and label-scarce segmentation. The results, as well as the visual illustration, validate the effectiveness of the proposed feature selection method, demonstrating significant improvements over state-of-the-art techniques.\n\nWEAKNESSES:\n-  The authors acknowledge the uncertainty about whether the findings can generalize to newer models like DiT (Diffusion Transformer) due to architectural differences.\n- The qualitative filtering approach proposed is novel, but its scalability and efficiency in very large-scale settings are not fully demonstrated. How do the authors alleviate this problem?\n- The paper focuses on empirical results and qualitative analysis. There is a lack of rigorous theoretical foundation or mathematical proofs to support the proposed feature selection methodology and the identified properties of diffusion U-Nets. A more robust theoretical framework would strengthen the claims made.\n\nQUESTIONS:\nBesides those in weakness, I wonder can we combine multiple features during the qualitative analysis to further improve the performance? And how many features do you select for each image?\n\nLIMITATIONS:\nYes, as stated in Sec.7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new feature selection method for diffusion models by evaluating a broader range of activations, particularly those in embedded Vision Transformers (ViTs). The authors identify the limitations of current approaches that consider only a narrow range of activations and introduce a qualitative analysis to filter out low-quality activations in diffusion U-Nets. They develop specific feature selection solutions for popular diffusion models SDv1.5 and SDXL. Experiments demonstrate that this method outperforms state-of-the-art techniques in tasks like semantic correspondence, semantic segmentation, and label-scarce segmentation, showcasing its effectiveness and generalizability.\n\nSTRENGTHS:\n- This paper extends the evaluation to a wider range of activations within diffusion models, especially those in embedded ViT modules, which were previously overlooked. Since feature selection is a basic problem for diffusion features, this paper can boost future work in this direction.\n- The authors introduce a qualitative analysis to effectively filter out low-quality activations, simplifying the subsequent quantitative comparison and improving feature selection efficiency. What’s more, the observed properties are not limited to the discussed models, which can also inspire future work.\n- Extensive experiments demonstrate that the proposed feature selection method significantly outperforms state-of-the-art techniques in various discriminative tasks, validating its effectiveness and generalizability.\n\nWEAKNESSES:\n- As pointed out by the authors, the observations and methods proposed do not necessarily generalize well to recently developed DiT models since their architecture is markedly different from diffusion U-Nets .\n- Some details can be more clarified. For example, in Appenidx C.3, the authors state that “this is also the setting for the quantitative comparison”. In other words, quantitative comparison is conducted on Label-Scarce Segmentation, and the experiments on the other tasks follow this setting. Is this understanding correct? If so, I wonder why the authors select this setting to conduct quantitative comparison. \n- Besides, there are some typos. For example, in the introduction, “a fundamental problem to select” should be “a fundamental problem in selecting”. In the caption of Figure 3, “existing knowledge of other models” should be “existing knowledge about other models”.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nYes, the authors have clarified the limitations in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}